Skip to content

Reject catalog evidence timestamps outside DuckDB BIGINT range #309

Description

@kstonekuan

Current behavior

The catalog stores observation timestamps and interval bounds as signed BIGINT columns at src/hflow/catalog.py:82-95.

_normalized_intervals() and _normalized_observations() normalize NumPy scalars, check the integer type, and reject negative values. They don't enforce the signed 64-bit upper bound.

Measured on main at fbfd553:

_normalized_observations(... timestamp_ns=9223372036854775808) -> accepted
_normalized_intervals(... start_ns=9223372036854775808) -> accepted
DuckDB INSERT -> ConversionException: value is out of range for INT64

The catalog append reaches a backend conversion error after HFlow's evidence boundary has accepted the value. The error doesn't name the check, observation, interval, or timestamp field.

Pattern to copy

_normalized_measurements() rejects values the storage/query contract can't represent before any table write and includes the owning check and key in the error. The duplicate-observation regression also proves a boundary refusal leaves no committed episode file at tests/test_catalog_curation.py:199-222.

What to build

Require every stored nanosecond timestamp to fit the catalog's nonnegative signed BIGINT domain: 0 through 2**63 - 1.

Definition of done

  1. Observation timestamp_ns, interval start_ns, and interval end_ns accept 0 and 2**63 - 1.
  2. Each field refuses 2**63 and larger values with a contextual ValueError.
  3. Python and NumPy integer scalars follow the same range rule after normalization.
  4. Existing boolean, float, negative, and inverted-interval refusals stay unchanged.
  5. An out-of-range value is rejected before any episode or dependent Parquet file is committed.
  6. The maximum accepted value round-trips through observations_latest or intervals.
  7. Add outcome-focused coverage to the existing catalog validation tests.
  8. No catalog format change or new dependency is needed.

Non-goals

  • Changing the catalog timestamp column type
  • Supporting negative relative timestamps
  • Adding range rules to unrelated numeric measurements

Validation

uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest -q tests/test_catalog_curation.py
uv run pytest -q

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomershelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions