Skip to content

Centralize ingest URI parsing across CLI, server, and SDK #314

Description

@kstonekuan

Current behavior

The CLI validates ingest URIs inline at src/hflow/cli.py:1113-1135. The workspace server implements the same rule again at packages/hflow-server/src/hflow_server/_runtime.py:412-428.

The two entry points already differ:

That leaves the validity and normalization of one runtime URI dependent on which public entry point received it.

Verified against main at fbfd553.

Pattern to copy

parse_step_version() parses a public string once and returns a refined StepVersion. _validated_relative_key() owns the analogous containment rule for storage keys.

What to build

Add one core parser for a data-root-relative ingest URI and return a refined type such as DataRootRelativeUri. Use that parser from the CLI, workspace server, and AirflowClient.ingest() so all three accept, normalize, and refuse the same values.

Keep transport-specific presentation at the edges: the CLI still prints an ingest error and exits 2, the server still returns a 400, and the SDK raises ValueError before making an HTTP request.

Definition of done

  1. One function owns trimming, empty-value refusal, absolute-path refusal, and parent-escape refusal.
  2. A normal URI such as episodes-in/run_0001.mcap reaches the trigger conf unchanged.
  3. Surrounding whitespace has one documented result at every entry point. Preserve the server's current behavior by trimming it.
  4. Blank strings, absolute paths, and paths that normalize outside the data root are refused by the CLI, server, and SDK before runtime resolution or HTTP.
  5. resolve_episode_reference() consumes the refined value and no longer repairs a leading slash.
  6. Existing CLI and server error status behavior stays unchanged.
  7. Extend the closest CLI, server, and client boundary tests. Assert accepted or refused outcomes, not private helper calls.

Non-goals

  • Changing the trigger conf key or API response schema
  • Supporting absolute host paths in scheduled runs
  • Normalizing safe internal segments such as a/../b into a different persisted identity
  • Reusing every storage-key rule, including reserved mirror suffixes

Validation

uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest -q tests/test_runtime_cli.py tests/test_ingest_in_process.py \
  tests/test_runtime_client.py packages/hflow-server/tests/test_server_runtime.py
uv run pytest -q

Metadata

Metadata

Assignees

No one assigned

    Labels

    advancedNeeds codebase familiarity; not a starter issuebugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions