Skip to content

fix(table): honor physical schemas in raw evolution reads - #636

Open
JingsongLi wants to merge 5 commits into
apache:mainfrom
JingsongLi:codex/data-evolution-single-file-predicate-pushdown
Open

fix(table): honor physical schemas in raw evolution reads#636
JingsongLi wants to merge 5 commits into
apache:mainfrom
JingsongLi:codex/data-evolution-single-file-predicate-pushdown

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • resolve the physical fields of each raw-convertible data-evolution file from its schema ID and write_cols
  • push predicates into raw-convertible reads when every row-id segment has a single file provider
  • preserve partial-column and missing-predicate-column NULL semantics by remapping predicates against the physical file schema
  • keep _ROW_ID, transformed BLOB/BLOB-view predicates, and column-merge reads on the residual-only path

Why

Data-evolution reads previously suppressed all file-level predicates because overlapping files may provide different columns for the same logical rows. That restriction is unnecessarily broad for raw-convertible splits: when each row-id segment has only one file, no column-wise merge is required, so the file reader can safely apply predicate pruning and row filtering. The exact post-read residual remains as a format-independent correctness backstop.

Raw-convertible files must also be decoded using their physical write_cols, even when their schema ID matches the table schema. This is required for positional formats such as .row. For example, if the table schema is [BLOB payload, INT id] but a data file physically stores only [id], decoding it as the full table schema interprets the id bytes as a BLOB offset and can read past the row payload. Resolving the physical fields fixes that pre-existing issue and also gives predicate pushdown the correct schema mapping.

Validation

  • added test_raw_row_file_uses_write_cols_when_blob_precedes_projection; it reproduces the positional decode failure with the old field resolution and passes with this change
  • cargo fmt --all -- --check
  • cargo test -p paimon data_evolution_reader::tests::test_raw_row_file_uses_write_cols_when_blob_precedes_projection -- --nocapture
  • cargo test -p paimon data_evolution_reader::tests::test_evolution_ -- --nocapture
  • cargo clippy -p paimon --lib -- -D warnings
  • git diff --check

@JingsongLi
JingsongLi marked this pull request as ready for review July 30, 2026 09:12
@JingsongLi JingsongLi changed the title perf(table): push predicates for raw evolution splits fix(table): honor physical schemas in raw evolution reads Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant