Skip to content

fix(table): align Data Evolution row-range pruning with Java - #628

Open
jerry-024 wants to merge 10 commits into
apache:mainfrom
jerry-024:fix_index_scan_manifest
Open

fix(table): align Data Evolution row-range pruning with Java#628
jerry-024 wants to merge 10 commits into
apache:mainfrom
jerry-024:fix_index_scan_manifest

Conversation

@jerry-024

@jerry-024 jerry-024 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Align Data Evolution row-range manifest pruning with Java two-stage direct intersection. This avoids retaining unrelated manifests and data files solely because their ranges belong to an overlapping transitive component, while preserving Java-compatible missing-provider semantics.

Changes

  • reuse RowRangeIndex normalization for sorted, merged query ranges
  • prune each manifest by its own [minRowId, maxRowId] envelope and each merged live entry by its actual row range
  • fail open per manifest or entry when row-range metadata is missing or invalid
  • apply ADD/DELETE netting before entry pruning and keep provider/read-type grouping in the later Data Evolution planning stage
  • NULL-fill missing nullable providers and reject missing non-nullable providers

Testing

  • focused regressions for overlapping manifest envelopes and entry ranges
  • fail-open coverage for missing, one-sided, invalid, zero-count, and overflowing ranges
  • Data Evolution end-to-end coverage for rolled BLOB/vector pruning and nullable gap NULL filling
  • real BTree lookup coverage from an ID predicate to an exact row range and nullable BLOB NULL filling
  • row-id pushdown rejects a missing non-nullable provider
  • Data Evolution reader module: 68 passed
  • cargo fmt --all -- --check and git diff --check

Notes

No planner memory option, temporary spill files, iterative candidate expansion, or additional scan trace fields are introduced.

@jerry-024 jerry-024 changed the title fix(table): bound row-range manifest planning memory fix(table): align row-range pruning with Java Jul 29, 2026
…fest

* upstream/main:
  fix(table): skip unrelated BLOB and vector files in row-range reads (apache#629)
  feat(datafusion): support database SQL statements (apache#627)
  fix(datafusion): apply dynamic options to vector search (apache#623)
  feat(io): let FileIO wrap a caller-provided filesystem operator (apache#618)
  feat(c): build a catalog-free Table from a resolved schema JSON (apache#595)
  feat: support batch incremental diff reads (apache#510)
  fix(spec): accept Java's long memory-size units and reject overflow (apache#621)
  fix(table): stop answering COUNT(*) from a placeholder row count (apache#624)
  feat(datafusion): support alter column (apache#625)
  fix(spec): validate bucket-key against table schema (apache#620)

# Conflicts:
#	crates/paimon/src/table/data_evolution_reader.rs
#	crates/paimon/src/table/table_scan.rs
@jerry-024
jerry-024 marked this pull request as ready for review July 30, 2026 02:05
@jerry-024 jerry-024 changed the title fix(table): align row-range pruning with Java fix(table): align Data Evolution row-range pruning with Java Jul 30, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

[Blocking] Null-fill partially uncovered ranges for nullable providers

retain_manifest_entry_row_ranges may retain a nullable BLOB/vector provider because it overlaps one selected range, even though other selected ranges fall into a gap in that provider.

For example, consider a normal anchor covering [0, 5], a nullable provider covering only [0, 1], and selected ranges [1, 1] and [3, 3]. The provider is retained because it overlaps [1, 1], so the source plan assigns the column to that provider. open_source_stream then rejects [3, 3] as uncovered and returns DataInvalid. However, selecting only [3, 3] prunes the provider completely and correctly returns NULL.

This makes the nullable missing-provider behavior inconsistent: adding a covered row turns a valid NULL-filled query into an error. Please make BLOB and vector sources emit NULL/placeholders for uncovered selected ranges, rather than NULL-filling only when the entire provider is absent. Please also add end-to-end tests covering mixed provider-covered and provider-gap ranges for both BLOB and vector columns. The existing test_selected_blob_fallback_rejects_uncovered_non_deleted_range currently codifies the erroneous behavior and should be updated accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants