Skip to content

fix(table): keep every split of a format table under a positive LIMIT - #644

Open
sundapeng wants to merge 1 commit into
apache:mainfrom
sundapeng:fix/format-table-limit-pushdown
Open

fix(table): keep every split of a format table under a positive LIMIT#644
sundapeng wants to merge 1 commit into
apache:mainfrom
sundapeng:fix/format-table-limit-pushdown

Conversation

@sundapeng

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #643

A format table has no manifest, so scan planning does not know how many rows a data file holds. FormatTableScan::apply_limit_pushdown kept only the first limit splits anyway, which reads the number of files as if it were a number of rows.

An empty data file is enough to break it, and an empty data file is ordinary output: an engine that writes one file per task leaves one behind whenever a task matched nothing. A table of three rows whose first file is empty answers SELECT id FROM t LIMIT 1 with zero rows, and reports success.

Brief change log

  • Keep every split for a positive limit and let execution stop once it has enough rows. A limit of zero still needs no split.

This is what Java does. FormatTableScan.FormatTableScanPlan.splits returns an empty list only for a non-positive limit, with the same reason in a comment. paimon-rust's own Paimon-table path is already stricter than the format path was: PaimonTableScan::apply_limit_pushdown skips a split whose merged row count is unknown rather than counting it as if it held enough rows.

Tests

crates/integrations/datafusion/tests/format_table_statistics.rs: test_format_table_limit_is_not_answered_from_a_file_count writes an empty file followed by a three-row file and asserts LIMIT 1 returns a row. It fails on main with left: 0, right: 1.

cargo fmt --all -- --check
cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings
cargo test --locked -p paimon --lib
cargo test --locked -p paimon-datafusion --lib --test format_table_statistics --test count_pushdown --test scan_pruning_trace --test read_tables

API and Format

No API or storage format change.

Documentation

None needed.

A format table has no manifest, so scan planning does not know how many rows a
data file holds. apply_limit_pushdown kept only the first `limit` splits anyway,
which reads the number of files as if it were a number of rows.

An empty data file is enough to break it. An engine that writes one file per
task leaves one behind whenever a task matched nothing, so a table of three rows
whose first file is empty answers `SELECT id FROM t LIMIT 1` with zero rows, and
reports success.

Keep every split for a positive limit and let execution stop when it has enough
rows. A limit of zero still needs no split. This is what Java does:
FormatTableScan.FormatTableScanPlan.splits returns an empty list only for a
non-positive limit, with the same reason in a comment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Format table LIMIT can return fewer rows than the table has

1 participant