Skip to content

Core: Select inspect.partitions snapshot metadata by commit time#3706

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:fix/partitions-last-updated-snapshot
Open

Core: Select inspect.partitions snapshot metadata by commit time#3706
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:fix/partitions-last-updated-snapshot

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

InspectTable.partitions() aggregates files by partition and records the snapshot that most recently updated each partition. Its update condition compared last_updated_snapshot_id with the next snapshot's timestamp_ms. Snapshot IDs are identifiers, not commit-order values, so comparing these unrelated fields made the reported metadata depend on manifest-entry order.

This change follows Apache Iceberg Java's PartitionsTable behavior: compare the current last_updated_at with the snapshot commit timestamp, then update spec_id, last_updated_at, and last_updated_snapshot_id together when the incoming snapshot is newer.

Are these changes tested?

Yes. A parametrized regression test processes the same snapshots in both possible orders. The snapshot IDs deliberately decrease while commit timestamps increase, verifying that commit time alone selects the latest spec_id, timestamp, and snapshot ID.

The following checks pass locally:

  • PYTHONPATH=. uv run pytest tests/table/test_inspect.py -q (6 passed)
  • PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k "inspect_partition" -q (2 passed)
  • PYTHONPATH=. uv run --extra datafusion --extra pyiceberg-core pytest tests/table -q (329 passed)
  • make lint

Are there any user-facing changes?

Yes. table.inspect.partitions() now reports the latest spec_id, last_updated_at, and last_updated_snapshot_id for each partition independently of manifest-entry order. There are no API changes.

@fallintoplace
fallintoplace force-pushed the fix/partitions-last-updated-snapshot branch from 9505c52 to a87a2e2 Compare July 25, 2026 15:36
@fallintoplace fallintoplace changed the title Core: Fix snapshot metadata in inspect.partitions Core: Fix inspect.partitions snapshot metadata ordering Jul 25, 2026
@fallintoplace fallintoplace changed the title Core: Fix inspect.partitions snapshot metadata ordering Core: Select inspect.partitions snapshot metadata by commit time Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant