Skip to content

Commit 3e4a179

Browse files
committed
refactor(manifest): unify streaming/eager and executor/no-executor planning paths in ManifestGroup
Extract shared manifest/residual evaluator caching and entry-filtering rules (ignore_existing_, file filter, custom predicate) into a single FilePlanningStream::Evaluators helper, and collapse the previously separate sequential vs. executor-backed state machines in FilePlanningStream into one FilePlanningStream::EntryStream, with batch size 1 without an executor and kManifestReadBatchSize with one. ReadEntries()/Plan()/Entries() are rebuilt on top of the same EntryStream instead of duplicating manifest iteration and filtering via a separate ParallelCollect-based loop, removing the FIXME about porting Iceberg Java's ParallelIterable design. Also: - Dedupe data manifests (matching Java's ManifestGroup, which stores them in a Set) so a manifest referenced by overlapping snapshots is only scanned once. - Update FilterManifestEntries() doc comment: the predicate is now always invoked sequentially, even when an executor is configured.
1 parent 6988eb0 commit 3e4a179

2 files changed

Lines changed: 248 additions & 342 deletions

File tree

0 commit comments

Comments
 (0)