Repository navigation
Commit 3e4a179
committed
refactor(manifest): unify streaming/eager and executor/no-executor planning paths in ManifestGroup
Extract shared manifest/residual evaluator caching and entry-filtering
rules (ignore_existing_, file filter, custom predicate) into a single
FilePlanningStream::Evaluators helper, and collapse the previously
separate sequential vs. executor-backed state machines in
FilePlanningStream into one FilePlanningStream::EntryStream, with
batch size 1 without an executor and kManifestReadBatchSize with one.
ReadEntries()/Plan()/Entries() are rebuilt on top of the same
EntryStream instead of duplicating manifest iteration and filtering
via a separate ParallelCollect-based loop, removing the FIXME about
porting Iceberg Java's ParallelIterable design.
Also:
- Dedupe data manifests (matching Java's ManifestGroup, which stores
them in a Set) so a manifest referenced by overlapping snapshots is
only scanned once.
- Update FilterManifestEntries() doc comment: the predicate is now
always invoked sequentially, even when an executor is configured.1 parent 6988eb0 commit 3e4a179
2 files changed
Lines changed: 248 additions & 342 deletions
0 commit comments