[core] Drop concurrently-built global indexes at materialize-deletion compaction - #8947
Merged
JingsongLi merged 1 commit intoJul 31, 2026
Conversation
… compaction DataEvolutionCompactGlobalIndexDropper scanned the compaction's base snapshot for stale global indexes. A global index built concurrently and committed after the compaction planned still encodes pre-materialization row-ids, but was not seen at the base snapshot and so survived, letting a later vector search resolve those row-ids against renumbered data. Scan the latest snapshot instead so such concurrently-committed indexes are also dropped.
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
DataEvolutionCompactGlobalIndexDropperdrops global indexes that amaterialize-deletioncompaction invalidates (the compaction renumbers row-ids,so pre-materialization indexes become stale). It scanned the compaction's base
snapshot for indexes to drop. A global index built concurrently and committed
after the compaction planned still encodes pre-materialization row-ids, but is
not visible at the base snapshot, so it survives — and a later vector search
resolves its stale row-ids against renumbered data.
Latent on master today (index build on DV tables is currently blocked); reachable
once Lumina indexes on DV tables are allowed. Self-contained; submitted
independently.
Change
Scan the latest snapshot for stale indexes instead of the compaction base, so
concurrently-committed indexes are also dropped (falling back to the base snapshot
when no newer one exists).