feat(idx): consult the hash index from vector-needle find / in / dict at, admit STR, carry across concat - #520
Merged
Merged
Conversation
… at, admit STR to it, carry it across concat Closes #519. A keyed lookup against a large vector rebuilt a hash over the whole vector on every call, and nothing attachable changed that: * find / in with a vector of needles hashed one operand per call even when the haystack carried a hash index; the index was only consulted for a scalar integer needle. * .attr.set 'unique / 'grouped refused RAY_STR, so a content-keyed column (a digest) could not carry an index at all. * concat answered a plain vector, so an append-only keyed vector lost its index on every append and re-attaching was a full build. * (at dict keys) ran one linear scan of the keys per probe. idxop: ray_index_find_atom / ray_index_find_vec probe a fresh hash index per needle (integer family by value, SYM through the column's domain, STR by the needle's bytes). ray_index_attach_hash admits RAY_STR: the key word is a 64-bit hash of the bytes and every word hit in the builder and the probes is confirmed against the payload of the group's first row, so a word collision is two groups, not one. The int64-keyed consults (eq/in rowsel, hash_group, find_row) decline a STR column instead of reading an out-of-range key as "provably absent". ray_index_carry_append copies the CSR tables onto a concat result and adds each appended row as a new group when every appended key is new (markers and the within-group order symbol carry); a repeated key, a null, a float column or a domain change leave the result unindexed as before, so an insert loop with repeating keys never pays a hidden rebuild. vec_all_distinct and attr_set_unique take SYM and STR too. collection: find and in try the index first; without one, when there are fewer needles than rows they hash the needles and scan the haystack once (early exit once every distinct needle is seen) instead of hashing the big side — the workaround the issue described, now the default orientation. (at dict typed-keys) resolves all positions with one find. dict.c consults the index for a scalar key. builtins: concat (vec+vec, vec+atom) carries. store: a STR hash persists through a splayed save verbatim (byte hashes and row ids are domain-free) in the column's single index slot; ray_col_save no longer stamps the inline-index marker on a STR column, which would have made the later append refuse the file. Measured, release, 362k keys, 100 needles: STR find 22 -> 1 ms with 'grouped (15 ms with no index at all); dict at 100 probes 139 -> 1 ms; I64 / SYM find 8 / 9 -> 0 ms; concat of 100 rows onto an indexed 362k I64 vector 4 ms with the index kept, where the re-attach cost 7 ms (STR: 9 vs 26 ms). Note 'unique alone is a marker with no table; 'grouped attaches the hash the lookups consult. Tests: test/rfl/ops/idx_find_in.rfl (drop-differential for I64 / SYM same and file domain / STR, duplicates, misses, null needles, the small-side orientation, dict lookups, concat carry incl. table growth and the repeat / null / STR / SYM cases, splayed round trips). Two existing expectations updated: .idx.hash on STR now succeeds; concat of new keys keeps the index. Claude-Session: https://claude.ai/code/session_017fCcYRuzNagw6nmLqZ9zbU
…rflow (cppcheck) Claude-Session: https://claude.ai/code/session_017fCcYRuzNagw6nmLqZ9zbU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #519.
A keyed lookup against a large vector rebuilt a hash over the whole vector on every call, and nothing attachable changed that. Three changes, matching the issue's three asks, plus the dict lookup it mentions.
What changes
1.
find/inwith a vector of needles consult the attached hash index.ray_index_find_vecprobes a fresh hash index once per needle: integer family by value, SYM through the column's domain (same or file domain), STR by the needle's bytes. Scalarfindand the dict scalar lookup go throughray_index_find_atom, which extends the old integer-only scalar path to SYM and STR.(at dict keys)with a typed key vector resolves every position with onefindinstead of a scan per probe.Without an index,
findand the hashset fallback ofinnow hash whichever side is smaller and scan the other once (early exit when every distinct needle has been seen). That is the orientation trick from the issue's workaround, made the default.2.
RAY_STRis admitted to the hash attach ('grouped,'unique,.idx.hash). The key word is a 64-bit hash of the bytes; every word hit in the builder and in the probes is confirmed against the payload of the group's first row, so a word collision becomes two groups rather than one merged group. The int64-keyed consults (eq/in rowsel,ray_index_hash_group,ray_index_find_row) decline a STR column instead of reading an out-of-range key as "provably absent". A STR hash persists through a splayed save verbatim (byte hashes and row ids are domain-free) in the column's single index slot, replacing the automatic string dictionary for that column.3.
concatcarries the index when every appended key is new.ray_index_carry_appendcopies the CSR tables onto the concat result and adds each appended row as a single-row group;uniqueand the within-group order symbol carry over. A repeated key, a null, a float column or a SYM domain change leave the result a plain vector as before, so an insert loop with repeating keys never pays a hidden O(n) rebuild. Applies to(concat v rows),(concat v atom)and therefore(alter 'v concat ...).Numbers (release, 362k keys, 100 needles, same box)
find,'groupedat, 100 probesfind,'groupedOne clarification for the reporter:
'uniquealone attaches a marker with no table, so the issue's "I64 with 'unique" cells never had an index to consult.'grouped(optionally with'uniqueon top) attaches the hash these paths use. Docs updated accordingly.Tests
test/rfl/ops/idx_find_in.rfl: drop-differential for I64, SYM (same and file domain), STR; duplicates, misses, null needles; the small-side orientation; dict lookups; concat carry including bucket-table growth, repeat / null / STR / SYM cases; splayed round trips of STR and carried indexes. Two existing expectations updated:.idx.hashon STR now succeeds; concat of new keys keeps the index. Full suite: 3784/3784, no UBSan output.Not in this PR
(== strcol "x")in awherestill scans a STR hash (the filter decode is int-keyed).'unique-only marker on a STR column does not persist (numeric ones do).https://claude.ai/code/session_017fCcYRuzNagw6nmLqZ9zbU