Skip to content

Support LEFT/RIGHT JOIN in the DuckDB IEJoin dialect by decomposing the outer join into INNER pairs plus unmatched rows — Closes #95 - #223

Merged
conradbzura merged 5 commits into
mainfrom
95-decompose-outer-join-intersects
Sep 2, 2026
Merged

Support LEFT/RIGHT JOIN in the DuckDB IEJoin dialect by decomposing the outer join into INNER pairs plus unmatched rows — Closes #95#223
conradbzura merged 5 commits into
mainfrom
95-decompose-outer-join-intersects

Conversation

@conradbzura

@conradbzura conradbzura commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Accelerate LEFT / RIGHT JOIN on a column-to-column INTERSECTS by decomposing the outer join rather than emitting one. Both halves reach DuckDB's IE_JOIN, where the shape previously declined to the naive overlap predicate — a hash join on a 24-value chromosome key with the position inequalities as a residual filter, quadratic and unable to finish at a million intervals per side. At 2^20 the decomposition returns 1,334,564 rows in 0.78s; at 2^22 it returns 7,758,528 rows in 2.0s, scaling linearly.

A LEFT JOIN is exactly the INNER pairs unioned with the row-preserving side's unmatched rows, NULL-filled on the other side. Both halves already had fast paths, so this emits them and unions the result instead of hoping the planner chooses well for an outer join. RIGHT is the mirror, reached by swapping the FROM and joined tables.

The issue originally prescribed emitting a per-chromosome LEFT JOIN. That design was benchmarked and rejected: it is faster below ~1e5 rows and then collapses, failing to finish within 300s at 2^22. EXPLAIN reports IE_JOIN for it throughout, so plan inspection cannot distinguish the two designs and only execution at scale does — which is why the test suite includes a scale test rather than relying on plan assertions.

The trade-off is two passes over the data, making the decomposition roughly 2x slower than a per-chromosome outer join at toy sizes. Predictable linear scaling is worth that.

Two rounds of independent review shaped what landed. The first found five defects a green suite passed straight through, each a shape that previously declined to the naive plan and answered correctly. The second found one behavioral defect — the rewrite renamed duplicate output columns — plus a regression lock that never reached the gate it was written to protect: deleting that gate left all 2131 tests passing while the dialect answered a semi join as an outer join. Roughly 1,900 randomized differential cases across seven independent fuzz runs found no row or schema divergence, so the correctness work here is in the gates and the tests rather than in the plan.

bedtools intersect -wao remains on the naive plan: its CASE projection is blocked by the projection gate independently of the outer join, and unblocks with #109.

Closes #95
Closes #226

Proposed changes

Return the IEJoin setup and SELECT separately

The dialect emitted a multi-statement script, and any rewrite composing on top recovered the pieces by splitting the rendered string on the statement separator. An identifier containing that sequence splits the script inside a quoted alias and the query no longer parses. _build_sql now returns (setup, select), transform_to_sql is a thin joiner, and composing builders consume the parts directly. This also repairs the same latent defect in the shipped count_overlaps path.

Decompose the outer join

_match_outer_join_decomposition claims LEFT/RIGHT shapes whose projections are side-attributable columns; _build_outer_join_parts emits the matched half, the unmatched half, and the union. The halves partition chromosomes differently by design — the matched half intersects both sides, the unmatched half enumerates the preserved side alone — which is what lets rows on a chromosome the other table lacks still surface.

Dispatch sits after the count_overlaps matcher, which keeps its faster zero-fill path, and before the existing outer-join decline, which still catches FULL OUTER and the WHERE-INTERSECTS shape.

Preserve rows whose chromosome is NULL

Such rows can never match, and neither half surfaces them on its own: both partitions come from SELECT DISTINCT over the chromosome, where a NULL renders as a NULL literal that string_agg skips, so no branch is emitted for it. They are unioned in directly. The root cause is shared with the standalone ANTI path, which has dropped these rows since #208 and is filed separately; what this fixes is LEFT/RIGHT inheriting it instead of declining safely.

Preserve duplicate output column names

The matched half is a UNION ALL branch directly rather than a derived table. SELECT * over a subquery makes DuckDB de-duplicate repeated output names, and that renaming becomes the union's schema, so the canonical bedtools projection — a.chrom, a.start, a.end, b.chrom, b.start, b.end — came back as chrom, start, end, chrom_1, start_1, end_1 under dialect="duckdb" and unchanged under every other dialect. A flag documented as a performance opt-in must not alter the result schema.

Gate on a whitelist rather than a blocklist

The rewrite re-emits the query as a union of two independently transpiled halves, so any top-level clause it does not itself read would be applied per half instead of over the union — LIMIT 2 returning four rows, QUALIFY silently dropped. Rejecting anything outside the set the builder consumes forecloses the class rather than the instance. TABLESAMPLE is rejected separately since it rides on the table node, not the top-level SELECT.

Decline case-insensitively colliding output names

DuckDB resolves identifiers case-insensitively even when quoted, so AS x alongside AS X bound both positions to the first column, returning the wrong value and widening that column to VARCHAR for the matched rows too. The uniqueness gate now case-folds through _normalize_alias.

Share one gate prelude between both matchers

_match_count_overlaps and _match_outer_join_decomposition opened with near-identical preludes differing only in the accepted join side. _resolve_intersects_join now performs those checks once and returns the resolved join. The duplication had a concrete cost: the untested-gate defect below existed in both copies, so it had to be found twice.

Mark the bedtools integration modules

pytest does not honour pytestmark declared in a conftest, so the entire bedtools oracle lane was invisible to marker-based selection — pytest -m integration collected 250 of 333 tests and -m "not integration" ran the binary-dependent lane it exists to skip. Each module now declares the marker, matching the datafusion lane. Collection is 335 of 335.

Correct stale documentation

The public transpile docstring listed LEFT/RIGHT among the shapes the dialect declines. The README, spatial-operators page, and performance guide carried the same INNER/SEMI/ANTI-only claim, as did the canonical grammar line on the page describing the feature, the registry entry point's decline list, and the naive-predicate module documenting when it is the fallback.

The performance guide now documents the third UNION ALL branch, which is load-bearing correctness the previous text omitted, and completes the decline list with self-joins, TABLESAMPLE, and case-only name collisions. Both documented -loj recipes project a star and therefore decline, so the migration guide says so rather than leaving users to conclude the flag does nothing.

Two execution details also changed: the emitted script is no longer always two statements, and a decomposed query declares one session variable per half.

Test cases

# Test Suite Given When Then Coverage Target
1 TestTranspileDuckDBIEJoinOuterJoinDecomposition A LEFT-join INTERSECTS with matched, unmatched, and other-chromosome rows The query is transpiled with dialect="duckdb" and executed Rows equal the Python LEFT-join reference Core semantics
2 TestTranspileDuckDBIEJoinOuterJoinDecomposition Duplicate rows on both sides The decomposed query runs Row multiplicity matches the reference exactly Union multiplicity
3 TestTranspileDuckDBIEJoinOuterJoinDecomposition A RIGHT-join INTERSECTS with unmatched right rows The query is transpiled and executed Rows equal the mirrored reference FROM/join swap
4 TestTranspileDuckDBIEJoinSQLStructure A RIGHT-join INTERSECTS The query is transpiled The unmatched half partitions on the preserved side alone Swap emission
5 TestTranspileDuckDBIEJoinOuterJoinDecomposition A projection whose six columns carry three output names The query is transpiled and executed Column labels match the naive plan rather than being renumbered Duplicate output names
6 TestTranspileDuckDBIEJoinOuterJoinDecomposition A NULL-fill alias shadowing a preserved output name ahead of it The query is transpiled and executed The bare reference binds the relation column, not the lateral NULL alias Name-resolution near-miss
7 TestTranspileDuckDBIEJoinOuterJoinDecomposition Preserved output names differing only by letter case The query is transpiled and executed It declines and matches the naive plan in rows and result schema Case-insensitive collision
8 TestTranspileDuckDBIEJoinOuterJoinDecomposition A preserved side containing NULL chromosomes, including all-NULL The query is transpiled and executed Every NULL-chromosome row surfaces NULL-filled NULL chromosome preservation
9 TestTranspileDuckDBIEJoinOuterJoinDecomposition A top-level QUALIFY, LIMIT, OFFSET, GROUP BY, or ORDER BY The query is transpiled and executed It declines and matches the naive plan Clause whitelist
10 TestTranspileDuckDBIEJoinOuterJoinDecomposition A TABLESAMPLE on the FROM table or on the joined table The query is transpiled and executed It declines and remains executable Sampled operand, gated per side
11 TestTranspileDuckDBIEJoinOuterJoinDecomposition An output alias containing the statement separator The decomposition and count_overlaps queries run Both execute and match the naive plan Structural parts seam
12 TestTranspileDuckDBIEJoinOuterJoinDecomposition Colliding preserved names, a WHERE, a star, a self-join, an ON residual, a repeated INTERSECTS, or a subquery operand The query is transpiled and executed It emits no session variable and matches the naive plan Decline as one unit
13 TestTranspileDuckDBIEJoinOuterJoinDecomposition A LEFT SEMI or LEFT ANTI join, which parses with side='LEFT' and reaches the kind gate The query is transpiled At most one session variable is declared and no unmatched half is emitted Kind gate
14 TestTranspileDuckDBIEJoinOuterJoinDecomposition A bare SEMI or ANTI join, which parses with no side and is rejected a gate earlier The query is transpiled Exactly one session variable is declared and no unmatched half is emitted Side gate
15 TestTranspileDuckDBIEJoinOuterJoinDecomposition Tables configured with a custom chromosome column and an other-side-only projection The query is transpiled and executed The configured column drives the unmatched half Table config resolution
16 TestTranspileDuckDBIEJoinOuterJoinDecomposition 20,000 rows per side with a share unmatched Each session variable is planned separately Both halves plan through IE_JOIN and neither through BLOCKWISE_NL_JOIN Both halves reach the fast operator
17 TestTranspileDuckDBIEJoinOuterJoinDecomposition Two decomposed queries sharing one connection Their setup statements are interleaved before either SELECT Each returns its own rows Session-variable isolation
18 TestTranspileDuckDBIEJoinOuterJoinDecomposition Empty, half-empty, and chromosome-disjoint tables The query is transpiled and executed Rows and result schema match the naive plan NULL-fill typing
19 TestTranspileDuckDBIEJoinOuterJoinDecomposition 262,144 rows per side with a sixth unmatched The decomposed query runs Every left row survives as a distinct key, the unmatched half contributes, and it finishes inside the bound Scale, which plan assertions cannot cover
20 TestTranspileDuckDBIEJoinOuterJoinDecomposition Hypothesis-generated intervals including NULL chromosomes and zero-length spans The query is transpiled and executed Rows equal the Python reference as a multiset Randomized correctness
21 tests/integration/bedtools/test_intersect.py Interval sets with an unmatched row and a chromosome absent from B The decomposed query is compared to bedtools intersect -loj Output matches exactly Real-oracle agreement
22 tests/integration/bedtools/test_intersect.py A RIGHT join against the same inputs exchanged The decomposed query is compared to bedtools intersect -loj Output matches exactly RIGHT against a real oracle
23 tests/integration/bedtools/test_intersect.py Byte-identical duplicate rows on both sides The decomposed query is compared to bedtools intersect -loj Output matches exactly Multiplicity against a real oracle
24 tests/integration/bedtools/test_intersect_property.py Hypothesis-generated interval sets A non-DISTINCT LEFT join is transpiled with dialect="duckdb" The fast path fires and output matches bedtools -loj Randomized oracle agreement
25 tests/integration/datafusion/test_cross_target_oracle.py The same query across generic, datafusion, and duckdb targets Each target runs it All agree and the duckdb target is asserted to have decomposed Cross-target equivalence

@conradbzura conradbzura self-assigned this Aug 21, 2026
@conradbzura
conradbzura force-pushed the 95-decompose-outer-join-intersects branch from a300f35 to ee6ab6e Compare August 24, 2026 13:34
@conradbzura conradbzura mentioned this pull request Aug 31, 2026
Open
@conradbzura
conradbzura force-pushed the 95-decompose-outer-join-intersects branch from ee6ab6e to 123058a Compare August 31, 2026 18:15
@conradbzura
fix: Make Table immutable so its validation cannot be bypassed
0d1f838 
Table validated coordinate_system and interval_type in __post_init__ but
was a plain dataclass, so assigning either field afterwards installed a
value the constructor rejects. The value is not inert: it selects the
coordinate translation the emitted SQL performs. Setting it to a third
string produced an ON predicate that is neither the 0-based form nor the
1-based one but a half-shifted hybrid matching no coordinate system, and
transpile raised nothing.

Freezing makes __post_init__ the single way in, so validation holds for
the object's whole lifetime rather than only at construction. Callers
vary a config with dataclasses.replace, which re-runs that validation.
Immutability also restores __hash__, which the generated __eq__ had set
to None.

Separately, _build_tables duck-typed every non-str entry for a .name
attribute. An arbitrary object therefore reached pass 1 and failed there
with an AttributeError naming an internal column attribute, telling the
caller nothing about which argument was wrong. It now rejects the entry
with a TypeError naming the offending type.

Adds tests for the two field validations, which had none.
@conradbzura
feat: Accept a Target instance as the dialect argument
5280fcd 
Target, Capabilities, GenericTarget, DuckDBTarget and DataFusionTarget
are all exported from the package root and autodocumented, yet the only
public function that consumes a target rejected every one of them and
reported the object's repr as though it were a misspelled name.

Selecting a target by name stays the documented default and is the right
seam for the plugin-distribution case, where a package ships a target and
users select it without importing it. What that does not cover is the
one-off: making a bespoke Target selectable meant mutating the
process-global registry, and the registration outlived the call. The
object path removes that side effect and is purely additive.

GenericTarget is now accepted as an instance while the name "generic"
still raises. That asymmetry is deliberate: None remains the one public
spelling for the generic target, but an instance is unambiguous.

Also drops the three transpile overloads. All three returned str and the
widest admitted a bare str, so the set collapsed to the implementation
signature and taught a type checker nothing. Their stated purpose was
editor completion of the built-in dialect names, which the DialectName
literal alias in the signature preserves without three public typing
artifacts.
@conradbzura
feat: Decompose LEFT and RIGHT INTERSECTS joins into two IEJoin halves
bb253eb 
DuckDB's IE_JOIN is INNER-only, so a LEFT or RIGHT outer join carrying a
column-to-column INTERSECTS fell through to the naive predicate: a hash
join on chrom with the position inequalities as a residual filter, which
is quadratic when the chromosome key has low cardinality.

The query is now rewritten as a UNION ALL of an INNER half for the
matched pairs, a NOT EXISTS half for the preserved side's unmatched rows,
and a third branch for its NULL-chromosome rows. The first two reach the
fast operator; the third is a filtered scan, and it is load-bearing
rather than defensive, because both partitions come from SELECT DISTINCT
chrom where a NULL renders as a NULL literal that string_agg skips. RIGHT
is served by swapping the FROM and joined tables so one LEFT-shaped path
covers both. Shapes the rewrite cannot express decline as one unit.

Two properties of the emission decide whether it is worth taking, and
both are settled by execution rather than by plan inspection, which
reports IE_JOIN either way.

The first is contig cardinality. One UNION ALL branch is emitted per
distinct chromosome, so cost tracks that count while the plain
predicate's does not: measured at 262,144 rows per side, the partitioned
form runs 0.79s against 4.53s naive at 24 contigs and 57.2s against 0.11s
at 3,000. The partition's cardinality is a property of the data, so the
choice is made at execution time by a CASE over the partition's own row
count, above which the same query binds with the chromosome equality
inlined instead. The unmatched half also partitions on the chromosome
INTERSECT rather than the preserved side's distinct chromosomes, carrying
its left-only chromosomes in one non-partitioned branch: those rows
cannot match, so a branch apiece scans both tables to prove an emptiness
the partition already knows.

The second is session state. Each half declares a DuckDB session
variable, and the emitted script cannot release them because the final
statement has to be the SELECT. Names are therefore a digest of the
variable's own rendered value, which bounds a session's variable set by
the number of distinct query shapes rather than the number of calls;
naming them per call retained 26 to 84 MB after 50 to 200 queries.

The naive-predicate fallback resolves through the registry rather than
calling the built-in directly, so a user expander registered on
(GenericTarget, Intersects) reaches the shapes this target declines
instead of applying under dialect=None alone.
@conradbzura
test: Mark the bedtools integration modules and pin the duckdb plan
450f3a9 
The bedtools lane declared its integration marker in conftest.py, where
pytest does not honour pytestmark, leaving all of its tests unmarked. Both
documented selection commands therefore inverted: running with the
integration marker skipped the whole lane, and running without it pulled
in the lane and its bedtools and pybedtools dependencies. CI was
unaffected because it runs the suite with no marker filter, so nothing
surfaced it. Each module now declares the marker itself, matching the
datafusion lane's working convention.

The cross-target oracle asserted only that the three targets agree, which
it would continue to do if duckdb silently stopped decomposing. It now
also pins which plan duckdb took, through a target-to-SQL map the oracle
fixture exposes.

Adds bedtools oracles for the RIGHT outer join, expressed as the left
outer join with the operands swapped, and for duplicate input rows, the
multiplicity axis a UNION ALL rewrite is most likely to break and which
the property lane could not reach because it draws from unique inputs.
@conradbzura
docs: Document the decomposition, the ceiling and the dialect contract
0d05e49 
The dialect parameter promised that an unqualified projection raises at
transpile time. That holds for the INNER, SEMI and ANTI shapes and not
for the outer joins, which decline silently to the naive predicate
instead, so one stated rule covered two behaviours with no way for a
caller to tell which applied. The same paragraph carried an inline list
of declined shapes that had drifted behind the guide it points at, so the
list is gone and the pointer stays: a second copy of an enumeration only
drifts again.

Returns said the result was a SQL query. Under duckdb an accelerated join
returns a multi-statement script, and a driver that splits statements or
forwards only the last drops the variable the SELECT reads and yields
empty results.

The partition-count ceiling decides at execution time whether a query
takes the per-chromosome form at all, and it is the difference between
winning and losing by orders of magnitude on a scaffold-level assembly.
The performance guide now carries the measured crossover and the
reasoning behind the bound, rather than generalising from a single
low-contig measurement to a claim of large speedups at scale.

Two further claims are restated from measurements rather than intuition.
The NULL-chromosome branch, described as costing a linear scan of the
preserved table, is pruned by null statistics to a quarter of a percent
of runtime on a base table. The per-chromosome LEFT JOIN comparison
asserted an inflection near 1e5 rows that the recorded figures do not
support; it now states the two points that were actually measured.

Also documents what the session-variable token is, now that it addresses
the variable's content rather than being random, and gives
transform_to_sql its own contract in place of a description of its
sibling: the script shape, the ValueError it raises, and that the query
it is handed is never mutated.
@conradbzura
conradbzura force-pushed the 95-decompose-outer-join-intersects branch from 123058a to 0d05e49 Compare September 1, 2026 19:25
@conradbzura
conradbzura marked this pull request as ready for review September 2, 2026 15:10
@conradbzura
conradbzura merged commit a08cb0e into main Sep 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@conradbzura conradbzura

Labels

None yet

Projects

None yet

Milestone

No milestone

1 participant

@conradbzura