Land cross/outer pairwise axes + LoopIR optimizations; release 0.0.40 - #34
Merged
Conversation
Streaming exports now reuse the target's storage end to end instead of clearing and re-pushing: - Returned arrays are written by cursor and truncated, never cleared upfront. Record elements and nested row arrays are mutated in place across calls, so steady-state frames allocate nothing. - Intermediate scratch arrays persist at module scope when their identity cannot escape (init at loop depth 0, never pushed, not read by length/shift), eliminating per-call backing-store regrowth. - Typed-array targets (Float32Array, Float64Array, ...) are supported for scalar array outputs, with a RangeError on overflow. Previously they were silently treated as object targets and left zeroed. - Targets that alias an input array now throw a TypeError instead of silently computing over zero elements (the old length = 0 reset cleared the input itself in feedback loops). Measured on simulation schemas (node 22): 5x faster frames and ~70x less GC time at 100k array elements, 1.7x on a 144x144 two-field reaction-diffusion grid, with byte-identical outputs over hundreds of feedback steps. Also fixes Ruby DSL let, broken since 0.0.37: inline: true leaked into the ValueDeclaration struct (ArgumentError: struct size differs) instead of flowing through hints. Bumps dev kumi-parser to 0.0.31 so text-frontend specs can parse codegen hints.
The LoopIR pipeline existed but ran no passes; the lowerer emits one loop per statement group, so loop-invariant scalar work between two vector statements split what could be a single traversal, and the intermediate vectors crossing the split were materialized as arrays. LoopFusion merges sibling loops over the same axis and source register, hoisting the independent scalar barrier above the fused loop and renaming the second loop's element/index registers. Fusion is blocked when the barrier depends on anything the first loop defines, pushes to, or accumulates, and when the second loop touches a first-loop array through anything other than index_read at its own loop index — shifts, lengths, and re-iteration keep their two-pass structure, so stencil schemas are untouched. ArrayContraction then removes arrays with exactly one push that are only read back at the same index in the same loop, rewiring readers to the pushed scalar. Both backends benefit since the passes run before codegen. Generated code shrinks ~2x on multi-statement schemas (us_tax_2024 fed_eff: two loops + intermediate array -> one fused loop). Streaming JS on the orbital simulation reaches hand-written parity: 100k records per frame in 2.57ms vs 2.58ms for a hand kernel with the same output contract (19.6ms before the streaming rewrite, 3.7ms after it, 2.6ms with these passes). Outputs verified byte-identical across 500 feedback frames, 42 golden schemas re-executed on both backends, snapshots regenerated.
Adds core.sqrt/core.sin/core.cos as data-only elementwise functions (spec + ruby/js kernels), verified identical on both targets. Unblocks trig/wave math in the kumi-play examples. Also adds docs/pairwise-design.md: decomposable pairwise already works via whole-axis reduce + broadcast; true N-body needs a new cross-axis op that mints a fresh axis token over an existing carrier.
`cross(v)` (a.k.a. `fn(:cross, v)`) re-exposes an array's carrier under a fresh, independent axis, so an array can be combined with itself into a rank-2 (i × j) intermediate and reduced back. This is the broadcast-dual of a reduction and makes non-decomposable pairwise math — true N-body, all-pairs forces — expressible for the first time. Implementation mirrors `shift`/`axis_shift`, threading a new `axis_cross` op through the pipeline: - DSL: `cross` in schema_builder + DSL_METHODS; registered as a unary function - analyzer: mints child axis token `<src>__x`, records carrier alias in :cross_axes; anchor pass drops cross tokens when resolving carriers - DFIR/VecIR: new AxisCross op, builders, validators, instruction cloners - Loop lowering: cross axis opens a second loop over the same carrier with an independent index; chain_read redirects the source-axis read to it Broadcasting and suffix-reduce absorb the new axis with no changes. Codegen needs no changes — the op fully lowers to nested loops. Verified Ruby + JS bit-identical on 1-D N-body; golden `pairwise_cross` passes both targets. Also adds elementwise sqrt/sin/cos (see prior commit) usage to SYNTAX.md.
0.0.32 recognizes cross(...) sugar in the text frontend, so the golden and docs now use the clean cross(expr) form. Adds a SYNTAX.md perf note: return multiple pairwise results from one object value to share the O(n^2) nest. No new optimization pass is needed for cross — it rides the axis_shift rails, so CSE/DCE/loop-fusion/load-dedup all handle it for free (verified: repeated cross of the same field dedups within a function; object-return fuses ax/ay into a single nest with shared accumulators).
Two fixes prompted by testing cross() inside an imported schema:
1. cross-through-import was broken. The cross-axis carrier was resolved from
an analyzer side-table (:cross_axes) that only carried the CALLER's crosses,
so a cross minted inside an imported (inlined) schema failed loop lowering
with "no carrier for axis :vals__x". Loop lowering now derives the cross
carrier from the axis_cross op's own source_axis attribute, which is present
regardless of origin — local and imported crosses are handled identically.
Locked in by golden `cross_import` (imports Kumi::TestSharedSchemas::Pairwise,
which uses cross internally) — verified Ruby + JS produce [4, 1, -5].
2. Import inlining no longer fails silently. inline_import had ~8 bare `return
nil` bail points that left an import_call in place with no signal — a lost
fusion (and slower path) the user could never see. Now:
- benign skips (callee unresolved / not self-contained / arity mismatch)
log their reason under KUMI_DEBUG_IMPORT_INLINING=1 and return nil;
- post-commit failures (unmappable use, missing arg, no final reg) raise a
clear SemanticError instead of dropping to a slow path that would also be
broken — surfacing the real IR bug at compile time.
`cross` self-joins one array (A × A') for all-pairs within it (e.g. N-body). `outer` is its cross-array sibling: it re-exposes a value from a DIFFERENT array as a fresh inner axis, so two distinct arrays can be paired all-pairs. `pixel_x - outer(light_x)` builds the (pixels × lights) grid; `fn(:sum, ...)` reduces the light axis back to per-pixel — the pattern a software rasterizer needs (every pixel reduces over every light). Mirrors the `cross` machinery across all layers: - DSL `outer(v)` + `outer` registry entry (stencil.yaml) - dimensional analyzer: `analyze_outer` mints a free `__o` axis alias and tags it; `lub_by_prefix` gains `merge_with_outer_axes` so an outer scope attaches as an inner axis against the surrounding bound scope - DF/Vec ops `AxisOuter` + builders/cloners/validators - DF `align_axes` accepts a suffix broadcast (adding an OUTER axis), not just a prefix one - Loop lower: `:outer` axis re-iterates the other root array's carrier as a fresh inner loop; read lazily at use so it pairs at the full nest - AttachAnchors handles purely-outer decls Ruby and JS bit-identical. cross unchanged (golden still green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two silent failure modes that masqueraded as compiler bugs (both hit while building `outer`): 1. InstructionCloner (DF and Vec) had an `else => instr` fallthrough: an opcode with no clone branch was returned UNCHANGED, keeping its original inputs. Any pass that remaps registers (dedup/inlining/CSE) then left a dangling reference and produced a wrong result with no error. Now raises ArgumentError naming the opcode — every opcode must have a clone branch. 2. The JIT compile cache keyed generated code on the SCHEMA digest only, not the compiler version. Editing the compiler and re-running silently reused stale generated code from the cache dir. Added Configuration#code_version (hashes the gem's Ruby sources by mtime; overridable via KUMI_CODE_VERSION or the setter) and folded it into the cache filename, so a compiler change invalidates the cache automatically. Also upgraded two bare/vague raises to clear typed SemanticErrors: the axis-merge failure now distinguishes an `outer` mismatch from a plain tree/prefix mismatch, and DF align_axes explains the prefix-vs-suffix rule. Specs added for both guards. No behavior change on valid schemas (933 -> still green; the lone pre-existing agg.any dup failure is unrelated). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
exp/log/tanh are elementwise transcendentals (Math.exp/log/tanh in both Ruby and JS — bit-identical). They unlock real activation functions: sigmoid and tanh compose directly, enabling neural nets. Two goldens demonstrate the payoff: - transcendentals: exp/log/tanh + a sigmoid built from exp. - mlp_backprop: a 2-H-1 net (tanh hidden, sigmoid output, BCE loss) whose ENTIRE forward + backward pass — every gradient, including the (hidden x inputs) weight-gradient matrix — is expressed in one schema. A harness applies the gradients and loops; the net learns XOR (verified separately), which a linear model cannot. Kumi has no autodiff: gradients are hand-derived but evaluated by the compiled schema, bit-identical on Ruby and JS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v2 (IR + codegen text snapshots) was meant to replace v1 (execute Ruby+JS, compare hand-written expected.json), but v1 was still the CI gate and provided runtime + Ruby/JS-parity coverage v2 lacked. Port that into v2, then delete v1. - New `runtime` representation (lib/kumi/dev/golden_runtime.rb): executes the generated Ruby AND JS against input.json, snapshots outputs to expected/runtime.json, and asserts Ruby == JS bit-identical. Parity is skipped (with a recorded note) for `import` schemas (need shared modules) and `to_decimal` schemas (Ruby BigDecimal vs JS float genuinely diverge) — Ruby outputs are still snapshotted for regression detection. - golden_v2 verify treats a representation that legitimately yields no output (e.g. runtime with no input.json) as skipped, not failed. - CI runs `golden_v2 verify` (+ setup-node); `bin/kumi golden` is now an alias for v2. precompile_schemas! moved to GoldenV2. - Deleted lib/kumi/dev/golden.rb, lib/kumi/dev/golden/**, and the orphaned hand-written golden/*/expected.json (superseded by expected/runtime.json). Full suite green (1 pre-existing unrelated agg.any dup fails on main too); golden_v2 verify: 49/49 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
load_functions globs its directory recursively, but the backward-compat spec pointed it at the shared system temp dir (File.dirname of a Tempfile). That picked up unrelated YAML written by other tempfiles during the run, tripping the duplicate-function-id guard (`agg.any`). Use Dir.mktmpdir so the glob only sees this test's fixture. Full suite now green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This checkout's uncommitted work — never committed from earlier sessions — is now recorded on top of the merged outer-axis work: - pass_manager: per-pass wall-clock compile budget (PassBudgetError + DEFAULT_ PASS_BUDGET_MS, KUMI_PASS_BUDGET_MS override) so a runaway pass fails with a located error instead of hanging. Spec in pass_manager_spec. - js loop emitter + pretty_printer: streaming / typed-array codegen handling (reuse record elements, truncate streaming outputs); schema_spec covers it. Plus post-merge tidy of the outer-axis files: drop a now-dead incompatible_axes_message (superseded by the outer-aware axis_merge_error), move the cloner spec under passes/support/, and clear rubocop (autocorrect + ratchet pass_manager/import_inlining length, drop the stale golden/reporter todo entry now that v1 is gone). Full suite green (946 ex, 0 failures); golden_v2 verify 50/50; rubocop clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- outer(...) usable from text schemas (kumi-parser >= 0.0.33); outer_let golden covers the text path with Ruby/JS parity. - Fix outer/cross value used through a let on the inner pairing axis reading nil past the inner array length; materialized read now matches by axis (Ruby + JS). - Array input arity errors explain the element rule and show the single-child form. - Expand docs/SYNTAX.md and add docs/INPUTS.md input-shape reference.
Root#digest folded RUBY_VERSION into the compiled module name, so the schema_ruby.rb goldens embedded a per-Ruby hash and could not pass the strict golden_v2 verify across the 3.1-3.4 CI matrix (and broke on the 0.0.40 bump). The generated code is plain Ruby with identical semantics across supported Rubies, so drop RUBY_VERSION from the digest and regenerate the codegen goldens for 0.0.40.
Removing RUBY_VERSION from the digest was not enough: the AST was folded in via
"#{self}-#{hints.inspect}", and Struct#to_s / Hash#inspect changed their
rendering in Ruby 3.4 ({:a=>1} -> {a: 1}). Schemas whose AST/hints contained
hashes or nested structs (imports, shift/roll stencils) therefore hashed
differently across the CI Ruby matrix, failing strict golden_v2 verify.
Serialize the AST through a format-pinned stable_encode encoder that does not
rely on inspect/to_s, and regenerate the codegen goldens.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the pairwise axis work (
cross/outerall-pairs primitives), the LoopIR optimization pipeline (loop fusion + array contraction), streaming JS codegen, a pass-conventions refactor (contract DSL + parameterized IR passes), and golden-suite consolidation onto v2. Cuts release 0.0.40.This branch already contains everything from
outer-axis(it's an ancestor), so this is the single PR that brings all of it ontomain.Highlights
Language / semantics
cross— all-pairs / self-join axis primitive (A × A'), the basis for N-body and pairwise math.outer— all-pairs over two different arrays (A × B). Now usable from text schemas too (requires kumi-parser ≥ 0.0.33). Goldenouter_letcovers the text path with Ruby/JS parity.crossthrough an import boundary; inliner skips are now observable.outer/crossvalue used through aletthat lives purely on the inner pairing axis readingnilpast the inner array length — the materialized read now matches by axis, not positional loop depth (Ruby + JS).sqrt,sin,cos,exp,log,tanh.Compiler backend
Pass conventions (refactor)
reads/writescontract DSL onPassBase, enforced inPassManager; contracts declared across DEFAULT/lowering/codegen passes.IRLowerPass/IRValidatePass.Passsuffix; NAST children protocol for traversal; pipeline contract self-check spec.docs/PASSES.md; rubocop ratchet scoped to pass dirs.Testing / tooling
transcendentals, MLP backprop,outer_let.Docs
docs/SYNTAX.md; newdocs/INPUTS.mdinput-shape reference.rake docs:portal) and the kumi-docs site — intentionally not in this PR.Release
[Unreleased]rolled into a dated0.0.40section,Gemfile.lockre-locked.Verification
bundle exec rspec→ 947 examples, 0 failures, 3 pending.bundle exec bin/kumi golden_v2 verify→ all schemas pass (IR + codegen snapshots, Ruby/JS parity).