feat(eql): absorb the EQL subsystem, and kill the version skew - #885
Open
tobyhede wants to merge 1872 commits into
Open
feat(eql): absorb the EQL subsystem, and kill the version skew#885tobyhede wants to merge 1872 commits into
tobyhede wants to merge 1872 commits into
Conversation
Follow-up to #369. Doxygen drops the real name of `CREATE FUNCTION <schema>.<name>(... a <schema>.<domain> ...)` when an operand type is schema-qualified (e.g. `b public.text_ord`), leaving the schema as <name>. ~290 of these internal "Unsupported operator blocker" helpers surfaced as bogus functions named `eql_v3_internal`/`eql_v3`, mislabeled public (is_private keyed on <type>, but the schema landed in <name>) — inflating the public surface 696 -> 986 and rendering a bogus `eql_v3_internal` entry on the docs page. Skip them, keyed on <definition> (CREATE FUNCTION) rather than the brief: their brief reads "Unsupported operator blocker for ...", which the operator-symbol recovery would otherwise mis-match and remap to a junk name (`Unsupported`). Genuine CREATE OPERATORs (definition CREATE OPERATOR) are still recovered from the brief. Manifest: 1680 -> 1390 functions (696 public, 694 private); no schema-named entries; all real functions retained. Adds test_schema_name_misparse_is_skipped. Claude-Session: https://claude.ai/code/session_01CqDNqLSEEkCi7xAJFq7HJA
Client-side half of the EQL v3 query-term surface: the enveloped,
per-capability query operand `{v, i, <terms>}` (envelope minus the
ciphertext `c`) for every term-bearing scalar domain, plus its
conversion from the v2 payload.
Generator (eql-codegen/src/bindings.rs):
- render_query_struct: a `<Name>Query` twin per term-bearing domain =
the storage struct minus `c`, on `public.<name>_query`, with
deny_unknown_fields enforcing the no-`c` contract. Storage-only
domains (no operators) get no twin.
- render_query_payload_rs: `QueryPayload` is now catalog-generated (one
variant per twin + the SteVec needle), superseding the hand-written
single-variant enum — enveloped + per-capability makes it
catalog-per-domain.
- all_query(): a separate query inventory, kept OUT of all() so query
domains never resolve as stored from_v2 conversion targets.
Bindings (eql-bindings, regenerated + hand-written):
- Regenerated family files + query_payload.rs + inventory.rs.
- 38 TypeScript bindings (bindings/v3/*Query.ts) + 38 JSON Schemas
(schema/v3/*_query.json); export.rs chains all_query().
- from_v2::convert_scalar_query hoists the v2 payload's required terms
into `{v:3, i, <terms>}` (drops c/k; bf reinterpreted to smallint[]);
both query entry points route through QueryPayload::parse per target;
storage-only scalars stay UnsupportedQueryTarget.
Verified: `mise run test:crates` (fmt + clippy -D warnings + tests) green.
SQL surface (public.<name>_query domains + consuming operators/functions)
and sqlx conformance are NOT in this commit — they need Postgres + CS
creds to validate. See the PR description.
CIP-3432
…432)
The SQL half, part 1: a `public.<name>_query` domain per term-bearing
scalar domain — the index-terms-only twin (`{v, i, <terms>}`) that query
operators will consume. CHECK asserts the envelope-minus-`c` + terms and
FORBIDS `c` (a query operand carries no ciphertext), mirroring the Rust
binding's deny_unknown_fields.
- context.rs: DomainBlock gains `forbidden_keys`; `query_domain_block`
builds the twin (keys = v/i + terms, forbidden = [c]).
- templates/query_types.sql.j2: idempotent CREATE DOMAIN with the
no-`c` CHECK.
- generate.rs: `render_query_types_file` → `<T>_query_types.sql`, wired
into `render_type` for families with any term-bearing domain.
Validated: `mise run build` assembles; the full surface (38 query
domains) installs cleanly into a fresh PG (`test:clean_install_v3`); and
the CHECK semantics verified directly — `{v,i,hm}` accepted, a payload
with `c` or a missing term rejected.
Still to do: query extractors + wrappers + operators binding
(storage_domain, <name>_query), and sqlx conformance (needs CS creds).
CIP-3432
…ins (CIP-3432)
The SQL half, part 2: per term-bearing domain, a `<name>_query_functions.sql`
+ `<name>_query_operators.sql` giving query operands a public SQL entry point.
- `render_query_functions_file`: query-operand extractor OVERLOADS (the same
eq_term/ord_term, on `public.<name>_query`) + comparison WRAPPERS binding
`(storage, query)` and its `(query, storage)` commutator — supported
operators only, `extractor(a) <op> extractor(b)`, no ciphertext cast.
- `render_query_operators_file`: `CREATE OPERATOR` for each, both directions,
reusing the existing planner metadata (COMMUTATOR/NEGATOR/RESTRICT/JOIN).
- Wired into `render_type` for term-bearing domains; reuses the storage
`functions.sql` / `operators.sql` templates unchanged.
Semantically validated against REAL fixture ciphertext (local PG): a
term-only operand `{v,i,hm}` (no `c`) matches the stored row through
`= (public.integer_eq, public.integer_eq_query)` — self-match hit all 17
integer fixture rows, and a plaintext=0 operand matched exactly the
plaintext=0 row. Full surface installs clean (`test:clean_install_v3`).
Next: sqlx conformance test with fresh in-test encryption (increment 6).
CIP-3432
…(CIP-3432)
End-to-end proof of the query-operand surface with FRESH ZeroKMS
encryption for both stored values and the query value (gated behind
`proptest-e2e`, like the rest of the fresh-encryption suite):
- eq: a term-only operand `{v,i,hm}` (no `c`), INDEPENDENTLY encrypted,
matches exactly the equal stored rows via `= (integer_eq,
integer_eq_query)`; a never-stored value matches nothing.
- ord: a term-only ORE operand orders correctly (`< 25` → the rows below
25), and the `(query, storage)` commutator direction resolves too.
- security: a ciphertext-bearing operand (full storage payload) is
rejected by the `<name>_query` domain's no-`c` CHECK.
Verified locally against Postgres 17: 3 passed.
CIP-3432
…en (CIP-3432) Regenerated eql_v3_public_surface.txt — 423 additions, 0 removals, all `*_query` wrappers/extractors. Confirms the query surface is purely additive (no existing operator changed).
…CIP-3432) Extends the shared oracle engine so EVERY generated `<name>_query` domain is exercised against real ciphertext, across all scalar types — folded into existing round trips (no added DB load, no new flakiness): - assert_eq_oracle / assert_ord_oracle: each all-pairs SELECT now also drives the term-only query operand (payload minus `c`) through the `(storage, <name>_query)` operators, both directions. Covers <T>_eq_query, <T>_ord_query, <T>_ord_ore_query for all 9 families in BOTH the fixture and e2e (fresh-encryption) suites. - Overload::DomainQuery: a fourth named-function overload (RHS = <domain>_query), reaching text_search_query, which the operator oracle (text runs via _eq/_ord/_ord_ore) never touches. - assert_match_smoke: four query-operand containment rows cover text_match_query (`contains`/`contained_by` with a bloom needle). - ope_ord_fixture_smoke!: each range/equality predicate also runs against the term-only operand, covering <T>_ord_ope_query for all families. jsonb_query already has semantic coverage (v3_jsonb_tests D4). With the standalone fresh-encryption conformance, all 38 scalar query domains + jsonb_query are now tested. Full fixture oracle suite: 42 passed. CIP-3432
…-3432) The new (storage, <name>_query) + commutator operators add 2 arg shapes per operator, so every term-bearing domain now has ops×5 operators (3 storage + 2 query), not ops×3. CI's SQLx shards caught this. Verified: planner_metadata 48/48 pass.
…te (CIP-3432) - generate.rs: comment in render_query_functions_file explaining why query twins emit no blockers (the realistic col<op>operand path is covered by the storage domain's jsonb blockers; blocking operand<op>operand would be the full matrix for zero real-world coverage). [review finding #1] - query_types.sql.j2: @note that query operands must be cast to their _query domain in a predicate (uncast literal RHS is ambiguous with the jsonb overload). [review finding #3]
… non-superuser) CREATE OPERATOR FAMILY / CLASS require superuser, so the single-transaction eql_v3 installer aborted at the first CREATE OPERATOR FAMILY on Supabase and other managed Postgres (SQLSTATE 42501, "must be superuser"), leaving eql_v3 uninstallable there despite the surface being otherwise managed-Postgres compatible. Wrap both src/v3/sem/*/operator_class.sql family+class creations in a DO block that catches insufficient_privilege and continues with a NOTICE. One artifact now installs everywhere: superuser installs create the default btree opclass as before (self-managed PG, SQLx matrix); non-superuser installs skip it and fall back to the OPE ordering domains, whose extractor return types carry a native btree opclass. Non-privilege errors still propagate. Also removes stale in-file comments claiming these files were excluded by a `**/*operator_class.sql` build glob — the v3 build globs src/v3 wholesale. Verified: live Supabase (both skipped, install commits, 0 opclasses, 36 OPE domains) and a local superuser cluster (both opclasses created).
The v3 encrypted domains are jsonb-backed, so introspection that resolves a domain to its base type (e.g. Supabase's table editor via postgres-meta) shows them as bare `jsonb` with no hint they are EQL encrypted columns. Attach a one-line COMMENT ON DOMAIN to every public encrypted domain so the type is self-documenting: visible via psql \dD, obj_description(), and any tool that reads pg_type comments (Supabase's types introspection surfaces it). Scalar domains: the comment is code-generated. A new DomainBlock.comment field derives capability text from the domain's terms (Term::operators_for_terms), so it tracks the generated CHECK/operator surface and can't drift; the DO-block templates emit COMMENT ON DOMAIN after each idempotent CREATE DOMAIN (re-applied on reinstall so comment-text changes propagate). Query-operand (_query) twins get a matching 'index terms only; no ciphertext' comment. The three hand-written jsonb SteVec domains (json / jsonb_entry / jsonb_query) get hand-written comments. No behaviour change; comments only. Generated SQL regenerated in place.
The first cut wrapped to ~3 lines in Supabase Studio's type picker. Drop the 'jsonb-backed CipherStash searchable-encryption domain.' boilerplate and compress capability to plain words (equality / ordering / containment / storage only), derived from the operator set. E.g. 'EQL encrypted numeric (equality, ordering)'. Longest is now 62 chars vs ~110.
feat(v3): COMMENT ON DOMAIN for every encrypted domain type
…operator-class fix(v3): install SEM btree operator classes conditionally (Supabase / non-superuser)
…ix (CIP-3442) Every scalar query twin is now public.query_<name> (query_integer_eq, ...), and the hand-written SteVec containment needle follows the same convention: public.jsonb_query -> public.query_jsonb. Domain::query_name (eql-domains) is the single source of truth for the twin naming; Domain::full_name carries the needle's documented exception alongside the existing public.json one. Why: the query operands live in public beside the column domains they twin, so Supabase Studio's Table Builder type picker interleaved never-a-column-type operands with the actual column types. A shared query_ prefix sorts every query operand together, apart from the column domains. Generated artifacts regenerated in place: src/v3/scalars (file names follow the domain names; old files orphan-swept), eql-bindings Rust/TS/JSON (query_<name>.json schema files), and the public-surface golden snapshot. CHANGELOG entries updated/added and U-002 added to docs/upgrading/v3.0.md (the suffix names shipped only in 3.0.0 pre-releases).
The #1 gate filtered operand types on nspname = 'eql_v3', but no type lives in that schema (domains are in public, SEM term types in eql_v3_internal), so the scan matched zero operators and passed vacuously — the invariant was really held up by the name checks and the codegen. Identify EQL operands by catalog domain name in the public namespace instead (including the query_<name> twins), and assert the scan matches a healthy floor of operators before trusting an empty offender list.
…-3442) Every query-operand domain — the scalar query_<name> twins and the jsonb containment needle query_jsonb — now lives in eql_v3, not public: WHERE col = $1::eql_v3.query_integer_eq; WHERE doc @> $1::eql_v3.query_jsonb. Why: a query operand is never a valid column type, so it does not belong in the column-type namespace. The survive-schema-drop rationale for public placement (dropping EQL-owned schemas must not drop application columns) doesn't apply to a type no application column should use; in eql_v3 the operands are versioned and uninstalled with the rest of the public API surface, and casting requires the same USAGE ON SCHEMA eql_v3 a querying role already needs for the extractors and wrappers. - codegen: query_domain_name qualifies query twins with SCHEMA; the query_types template creates/comments the domains in eql_v3. - bindings: DomainType::sql_domain is eql_v3.query_<name>; DomainType::domain now strips whichever schema qualifies sql_domain. - Uninstall semantics pinned: a column misusing a query-operand domain is dropped with the schema (CASCADE); column-domain tables still survive. - New public-surface pin: query_operand_domains_are_eql_v3_jsonb_domains (mirror of the user-column placement pins, which now exclude the needle). - Docs: U-002 extended to cover the schema move; permissions.md gains the query-operand cast row; CHANGELOG entries updated.
…prefix-naming feat(eql v3)!: query-operand domains — query_<name> prefix, homed in the eql_v3 schema (CIP-3442)
The plan's first "Not verified" item is closed. Pointed `packages/protect-ffi/crates/protect-ffi/Cargo.toml:25` at the EQL tree and ran `cargo build -p protect-ffi`: dev profile finished in 3m 10s from a cleared target, no errors, 41 MB `libprotect_ffi.dylib`. `Cargo.lock` moved the entry to 3.0.4 with no `source` line — the path-dep signature. Scaffolding reverted; tree clean. Byte-identity re-confirmed the same day by `diff -rq` against the vendored registry 3.0.2 sources: `src/`, `bindings/` and `schema/` all identical, only `sql/cipherstash-encrypt.sql` differing (2 642 603 vs 2 649 631 bytes). So the lockfile's 3.0.2 -> 3.0.4 move is a version number changing over identical Rust, which is what makes Phase 3's early sequencing safe. The credentialed half is still open: `integration-tests` has not run against the path dep, and that is what turns "no-op" from an argument about byte-identity into an observed result. Tracked on CIP-3740.
Mirrors the protect-ffi cleanup (1e922ec). Nothing here changes EQL's behaviour; it removes files the monorepo already owns or that never had anything behind them. - `.gitmodules` declared `languages/go/goeql`, a path with zero tracked files. Dead on arrival. - `pnpm-lock.yaml`, `pnpm-workspace.yaml`, `.npmrc` — the root ones govern once the package is a workspace member. - `scripts/lint-no-workflow-caching.mjs` and its test — this repo's copy is the descendant, and carries a different target list. - `CODE_OF_CONDUCT.md`, `LICENSE` — duplicates of the root files. - `biome.json` — schema 1.8.3 against the root's 2.5.2. - `.changeset/config.json` and `.changeset/README.md` — the root changeset config governs. `packages/eql/.changeset/rename-ste-vec-contains.md` is unreleased work, so it moves to the root `.changeset/` rather than being deleted — but **parked** as `.md.deferred`, not live. It declares `'@cipherstash/eql': major`, and the package is not a workspace member until the next commit in this stack; a live changeset would have `changeset version` trying to bump a package changesets cannot see. Changesets and the guards both select on `.endsWith('.md')`, so the extension is inert. It is activated at the release cutover (CIP-3743) alongside the two FFI files already parked. `CLAUDE.md` becomes `AGENTS.md`, matching this repo's convention. `packages/eql/package.json` is deliberately left in place; deleting the private `@cipherstash/eql-workspace` root belongs with the workspace wiring (CIP-3739), where the reasoning for it lives. Part of CIP-3738.
Found by executing Phase 1. The plan ordered the Biome 1.8.3 -> 2.5.2 reflow in Phase 1 and the `biome.json` ignore entries in Phase 2. That order is wrong, and the failure is not cosmetic. The imported tree carries 211 generated `.ts` files: 104 under `crates/eql-bindings/bindings`, 92 schemas beside them, 199 synced into the nested package's `src/generated`. Every one carries a ts-rs "Do not edit this file manually" header and is drift-gated by `mise run types:check`, which regenerates and `git diff`s. Running `code:fix` before the ignores land reformats all of them into a shape the generator never emits, so the gate fails and keeps failing until the formatting is reverted. The plan's ignore list was also short: it named `packages/eql/packages/eql/src/generated` but not the two `crates/` directories the sync reads from, which is 196 of the 211 files. Secondary reason for the move: Biome is not installed until `pnpm install` runs, and installing at that point enrols the private `@cipherstash/eql-workspace` manifest that Phase 2 deletes. Phase 2 now adds the (expanded) ignores first and runs the reflow after, with `mise run types:check` as the check that the ordering held.
Must land BEFORE the first `code:fix` over the subtree, which is why it is its own commit rather than part of the workspace wiring. The import brings 211 generated `.ts` files: 104 under `crates/eql-bindings/bindings`, 92 schemas beside them, and 199 synced into the nested package's `src/generated`. Every one carries a ts-rs "Do not edit this file manually" header, and `mise run types:check` gates them by regenerating and `git diff`-ing. Biome reformatting them produces a shape the generator never emits, so that gate fails and keeps failing until the formatting is reverted. Naming only the nested `src/generated` would miss 196 of the 211 — the two `crates/` directories are where the sync reads FROM. `release`, `target` and `docs/api` are untracked build outputs, listed so a local build does not put them in front of the formatter.
Mechanical only. EQL's own `biome.json` declared schema 1.8.3 against this repo's 2.5.2, so the first `pnpm run code:fix` over the subtree rewrites line breaking, adds trailing commas, and sorts imports. Separated from the wiring commit so neither diff hides in the other. Verified token-preserving rather than by eye: for each file, the multiset of identifiers, numbers and string bodies is identical to the previous commit's, with punctuation a formatter may add or remove (commas, wrapping parens) and quote style excluded. Import reordering and added trailing commas are the only reason a naive whitespace-strip comparison flags these files. Eight files, all inside `packages/eql`. The 211 generated `.ts` files are untouched, because the ignore entries landed in the preceding commit.
`@cipherstash/eql` becomes a first-party workspace package that root `build` and root `test` both reach, and the lockstep version hook moves to where Changesets will actually run it. **The subtree-root manifest is deleted.** `packages/eql/package.json` was the private `@cipherstash/eql-workspace`, and `packages/*` already matches `packages/eql`, so leaving it enrols a second workspace root. Nothing load-bearing was left once `version` moved: three scripts conflicted with this repo's root, two were already defined here under the same names, and the remaining four are delegations CI never calls (`test-eql.yml` runs `mise run types:check`, not the npm alias). The subtree needs no `node_modules` — its mise tasks call bare `node` and say so, and no task under `tasks/` invokes pnpm. **The lockstep hook moves in three parts, because the script alone does nothing.** `sync-lockstep-versions.mjs` moves to `scripts/`, its test to `scripts/__tests__/` (this repo's vitest config globs only that directory — flat beside the script it would have run nowhere). The script resolved its own paths relative to its parent directory, which was the EQL root and is now the monorepo root, so it now derives `eqlRoot` explicitly; without that it would have read the deleted subtree manifest and looked for `crates/` at the wrong level. Root `package.json` gains `version`, and `release.yml` gains `version: pnpm run version` — the action ignores the root script without it, so npm would bump while `Cargo.toml` and the bundled SQL kept the old version. `scripts/__tests__/release-version-hook.test.mjs` asserts all three. **Turbo test selection is fixed.** `--filter './packages/*'` is one level. Proved with `--dry=json`: under the old filter the graph contained `@cipherstash/eql#build` (pulled in by `^build` from its consumers) and no `#test` at all. `./packages/**` yields both. Safe because the six `platforms/*` manifests declare no scripts, so the wider glob selects them and finds no task. `lint-typecheck-scope.mjs` gains the two nested roots for the same reason — its walk is one level deep, and a package outside the scan reads exactly like one that passed. Mutation-checked, all three: - deleted the package's `dist/`; root `pnpm build` recreated it - broke a unit test; root `pnpm test` failed with `@cipherstash/eql#test ... exited (1)` - `pnpm run test:scripts` runs the moved lockstep test (4 tests) `bench-index-expressions.test.mjs` needed a fix the plan did not anticipate. It read the bundle from `@cipherstash/eql/dist/sql/`, which resolved into a published tarball and now resolves to a workspace package whose `dist/` is a build output — absent on a clean checkout and in the `lint` jobs that run this suite, neither of which builds. It now reads the committed `sql/`, with a cross-check that the shipped copy is byte-identical when a build has run. Also: `EQL_ROOT` is set in `packages/eql/mise.toml` so `doc-anchors.sh` and `known-failures.sh` stop resolving the monorepo root (verified: the override resolves to the subtree, and without it `known_failure.rs` is missing). Consumers repointed to `workspace:^`. The nested package's default-`catalog:` references become `catalog:repo`, with `@types/node` added to the catalog. `@cipherstash/eql` removed from `minimumReleaseAgeExclude` as dead config. Part of CIP-3739.
…hat were wrong Closes the three checks CIP-3739 left stated-but-unrun, and corrects two of them where running the check showed the stated method could not have worked. `doc-anchors.sh`: the checklist said to confirm it FAILS with `EQL_ROOT` unset. It does not — it exits 0 either way. Unset, it resolves the Stack root and checks 170 links instead of EQL's 117, a superset. So the evidence is the count, not the status, exactly as `mise.toml`'s own comment already warned. `known-failures.sh` does fail loudly (exit 2, missing registry), and both pass under mise. `lint-typecheck-scope.mjs`: the mutation needs two edits, not one. The nested package's `build` is `tsup` and it declares no `typecheck` script, so the linter skips it by design whatever its tsconfig says. Unscoping the tsconfig AND wiring a gate produces the offender line; restoring `include` alone clears it. The `WORKSPACE_ROOTS` entry is therefore coverage for the day that package wires a gate, not a live check today. `mise run types:check` passes after the Biome reflow — 104 TS bindings and 92 JSON schemas regenerated to a clean `git diff`, which is the gate the ignore-before-reflow ordering exists to protect. Also records the cargo PATH trap: 14 turbo tasks under `TURBO_FORCE=true` (without which a cache hit skips the task and the trap proves nothing), zero cargo invocations. And one finding that belongs to the FFI cutover rather than here: `@cipherstash/protect-ffi` carries no `publishConfig`, so Changesets hands it `--access restricted`. Its six platform siblings escape only because `release.yml:149` publishes them with an explicit `--access public` first. Recorded in Phase 5 with the reason it is not fixed in this branch.
The subtree import deposited EQL's ten workflows at
`packages/eql/.github/workflows/`, which GitHub never reads — it takes
workflows from the repository root alone. So since the import the entire
EQL suite has run nowhere: the sharded SQLx matrix, the e2e property
oracle, the codegen parity gates, the doc and known-failure checks. A
suite that never starts reads exactly like a suite that passes, which is
the same failure the protect-ffi absorption hit one round earlier.
This ports `test-eql.yml`, the one that gates merges. Four mechanical
changes, each silent when wrong:
* `defaults.run.working-directory: packages/eql` — one line instead of
~30 per-step copies, each independently forgettable.
* `working_directory:` on all 13 mise-action steps. `defaults.run` does
not reach a `uses:`, and mise reads config from cwd and its PARENTS,
so an action at the repo root never sees packages/eql/mise.toml.
* `workspaces: packages/eql` on all 10 rust-cache steps.
* `packages/eql/` on the artifact upload/download paths, which are
workspace-root relative for the same reason.
And two that are not mechanical. The `dorny/paths-filter` globs are
repo-root relative, so the unprefixed `src/**` would have selected
packages/stack and never packages/eql — the heavy jobs would skip on
exactly the changes they exist to check, report `skipped`, and
`ci-required` treats skipped as pass. And the two credentialed jobs move
CS_WORKSPACE_CRN / CS_CLIENT_ID from `secrets.` to `vars.`, matching this
repo's split; reading them from `secrets.` yields the empty string rather
than an error.
Both credentialed jobs now run `require-cs-secrets` first, and both are
recorded in BINDING_EXEMPT_JOBS with the reason: they encrypt through
`cipherstash-client` in Rust, not through the Node binding, so they are
the first jobs here to hold CS_* legitimately without building index.node.
Two existing guards needed extending rather than exempting:
* `workflow-dispatch-job-conditions.test.mjs` compared whole conditions
for the fork guard. EQL's two jobs `&&` the guard with a relevance
gate, so the single-spelling rule now applies to the CLAUSE, compared
verbatim, with another conjunct permitted beside it. Contexts gained
`merge_group` and a permissive `needs`, without which the compound
conditions read as skipped on every event. The verdict table is now
derived from the workflow's declared triggers — `test-eql.yml` has no
`push:`, and asserting about a run that cannot happen is not a check.
The evaluator models `always()`, which is total and therefore
modellable; `success()` and argument-taking calls still throw.
* `ffi-binding-step-order.test.mjs` gained the two jobs and their
exemptions.
Mutation-checked, six ways: unprefixing a paths-filter entry, deleting
the partition step, demoting it to a comment, pointing rust-cache back at
the monorepo root, dropping the e2e pre-flight, and respelling the fork
clause with its disjuncts swapped. Each fails the intended guard and only
that one.
actionlint reports the identical 11 shellcheck findings before and after
the port and nothing else, so the port introduced no new diagnostic.
The deposit keeps its four release workflows: porting those is what makes
them fire, and it is gated on repointing npm and crates.io trusted
publishing. `eql-suite-ci.test.mjs` holds them as a shrinking allowlist
that fails in both directions, so the last removal has to delete the
directory rather than quietly leaving a check that means nothing.
The two scheduled Rust jobs, same move as test-eql.yml and the same four
path rewrites. Neither gates a merge, which is exactly why they are worth
porting rather than leaving: nobody watches a nightly, so one that never
starts is invisible for as long as it takes someone to ask.
`macro-expand-eql.yml` needed only paths. `bench-eql.yml` needed three
things more:
* Its `paths:` filter sits under `push: branches: [main]`. Unprefixed,
`src/**/*.sql` and `tests/sqlx/**/*` match nothing under this
repository's root, so the bench would have stopped running on pushes
without reporting anything — there is no status for a workflow that
never triggers.
* It holds CS_* (scoped to the one step, upstream's least-privilege
choice, kept). CS_WORKSPACE_CRN and CS_CLIENT_ID move to `vars.`, and
the job now runs `require-cs-secrets` first. That matters more here
than on a PR job: this is scheduled and budgeted at 60 minutes, so
the choice is between a legible nightly failure and "Auth strategy
error: Not authenticated" an hour in.
* Its three actions were on floating major tags. SHA-pinned to the same
commits test-eql.yml uses — the rest of the imported suite already
pins by SHA, and a mutable tag on a job holding live credentials
means the code running there can change without a commit here.
checkout moves v4 -> v6 with that, matching the rest of the suite.
`bench-eql.yml / bench` is registered as credentialed and
binding-exempt for the same reason as the EQL jobs already there: it
encrypts through cipherstash-client in Rust and never loads index.node.
It also joins EXPECTED_ASYMMETRIES in the paths-filter parity guard —
one event, no second list to drift from, and PRs never run it.
actionlint reports only the four pre-existing shellcheck infos across
both files and nothing structural.
…kflow The other half of the split `lintWiring.test.ts` holds for protect-ffi. There, two properties: no cargo on the default `test` path, and every cargo check reachable from `test:cargo`. EQL has no npm-script layer over its cargo work — its checks are mise tasks invoked straight from workflows — so the first is free (`@cipherstash/eql`'s `test` is `vitest run`) and the second had nowhere to attach except CI itself. So the property is asserted one level up: a mise task that compiles or runs Rust must be reached by some workflow GitHub actually executes, directly or through `depends`. Nineteen tasks qualify. Five are not reached, and each is now named with its reason — a watcher, two `:regen` halves whose read halves run in CI, an unsharded local variant of the sharded suite, and a unit-test task `test:crates` subsumes. The exemptions are checked in both directions: an entry for a task CI has started running, or one that no longer exists, fails. Two things this needed that are easy to get subtly wrong, so both are pinned by their own test: * Most heavy tasks are one line — `run = "bash tasks/test/foo.sh"` — with every cargo invocation in the script. Reading only the task block finds cargo in 15 tasks and misses `test:sqlx:archive`, `test:sqlx:partition` and `codegen:parity`, i.e. the class most worth checking. Following that one hop takes it to 19, and the floor is set at 18 so losing the resolution fails rather than quietly shrinking the set. * `test:sqlx` is a prefix of `test:sqlx:archive`. A substring search marks it reached by the archive step and drops it out of the exemption list silently — so the name match is bounded on both sides. No TOML parser: adding a dependency is an audit decision here, and the shape needed is small. The scan is guarded on task count, on a named sample, and on a known `depends` edge, so a format change fails instead of emptying it. Mutation-checked four ways: removing a legitimate exemption, adding one for a task CI runs, dropping the script-body resolution, and the prefix-collision case as a unit test.
The subtree brought a second Cargo workspace — five members and its own Cargo.lock — and Dependabot's cargo `directory:` names a single workspace root, not a glob, so the protect-ffi entry could not reach it. Worth saying plainly: the existing supply-chain e2e check would NOT have caught this. It derives required coverage per ECOSYSTEM from the lockfiles present, and `cargo` was already covered by the protect-ffi entry, so `packages/eql/Cargo.lock` looked monitored while nothing proposed an update for it. The symptom would have been no PR ever arriving. The `ignore` list carries the same four CipherStash crates as protect-ffi, and here the reason is sharper rather than merely analogous: `tests/sqlx/Cargo.toml` pins `cipherstash-client = "=0.42.0"`, the same exact pin at the same version as protect-ffi. A Dependabot PR that moved one workspace and not the other would reintroduce exactly the skew this absorption exists to remove. `eql-bindings` is ignored too — it is published FROM this workspace, so a proposed registry version would fight the lockstep version hook. Mutation-checked: pointing the entry at `/packages/eql/crates` fails the existing "every entry's directory contains the manifest its ecosystem reads" assertion, so the directory is verified by discovery rather than by the comment claiming it.
Phase 4 is done apart from two items that turned out to belong to the
release cutover — an ordering defect of the same class as the Biome one
Phase 1 turned up, and found the same way.
`lint-release.yml` points actionlint at four release workflows, and
`_build-sql.yml` / `_build-docs.yml` are `workflow_call`-only, reached
from EQL's `release.yml` alone. Porting them in this phase lands a gate
aimed at files that do not exist yet. Deleting `packages/eql/.github/`
here is worse: Phase 5 ports the release machinery FROM those files, so
the deletion would mean reconstructing a publish pipeline out of git
history at the one moment nobody wants to. Both move, with the
`cache: false` and caching-lint work that goes with them.
What lands instead is the shrinking allowlist in `eql-suite-ci.test.mjs`.
It is an equality, so it will already be red by the time the last file is
ported — which is what makes the final `rm -r` forced rather than
remembered.
Three plan items also turned out to need more than they said, each
recorded where it will be read:
* `working-directory` belongs on `defaults.run` — but that does not
reach a `uses:`, so the mise-action steps and the artifact paths need
their own treatment.
* "Fork-PR-skip the credentialed jobs" was already true upstream; what
it actually cost was extending two of this repo's guards, because
EQL's conditions are compound where every existing one is bare.
* "reachable from `test:cargo`" has nothing to attach to — EQL has no
npm-script layer over cargo at all — so the property moved up a level
to "reached by a root workflow", asserted over the mise task graph.
And one thing the plan asserted that running it disproved: the existing
supply-chain lockfile check would NOT have caught the missing Dependabot
entry, because it asserts coverage per ecosystem and `cargo` was already
covered.
The reason for the import, and one line. `packages/protect-ffi` pinned
`eql-bindings = "=3.0.2"` from crates.io while the EQL tree carried
3.0.4 — the Rust that emits EQL payloads on a different release from the
SQL that stores them, with nothing anywhere asserting they agree.
Today that skew is benign, which is exactly why now. 3.0.2, 3.0.4 and
the tree are byte-identical across `src/`, `bindings/` and `schema/`
(`diff -rq`); what 3.0.3 and 3.0.4 changed was SQL, carried on the
lockstep version number. So the flip lands with zero behaviour change,
and from here the skew is unrepresentable. Wait for the first divergent
release and it becomes a behaviour change that has to be reasoned about
under credentialed test.
Cargo accepts the path dep across workspace boundaries: `eql-bindings`
is a member of the EQL workspace, declares no `[lints] workspace = true`,
and so carries nothing to inherit. `Cargo.lock` moves the entry to 3.0.4
and drops `source` and `checksum`, which is the path-dep signature.
Verified without credentials, which covers three of the four things
worth covering:
* `cargo build -p protect-ffi` — clean, 4m 51s from cold.
* `pnpm --filter @cipherstash/protect-ffi test:cargo` — 310 passed, 0
failed; `cargo fmt --check` clean.
* `cargo build -p protect-ffi --target wasm32-unknown-unknown` —
clean, 1m 18s. Not in the plan's list and it should have been: the
EQL workspace never builds for wasm32, and a merged workspace was
rejected partly on that ground, so it is the target where a
cross-workspace path dep would plausibly break first. It does not.
The fourth is the credentialed `integration-tests` suite, which needs
Docker and `CS_*` and has NOT been run against the path dep. That is
what turns "no-op" from an argument about byte-identity into an observed
result — it is the only path here that carries an EQL payload through
Postgres — so the plan keeps that box unticked and says why.
The path dep in the previous commit is one line, and nothing notices if it is undone. A registry version compiles clean, passes every unit test, and emits payloads the installed SQL may not read — a failure that surfaces in a database, not in CI. The property is invisible at every layer that would otherwise catch it, so it takes a linter. `eql-bindings` is in-tree when it carries a `path`, or `workspace = true` (which defers to the workspace root's own entry — itself scanned, so the deferral cannot launder a pin). `@cipherstash/eql` is in-tree when the specifier starts with `workspace:`. One exemption: `packages/protect-ffi/integration-tests`, which installs with `npm ci` and cannot resolve a `workspace:` specifier, keyed with a written reason in the shape of `BINDING_EXEMPT_JOBS`. Three things the plan's one-line instruction did not anticipate, each found by executing it: * Staleness has to mean "excuses nothing", not "names nothing". The obvious spelling — stale when no manifest declares what the exemption names — passes on the day `integration-tests` joins the workspace, because that manifest still declares `@cipherstash/eql`; it just no longer needs excusing. The entry would outlive its reason and stand as a permanent permission, inherited by whatever lands at that path next. Measured against the registry-pinned declarations instead, that follow-up PR cannot go green until the entry is deleted. * The two exit-2 branches cannot fire against the tree the tests run in — that is what they are for — so the code/message mapping is exported as `report()` and driven with synthetic results. Asserting only that `lint()` detects a stale exemption proves the condition is computed, not that anything happens next, and "detected, then exit 0" is this branch's recurring failure shape. * A broken scan is reported ahead of any offender it found. A scan that lost its subject cannot be trusted to have found every offender either; the alternative sends the reader to fix a manifest, and the fix makes the linter exit 0 with its coverage still gone. Exit 2 rather than 1 for both, following `lint-no-hardcoded-runners.mjs`: the linter could not do its job, which is a different thing to go and fix. `EXPECTED_DECLARERS` is the floor — held as a minimum, and itself floored, since a list emptied out satisfies "nothing missing" trivially. No TOML parser, for the reason `eql-suite-ci.test.mjs` gives. The scan reads the `[dependencies.eql-bindings]` table form, where the dependency name is in the header and the line carrying the version never mentions it, and on the npm side `resolutions` plus both `overrides` spellings — an override is the quietest route back to a registry, since every `workspace:^` in the tree still reads correct while what installs has moved. Mutation-checked four ways against the real tree, each hitting only its own guard: restoring the `=3.0.2` pin, registry-pinning `@cipherstash/eql` in `packages/stack`, blinding the walk by adding `packages` to `SKIP_DIRS` (exit 2, not 0), and flipping the exempt pin to `workspace:^` (exit 2 under the tightened rule, and 0 under the loose one — which is how that rule was found). Wired as `lint:eql-pins` and run from `tests.yml`'s `lint` job, with a test asserting a root workflow invokes it. Nothing else would: it is not a package task, so `pnpm test` does not reach it, and `test:scripts` runs the self-test rather than the script.
Phase 3 is done except for the credentialed run, and the recording is mostly about keeping that distinction from eroding. The verification checklist had `cargo build -p protect-ffi` succeeding against the path dep and the credentialed integration suite passing as ONE box. Half of it is runnable on any machine and half needs Docker and `CS_*`, and a box that is half-runnable is a box that gets ticked. Split in two: the build half is ticked with what actually ran (host, wasm32, 310 cargo tests, `cargo fmt --check`), the credentialed half is left open with the command that closes it and an explicit note not to infer it from the other three. Phase 3's guard item gains what executing it turned up — the staleness rule that had to mean "excuses nothing" rather than "names nothing", the exported `report()` that makes the two exit-2 branches reachable from a test, and the ordering that reports a broken scan ahead of any offender it found. Also the wasm32 build, which was not in the plan and should have been. "Not verified" item 1 now says the credentialed suite is the only unfinished item in Phase 3, and that it cannot be closed by reasoning.
🦋 Changeset detectedLatest commit: ea9e140 The changes in this PR will be included in the next version bump. This PR includes changesets to release 11 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…tory `test-eql.yml` sets `defaults.run.working-directory: packages/eql` at workflow level. That is the right call — every `run:` in the file is written against the EQL root, and the alternative is ~30 per-step lines that can each be forgotten independently. But a workflow-level default reaches every job, including the two that deliberately never check out: `setup`, which is pure bash deriving the matrix, and `ci-required`, which only reads `needs.*.result`. Both fail before their first line with `chdir: packages/eql: No such file or directory`. Found by opening the PR: `Compute matrix` went red in four seconds. `ci-required` is the worse of the two, and it had not run yet — it `needs:` everything. It is the required check, so it would have failed for a reason unrelated to any job it reports on. A red `ci-required` reads as "a test failed"; this one would have meant "the gate could not cd". Both take a job-level `working-directory: .`. Adding a checkout would also work and is wrong: it buys nothing either job uses. The interaction is invisible in review — the job is correct, the default is correct, and the thing that breaks them is somewhere else in the file. So `scripts/__tests__/workflow-working-directory.test.mjs` discovers it instead: a job under a non-`.` working directory, with `run:` steps that do not set their own, and no `actions/checkout`. Steps carrying their own `working-directory:` are excluded — they have overridden the default and answer for themselves. Floored on the two known opt-outs so the scan cannot empty out silently, and mutation-checked: removing either opt-out fails both the offender check and the floor.
…t it `stack-prisma` reaches `@cipherstash/eql/sql` through that package's `exports` map, which resolves to `./dist/sql.d.ts`. While `@cipherstash/eql` came from the registry, `dist/` arrived inside the tarball and a bare `pnpm --filter ... run typecheck` resolved it whatever CI had built. Phase 2 made it a workspace package at `workspace:^`, so `dist/` is now a build output — and this step turned out to be the first in its job that needs one. It fails `TS2307` on three files. The step's own comment said no build was needed first, and it was right about the half it was describing: the tsconfig resolves *stack* subpaths to source. It said nothing about `@cipherstash/eql`, because at the time there was nothing to say. Routed through turbo, whose `typecheck` task declares `dependsOn: ["^build"]`. Reproduced locally by deleting `packages/eql/packages/eql/dist` — the bare command fails with exactly CI's three errors, and the turbo one builds four tasks and passes. With `dist/` present, as any developer machine has it after one `pnpm build`, both pass; that is why this reached CI. `stack-prisma` is the only step affected. Checked the other two bare invocations against a deleted `dist/`: `@cipherstash/stack`'s `test:types` and `@cipherstash/wizard`'s `typecheck` both still pass, and wizard does not depend on `@cipherstash/eql` at all. `workflow-turbo-build-deps.test.mjs` already existed for exactly this trap, and this step was on its grandfathered KNOWN_BARE list — "they pass today only because an earlier step in the same job builds the workspace ... do not add to it". The import is what collected. The entry is removed and replaced by a comment recording which bet lost, since the useful part of that list is not which lines are on it but that each one is a wager that no future change makes the implicit ordering false. Mutation-checked both directions: reverting the workflow line fails the offender check by name, and restoring the allowlist entry fails the staleness check.
The prisma-next typecheck failure was a missed step, not a broken one: `@cipherstash/eql` stopped being a registry tarball with `dist/` inside and became a workspace package whose `dist/` is a build output. So the question is not whether that one step is fixed — it is whether the rest of EQL's build structure survived the same transition. Four checks, all run rather than reasoned about. `mise run build` works here and produces the same bytes. 268 files in declared order, both installer-completeness gates passing, 2 649 625 bytes. Against the two committed copies it differs in exactly three lines: `DEV` where they say `3.0.4`. That is the version stamp `release:prepare_bindings_assets` applies, which the root `version` script invokes at `changeset version`. The committed SQL is not stale, and the lag between releases is the design. No CI step was dropped. Diffing `mise run` invocations between upstream's ten workflows and this repo's, all 36 tasks are reached — except five that appear only in the four release workflows Phase 5 ports. Comparing invocation sets rather than reading the ported files is the point: a faithful-looking port cannot pass that by accident. The turbo cache restores a complete package. Worth checking because `dist/sql/` and `dist/schema/` are copied by tsup's `onSuccess` hook rather than emitted by tsup, which is the shape that produced the `@cipherstash/protect-ffi#build` bug — a cache hit restoring nothing while reporting success. `diff -rq` between a cache-restored `dist/` and a `--force` build: identical. No `outputs` override needed; `dist/**` is already what the repo-wide task declares. `stack-prisma` was the only consumer affected. With eql's `dist/` deleted and everything else left built, the six other bare `pnpm --filter` steps in the workflows all still pass, and `dist/` stayed absent throughout — without that last check a step that rebuilt it would have made every later PASS meaningless. One gap recorded rather than closed: `workflow-turbo-build-deps.test.mjs` matches bare invocations of scripts `turbo.json` declares as TASKS, so it is structurally blind to five of those six steps (`test:types`, `test:typecheck:wasm`). They are verified by hand today and nothing keeps them verified.
…task list
`workflow-turbo-build-deps.test.mjs` asked "is this script a task
turbo.json declares with `dependsOn: ["^build"]`?". That is a question
about turbo's configuration, and the risk is not in turbo's
configuration — it is in the package graph. Five bare steps ran
`test:types` and `test:typecheck:wasm`, which are package scripts and not
turbo tasks at all, so the guard skipped them in silence. Their packages
import build output from workspace dependencies exactly as
`stack-prisma` does. They simply had not broken yet.
The second check asks what the risk is actually made of: does the package
whose script is being run declare a workspace dependency that emits a
build? If it does, a bare `pnpm --filter` resolves a `dist/` that nothing
in the command guarantees exists.
That question sorts the tree cleanly, with no allowlist:
* `@cipherstash/stack`, `test-kit`, `stack-drizzle`, `stack-supabase`
and `prisma-example` all qualify, so their seven bare steps are now
routed through turbo. `test:types`, `analyze:complexity` and `emit`
are declared in turbo.json to make that possible — `analyze:complexity`
without `^build`, because `fta` reads source text and resolves no
imports, which is worth stating in the task rather than in a comment.
* `@cipherstash/protect-ffi` and `@cipherstash/wizard` have no workspace
dependency that builds, so their five remaining bare steps are safe on
the merits. They are NOT exempted — if either ever gains one, the
guard starts covering them, and a test pins that both are currently
outside it.
KNOWN_BARE is now empty. `@cipherstash/wizard`'s entry went with it: not
because it was at risk, but because an empty list is a rule and a
one-entry list is a habit.
Composite actions are scanned too. `.github/actions/*` steps run on the
same runner as the calling job and nothing about the trap changes one
file away, but `workflowFiles()` never looked there. A floor assertion
fails if no action step is seen, since that is how the widened scan would
narrow again without anyone noticing.
Mutation-checked five ways. The decisive one is a bare step invoking a
script turbo.json does not declare: all three of the original checks pass
on it and only the new one fails, which is the gap this closes stated as
a test. Also reverting the stack-prisma fix (the failure that started
this), reverting a routed `test:types`, reverting a step in a different
workflow, and dropping the EQL glob from pnpm-workspace.yaml to blind the
graph — that last one fails the floor rather than passing over an empty
set.
…ibling The SteVec selectors the sqlx suite pins are MACs of (column context, JSONPath) under the CipherStash workspace keyset. This repo's CI names a different workspace than the one EQL was developed against, so the pins re-pinned on the move — no Rust, no SQL and no fixture logic changed. v3_doc_integer::SELECTOR fce8be75… -> 606a4a44…, the value two independent runs reported. Its drift guard did its job: one copy-pasteable message naming the pinned value, the live value and the cause. Its sibling SEL_HELLO_OP is keyed identically and had no such guard, so its drift surfaces as wrong ANSWERS — LB3 counting 0 distinct ops, ORDER BY arms returning insertion order — with the live value nowhere in the output. v3_jsonb_sel_hello_op_matches_fixture closes that asymmetry. It prints every op-carrying selector with a row count and op-length profile rather than inferring the replacement: guessing wrong re-pins to the wrong leaf silently, which is the bug already in SEL_HELLO_OP's history, where it named $.number while claiming $.hello and survived because equality-only suites cannot separate them. The shards also ran fail-fast, so shard 1 reported 11 failures and skipped 643 of its 710 tests — one environmental fault answered one question per CI round trip. sqlx-partition.sh now passes --no-fail-fast, which is nextest's own suggestion in that output; the shards run ~5s tests in parallel, so a fully-failing shard still finishes in minutes. Recorded in the plan, including the consequence that these pins are now coupled to this repo's CI workspace, and that selectors.rs holds five more workspace-keyed constants with zero consumers and no guard.
b325a0c7… -> 6f1db3bd…, the second and last of the workspace-keyed SteVec selectors. The new drift guard reported the candidates instead of choosing one, so the leaf is identified rather than guessed: at 16 * len + 20 hex chars all six op-carrying leaves reconcile against the fixture's known documents — $.empty 20, $.accented 84/180/196 (three lengths, not four, because the café/cafe collision pair shares one), $.nested.deep 148 for "constant", $.number and $.large a fixed-width 132, and $.hello alone spanning 132/148 for world-1..9 vs world-10. --no-fail-fast paid for itself immediately: the run reported all seven remaining failures at once (695/671/655 tests per shard, shard 1 fully green) and every one is a SEL_HELLO_OP consumer, so this is expected to be the last of it rather than the next round of one.
…sed HEAD The Test JS lockstep failure is not a CI problem. `@cipherstash/eql` in the workspace calls itself 3.0.4 but its install SQL is a92cc041…, where the published 3.0.4 tarball is 63104a81…. The whole diff is one unreleased rename, ste_vec_contains -> jsonb_document_contains, which arrived with the import (63af028) and was never released: npm's newest is 3.0.4, the version file says 3.0.4, and there is no changeset for it. stack-prisma's lockstep test is the only thing that noticed, and it is right to fail — the installed release's SQL is baked by no published migration. The reach is wider: `stash eql install` calls readInstallSql() with no digest check, so a database would carry the renamed function while reporting a version whose published SQL defines the old one. Left open rather than patched green. Both routes to a passing test — bumping the EQL version with a new baseline migration, or reverting the rename — are release decisions with customer-facing artefacts, and the published migrations are frozen history by construction.
…version The tree has shipped an unreleased rename since the import: eql_v3.ste_vec_contains -> eql_v3.jsonb_document_contains, carried under the name 3.0.4 while published 3.0.4 defines the old name. stack-prisma's lockstep test is what surfaced it. This applies the major bump the parked changeset already specified, so the package stops claiming a version whose published SQL it does not contain. Applied by hand rather than by `changeset version`: eleven unrelated changesets are pending, so versioning here would have released the whole repository. The parked changeset is deleted with the bump so the cutover cannot apply it twice, and its text is now the 4.0.0 CHANGELOG entry. The bump changes exactly three lines of SQL — the version stamps — and nothing else; both release manifests, the crate and the npm package agree at 4.0.0. Found while doing it, and fixed here: prepare-bindings-assets.sh could package SQL stamped with a DIFFERENT version than the manifest it wrote. `--version` is not in tasks/build.sh's `#MISE sources`, so `mise run build --version X` is a cache HIT whenever the SQL sources are unchanged and re-serves whatever the previous build stamped; the script's only check was `test -f`. On the first real use it copied a DEV-stamped bundle under a manifest asserting 4.0.0 — and the digest still verified, because it is taken from the copied bytes. The build is now forced and the stamp is checked against the requested version before anything is written. Two cases added to the script's test, mutation-checked by neutering the comparison. Not yet green: the lockstep test still needs a stack-prisma migration that bakes 4.0.0's SQL. And nothing may publish until npm trusted publishing for @cipherstash/eql points at cipherstash/stack — three published packages depend on it via workspace:^, which packs as ^4.0.0.
Maintainer decision, and consistent with EQL's own precedent: 3.0.1 shipped the fuzzy-match operator change (`@>` / `<@` -> `@@`) as a patch, so a function rename that leaves the operators intact sits at the same level. The parked changeset had proposed major. The trade is recorded in the changelog rather than left implicit: a patch reaches every consumer already on a `^3.x` range at their next install, which a major would not have done. Whole tree re-synced through the same lockstep path — package.json, both release manifests, the generated TS manifest, eql-bindings' Cargo.toml, and both bundled SQL copies all read 3.0.5, with the SQL stamps and digests regenerated rather than edited. No 4.0.0 remains anywhere.
…db init additive The lockstep test has been red since the tree released the jsonb_document_contains rename as 3.0.5: it requires the installed @cipherstash/eql's install SQL to be baked by some published migration, and 7ad9c9f8 was baked by none. 9b1c44d said as much when it applied the version bump. This ships the migration it deferred. Two artefacts carry the new bundle, and the second one breaks the append-only rule on purpose. 20260814T0000_upgrade_eql_v3_3_0_5 is the ordinary half - a self-edge carrying cipherstash:upgrade-eql-v3-bundle-3.0.5-v1, modelled on the 3.0.2 and 3.0.4 edges, which is how an already-baselined database reaches the new bundle under `migrate`. The baseline is re-emitted, which the file's own header forbids. No upgrade edge can ever be walked by `db init`: every one of them is a self-edge, checkIntegrity requires a self-edge to carry a data-class op, and db init runs allowedOperationClasses: ['additive']. So a fresh database has to collect every head-ref invariant from the genesis edge it walks, and a newly required invariant either lands there or arrives on a second from: null edge duplicating the full 2.6 MB bundle - permanently, once per EQL release. Re-emitting was taken instead while @cipherstash/stack-prisma@1.0.0 was 14 days old at ~253 monthly downloads, so the blast radius was small and knowable; the changeset carries the delete-and-re-plan instruction, and both the migration header and the plan doc record that the second genesis edge is the correct shape once adoption is real. The trade is to be re-argued at the next bump, not defaulted to. The baseline moves fc495f7f -> 1ae73282 and its baked SQL 63104a81 -> 7ad9c9f8. The 3.0.2 and 3.0.4 edges keep their own frozen digests untouched - that separation is what makes the lockstep check meaningful rather than self-satisfying. Three new cases: the 3.0.5 edge's shape, that its baked SQL actually carries the rename (jsonb_document_contains present, ste_vec_contains absent), and that the genesis edge alone covers every head-ref invariant with all-additive ops - the db init property the carrier ops exist for, which nothing asserted directly. Found while doing it, and fixed here: test/live/migration-apply-live-pg.test.ts has been broken since #820. It demanded exactly one baseline op and asserted operationClass 'data', both stale since the carriers landed and the install op became additive. It only runs against a live database, so CI has never seen it. It now selects the install op by id. Skills: stash-postgres claimed the CLI pins @cipherstash/eql to an exact version, which stopped being true when EQL moved in-tree earlier in this PR. stash-supabase re-states its PostgREST limits against 3.0.5 (unchanged in substance). And packages/eql/AGENTS.md still named ste_vec_contains as a public wrapper. Verified: stack-prisma 348 passed / 27 skipped, typecheck clean, stash 1232 passed, test:scripts 515 passed, examples/prisma typecheck clean, biome clean. Full `pnpm test` cannot complete locally - @cipherstash/stack needs the protect-ffi binding this machine has no cargo build for; CI builds it.
tobyhede
marked this pull request as ready for review
August 14, 2026 02:55
freshtonic
requested changes
Aug 14, 2026
freshtonic
left a comment
freshtonic
left a comment
Contributor
There was a problem hiding this comment.
Reviewed with the diff split into its parts: the subtree import verified structurally (genuine two-parent merge, 975 files, insertions only), and the ~4,200-line non-import surface reviewed in depth — workspace/turbo wiring, the release hook, all three ported workflows against their upstream versions, the path-dep flip and its linter, the four new guards, and the post-description commits (the 3.0.5 story and the stack-prisma baseline re-emit).
First, credit where due: the port is faithful (all 16 test-eql jobs, steps and matrix logic byte-equivalent to upstream, with each of the four port deltas guarded), the path-dep flip is verifiably zero-behaviour-change (git diff from the import commit to HEAD over eql-bindings/{src,bindings,schema} is empty, and upstream history shows zero commits to those trees since the 3.0.2 release commit), the baseline re-emit mechanics are sound and well-tested in both fresh-DB and upgrade paths, and the guard engineering (deposit-equality, floors, reasoned exemptions, the dispatch-conditions evaluator extension) is genuinely good. Also: the credentialed protect-ffi integration suite ran green on this PR (6m33s) — the Cargo.toml flip trips its path filter — so the PR body's own "do not merge without it" box is now observably closed; please tick plan line 285/356 citing that run.
Requesting changes on four blocking findings:
Blocking
1. The release job cannot run the version hook it now depends on. scripts/sync-lockstep-versions.mjs ends in execFileSync('mise', ['run', 'release:prepare_bindings_assets', …]), which chains to cargo run -p eql-codegen (via packages/eql/tasks/build.sh). The release job in .github/workflows/release.yml installs pnpm, Node and node-gyp — no mise, no Rust (and GitHub's ubuntu image ships neither mise nor a guaranteed-compatible toolchain). The moment changesets/action takes its version branch — which it will on the first main push after this merges, because this PR carries two live changesets — pnpm run version dies with ENOENT. release-version-hook.test.mjs asserts the wiring exists but nothing asserts it is runnable in that job's environment.
2. Merging arms a release that cannot succeed, with no mechanical guard. The in-tree @cipherstash/eql is 3.0.5, public, with sql/release-manifest.json committed at 3.0.5 so its prepublishOnly gate passes — meaning changeset publish will attempt it (it publishes any public workspace package whose version is absent from npm, changeset or no changeset; npm's max is 3.0.4). That publish fails: trusted publishing still names cipherstash/encrypt-query-language, as do the manifest's repository/bugs fields, and the job has no NPM_TOKEN by design. Meanwhile changeset publish guarantees no ordering, so stash, @cipherstash/stack and @cipherstash/stack-prisma can publish successfully in the same run carrying workspace:^ → ^3.0.5 in runtime dependencies (packages/cli/package.json:50, packages/stack-prisma/package.json:80) — a range no registry version satisfies, breaking every consumer install. Nothing in release.yml, scripts/release-gate.mjs, or any lint prevents this; the planned lint-no-eql-changeset.mjs (plan line 318) doesn't exist, and the FFI guard's "workspace version already on npm" no-op assumption is exactly what the hand-applied 3.0.5 bump breaks. 9b1c44d9's "nothing may publish until trusted publishing is repointed" is prose, not a mechanism. Needed before merge: a mechanical block (publish-side exclusion for eql, or a release-gate check that every packed workspace:^ range is satisfiable on npm) — or complete the Phase-5 repoint first.
3. build-ffi-binding's cache keys don't hash the new compile input. Both the native and wasm keys hash only packages/protect-ffi/** Rust inputs. After the flip, packages/eql/crates/eql-bindings/** is a compile input to index.node, but an edit there changes no hashed file (path deps carry no lock checksum; a src-only edit doesn't touch the lock) → cache hit → every credentialed job runs a stale binding. This contradicts the action's own stated invariant, and this PR itself demonstrates the hole: the 3.0.4→4.0.0→3.0.5 bumps moved nothing in either key. Add packages/eql/crates/** (or the bindings crate specifically) to both hashes.
4. The PG 14–16 matrix is dead code in this repo, and ci-required blocks nothing. test-eql.yml runs the full PG 14–17 matrix only on merge_group — but it is the only workflow in the repo with that trigger, main has no merge-queue rule, and the active ruleset has no required_status_checks. Upstream ran PG 14–16 on every merge; here they run never, and the port also dropped the push: trigger on the (now-false) "required merge queue makes push redundant" justification — so nothing EQL-related runs post-merge at all except the nightly bench. Fix is repo settings (merge queue + require ci-required) or a push: main/scheduled full-matrix run, landing with this PR, not after.
Should-fix
eql_v3.ste_vec_containsis hard-removed (no alias) and ships as patch 3.0.5. The maintainer decision is recorded honestly in the CHANGELOG with the 3.0.1 precedent, and no in-repo consumer references the old name — but pre-3.0.5permissions.md-derived GRANT scripts silently target a nonexistent function after upgrade. A deprecated delegating alias would make the patch genuinely non-breaking; failing that, the changeset's "in practice means Supabase/PostgREST" claim overstates (the PostgREST entrypointsjsonb_contains/jsonb_contained_bydidn't change; the exposure is hand-written SQL and GRANT scripts).eql-suite-ci.test.mjs's task scan misses three classes of task. The[tasks.…]header regex parses 0 of 6 tasks intasks/postgres.toml/tasks/fixtures.toml(bare-header form), mise file tasks (tasks/**.shvia[task_config].includes) aren't parsed at all, and reachability followsdependsbut notmise run Xin run-bodies. Concretely: deletingbench-eql.ymlfails no test — the exact "suite goes quiet" failure this guard exists to prevent — andtest:lint/docs:generate:jsonare unreached by any root workflow today with no exemption entry. The floors (≥30/≥18) are satisfiable by mise.toml alone, so two of the three configs contributing nothing is undetectable.- Relevance/path filters miss inputs the gated jobs consume:
packages/eql/packages/**(the npm package whosesrc/generated/**freshness therust-cratesjob gates),packages/eql/docker/**, and.github/actions/require-cs-secretsin both test-eql and bench-eql; bench's push paths also omitpackages/eql/mise.toml. packages/protect-ffi/Cargo.lockis stale at HEAD (records eql-bindings 3.0.4; the crate says 3.0.5) and will re-stale on every future lockstep bump —sync-lockstep-versions.mjstouches the Cargo.toml but not this lock. Regenerate here and add the lock (or a--lockedfreshness check) to the sync script.changeset:versionbypasses the lockstep hook. The root alias still runs barechangeset version, and AGENTS.md documents it as the release flow — a human using it reproduces exactly the skew the hook prevents.lint-no-eql-registry-pins.mjsnever scanspnpm-workspace.yaml, where pnpm 10 acceptsoverrides:— the quietest way to re-pin what installs while every package.json still readsworkspace:^. (Nested npm overrides and renamed/aliased declarations also slip through; lower priority.)- AGENTS.md rule 7 not followed: the Repository Layout gained no
packages/eqlentry andSECURITY.mdis untouched, despite this adding the largest package in the repo — the protect-ffi absorption set the precedent (and this package has at least as many working-on-it gotchas worth recording: the nested npm package, the mise task surface, the CI-workspace-keyed test constants).
Nits / observed
- The PR body is stale at HEAD in ways that matter to a reviewer: "Changesets: None" (two live changesets now ship, and via the
fixedgroup they release the whole Stack set), and "Draft on purpose" (now ready for review). - The re-emitted baseline keeps
createdAt: 2026-07-14with new bytes and a newmigrationHash— misleading provenance. test-eql.yml's build-archive fork-guard comment references apush:trigger that doesn't exist; the clean-DB smoke step runs undershell: bash {0}without-ewhile its sibling gotset -euo pipefail.- No test covers the consumer who ignores the changeset's delete-and-re-plan instruction (stale vendored
migrations/cipherstash/against the new package) — whether that fails clearly or confusingly is unverified.
Happy to re-review quickly once the four blockers are addressed — 1 and 3 are small mechanical fixes; 2 needs a decision (publish exclusion vs. gate vs. doing the repoint first); 4 is repo settings plus either a trigger or a schedule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Absorbs the EQL subsystem from
cipherstash/encrypt-query-languageinto this repo, and closes the version skew between the Rust that emits EQL payloads and the SQL that stores them.Draft on purpose. Phases 1–4 are done. Phase 3 has one open item that no machine without Docker and
CS_*can close, and Phase 5 (release cutover) has not started. Opened now so CI runs the build — several jobs here have never executed anywhere.docs/plans/2026-08-13-eql-monorepo-absorption.mdis the authoritative artifact. Every decision is recorded there with its reasoning, including the ones that turned out to be wrong. The commit messages carry the rest — each says what was done and, more usefully, what the instruction got wrong.What lands
A verbatim-prefix subtree import at
packages/eql/(1844 commits, 975 files, ~13 MiB packed). No path rewriting:sync-generated.mjs, the 887-linemise.toml, the 47 scripts undertasks/,Doxyfile,docker-compose.ymland every workflow path filter are repo-root-relative, so a verbatim prefix keeps them resolving with aworking-directorychange and nothing else. The npm package therefore lands atpackages/eql/packages/eql, on thepackages/protect-ffi/platforms/*precedent.eql-bindingsnow resolves by path, not from crates.io. This is the reason for the import.packages/protect-ffipinned=3.0.2while the EQL tree carried 3.0.4 — the emitting Rust on a different release from the storing SQL, with nothing asserting they agree. Today that skew is benign, which is exactly why now: 3.0.2, 3.0.4 and the tree are byte-identical acrosssrc/,bindings/andschema/(diff -rq); what 3.0.3 and 3.0.4 changed was SQL. The flip lands with zero behaviour change, and from here the skew is unrepresentable.Three EQL workflows now run from the root. They arrived under
packages/eql/.github/, a directory GitHub never reads — the same defect the protect-ffi absorption kept finding, where a check arrives as a file and executes on no event.test-eql.yml(the SQLx matrix),bench-eql.ymlandmacro-expand-eql.ymlare ported, path-filtered topackages/eql/**, credentialed to this repo'svars/secretssplit, and fork-PR-skipped.Review order
The diff is 985 files, and almost all of it is the import. Suggested order:
docs/plans/2026-08-13-eql-monorepo-absorption.md— the plan, with results recorded inline.8fda740c— the workspace wiring. The load-bearing commit.3830f066,e614cf72,7f937286— the CI port and the guard that keeps it honest.7a9604c6,e275b093— the path-dep flip and its linter.e54aa5b6— the import itself. Verified faithful (975 in, 975 out) before anything else touched it; nothing to read line by line.57dd5bf6is a pure Biome 1.8.3 → 2.5.2 reflow, committed separately so it never confounds a semantic diff. It is ordered afterf29fdfb0deliberately: the imported tree carries 211 generated.tsfiles that Biome would rewrite into a shape thets-rsgenerator never emits, which breaksmise run types:checkuntil reverted.Things worth a reviewer's attention
Root
pnpm testnow uses--filter './packages/**'. Turbo's one-level glob selectedpackages/eqland notpackages/eql/packages/eql, so@cipherstash/eql's own Vitest suite would have run nowhere.buildwas already covered by^buildtraversal;testwas not. Verified safe because the sixplatforms/*manifests declare noscriptsblock. Mutation-checked both directions.The lockstep version hook moved to the root.
scripts/sync-lockstep-versions.mjsis what writes the computed npm version intoCargo.toml, and Changesets only runs the rootversionscript.release.ymlnow passesversion: pnpm run version— without that line the hook is present, plausible, and dead. Asserted byscripts/__tests__/release-version-hook.test.mjs.The private
@cipherstash/eql-workspaceroot manifest is deleted.packages/*already matchespackages/eql, so leaving it enrols a second workspace root. Its ten scripts sort into four groups and onlyversionsurvives the move; the subtree needs nonode_modulesfor its mise tasks at all.Four new guards, each mutation-checked. In this repo's house style — discovery over the tree rather than a hand-maintained list, with a floor so a scan that matches nothing fails instead of passing, and exemptions that carry a written reason and go stale loudly:
scripts/__tests__/eql-suite-ci.test.mjs— every mise task that invokes cargo must be reached by a root workflow, throughdependsor atasks/*.shhop. It also holds the un-portedpackages/eql/.github/deposit as an equality, so the final deletion in Phase 5 is forced rather than remembered.scripts/lint-no-eql-registry-pins.mjs— no manifest may nameeql-bindingsor@cipherstash/eqlby registry version.supply-chain.e2e.test.tsasserts coverage per ecosystem, andcargowas already covered by protect-ffi — sopackages/eql/Cargo.lockread as monitored while nothing proposed updates for it.workflow-dispatch-job-conditions.test.mjs(EQL's jobs&&the fork guard with a relevance gate, so the single-spelling rule now applies to the clause) andffi-binding-step-order.test.mjs(EQL's credentialed jobs encrypt through thecipherstash-clientcrate and never loadindex.node, so they take aBINDING_EXEMPT_JOBSentry with that reason — but not a pre-flight exemption; they runrequire-cs-secrets).Verification
Run locally, on this branch:
cargo build -p protect-ffi(host)cargo build -p protect-ffi --target wasm32-unknown-unknownpnpm --filter @cipherstash/protect-ffi test:cargocargo fmt --checkcleanpnpm run test:scriptspnpm run code:checkpnpm install --frozen-lockfilepnpm testunder aPATHtrap for cargoThe wasm32 build was not in the plan's list and should have been: the EQL workspace never builds for that target, and merging the two Cargo workspaces was rejected partly on that ground. It is where a cross-workspace path dep would break first. It does not break.
What is NOT verified
The credentialed
integration-testssuite has not run against the path dep. It needs Docker andCS_*, and it is the only path in the repo that carries an EQL payload through Postgres — so it is what turns "no-op" from an argument about byte-identity into an observed result. The plan's box is deliberately split in two so the runnable half being green cannot be mistaken for the whole thing. Do not merge without it.The SQLx matrix's first run in this repo is also unproven by construction — that is what this PR being open is for.
Deliberately out of scope
cipherstash/encrypt-query-language, and four of the ten imported workflows are the release machinery — sopackages/eql/.github/is kept until Phase 5 ports them, rather than reconstructing a publish pipeline from git history at the one moment nobody wants to be doing that.target/directory, and it would make EQL'sdead_code = "deny"anddefault-membersrepo-wide policy as a side effect of a move. The lockfiles already agree oncipherstash-client(=0.42.0, matching checksums).packages/protect-ffi/integration-testsinto the workspace. It moves@cipherstash/auth,vitestand the last@cipherstash/eqlregistry pin at once, and only a credentialed run can show that is neutral. It carries a named, reasoned exemption in the new linter until then.Changesets
None.
@cipherstash/eqlis not published from here yet, and the protect-ffi change is byte-identical Rust with no observable behaviour change. Three changesets remain parked as.changeset/*.md.deferred, including EQL's ownrename-ste-vec-contains; the cutover PR renames all of them back.