Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .changeset/safe-eql-reinstall.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
"stash": minor
---

Preserve encrypted data and reconstruct functional indexes when reinstalling EQL v3, while refusing unsupported external dependencies before mutation.

`stash eql install` and `stash eql upgrade` now capture dependent functional
indexes before replacing the disposable EQL schemas, then restore and verify
their definitions, clustering, replica-identity role, comments, explicit
statistics targets, and health in the same transaction. A
reconstruction failure rolls the replacement back.
PostgreSQL derives index ownership from the table owner, so reinstall verifies
the resulting owner and rolls back on a mismatch rather than independently
restoring ownership.
Unsupported dependencies—including views, policies, constraints, and
partitioned indexes—are named and refused before mutation.

Reinstall remains a maintenance-window operation: its advisory lock serializes
`stash` lifecycle commands, not unrelated database DDL. Generated EQL migrations
contain the raw bundle and do not include these reinstall protections.
21 changes: 16 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,23 @@ jobs:
strategy:
matrix:
node-version: [22, 24]
postgres-version: [16, 17]
exclude:
- node-version: 22
postgres-version: 17
- node-version: 24
postgres-version: 16

# Postgres + EQL for the integration tests. Official EQL image —
# PostgreSQL 17 with EQL pre-installed via /docker-entrypoint-initdb.d.
# PostgreSQL 16 and 17 with EQL pre-installed via
# /docker-entrypoint-initdb.d. Keeping one supported Node line on each
# server version exercises the pre-PG17 attstattarget representation
# without adding a third full test leg.
# Pinned to eql-2.3.1 to match the EQL payload format the code emits
# (protect-ffi 0.23.x); bump in lockstep with the protect-ffi upgrade.
services:
postgres:
image: ghcr.io/cipherstash/postgres-eql:17-2.3.1
image: ghcr.io/cipherstash/postgres-eql:${{ matrix.postgres-version }}-2.3.1
env:
POSTGRES_USER: cipherstash
POSTGRES_PASSWORD: password
Expand Down Expand Up @@ -333,11 +342,13 @@ jobs:
# (`installer/__tests__/verify.live.test.ts`) would run in no CI
# workflow at all: a routine `@cipherstash/eql` bump could then make
# every `stash eql install` fail with phantom damage, on green CI.
# These suites need Postgres only, no CipherStash credentials; the
# verify suite installs EQL v3 into its own schemas, which coexists
# Most suites need Postgres only. The encrypted-index upgrade suite also
# loads this job's CipherStash credentials from packages/stack/.env and
# uses the binding built above to create genuine ciphertext. The verify
# suite installs EQL v3 into its own schemas, which coexists
# with the image's pre-installed EQL v2 that the stack tests use.
# They share that one database, so the CLI vitest config runs them
# serially (the `live` project sets `fileParallelism: false`
# serially (the `live` project uses a single fork
# verify.live's bundle install opens with DROP SCHEMA … CASCADE, which
# races destructively under the other suites in parallel forks).
# (`supabase-push.live.test.ts` gates on different env vars and still
Expand Down
25 changes: 25 additions & 0 deletions docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
status: accepted
---

# Keep encrypted data durable and EQL schemas disposable

EQL data-bearing domains live in `public` and must survive install, uninstall,
and reinstall, while the `eql_v3` and `eql_v3_internal` schemas remain
disposable and may be dropped with `CASCADE`. Search indexes are derived state:
tooling must capture, rebuild, and verify them around reinstall. Tooling must
refuse before mutation when it finds customer-owned dependencies such as
policies, constraints, or views that it cannot reconstruct safely. This follows
the EQL v2 persistence boundary and deliberately rejects brittle
object-by-object in-place upgrades and permanently versioned implementation
schemas.

## Consequences

- Losing an encrypted application column or stored encrypted value during any
EQL lifecycle operation is a correctness failure.
- Reinstall may incur an explicit, potentially expensive index rebuild.
- Index restoration failures are loud and actionable; they never degrade
silently to sequential scans.
- Changes that make an existing index definition invalid require operator
intervention rather than guessed migration semantics.
15 changes: 14 additions & 1 deletion packages/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,16 @@ npx stash eql upgrade [options]
| `--dry-run` | Show what would happen without making changes |
| `--supabase` | Use Supabase-compatible upgrade |

The install SQL is idempotent and safe to re-run. If EQL is not installed, the command suggests running `npx stash eql install` instead.
Encrypted columns and rows live outside the disposable EQL schemas. Before
replacing those schemas, the CLI captures dependent functional indexes and then
restores their definitions and supported catalog properties in the same
transaction. It refuses unsupported dependencies before mutation and rolls back
if restoration fails. If EQL is absent, the command suggests
`npx stash eql install` instead.

Run upgrade in a schema-migration maintenance window. Its advisory lock prevents
overlapping `stash` lifecycle commands, but unrelated sessions must not create,
alter, or drop EQL-backed indexes while replacement is running.

---

Expand Down Expand Up @@ -305,6 +314,10 @@ Reads `databaseUrl` from `stash.config.ts`.

Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema.

**Generated migrations contain the raw EQL bundle, not the CLI's reinstall
protocol.** A first install is safe. To replace an existing installation, use
`eql upgrade` or recreate every dependent object in the same migration.

### Drizzle

```bash
Expand Down
2 changes: 2 additions & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@
}
},
"devDependencies": {
"@cipherstash/eql-upgrade-baseline": "npm:@cipherstash/eql@3.0.2",
"@cipherstash/protect-ffi": "workspace:*",
"@cipherstash/stack": "workspace:*",
"@types/pg": "^8.23.1",
"node-pty": "^1.1.0",
Expand Down
Loading
Loading