diff --git a/.changeset/safe-eql-reinstall.md b/.changeset/safe-eql-reinstall.md new file mode 100644 index 000000000..8f604fa59 --- /dev/null +++ b/.changeset/safe-eql-reinstall.md @@ -0,0 +1,20 @@ +--- +"stash": minor +--- + +Preserve encrypted data and reconstruct functional indexes when reinstalling EQL v3, while refusing unsupported external dependencies before mutation. + +`stash eql install` and `stash eql upgrade` now capture dependent functional +indexes before replacing the disposable EQL schemas, then restore and verify +their definitions, clustering, replica-identity role, comments, explicit +statistics targets, and health in the same transaction. A +reconstruction failure rolls the replacement back. +PostgreSQL derives index ownership from the table owner, so reinstall verifies +the resulting owner and rolls back on a mismatch rather than independently +restoring ownership. +Unsupported dependencies—including views, policies, constraints, and +partitioned indexes—are named and refused before mutation. + +Reinstall remains a maintenance-window operation: its advisory lock serializes +`stash` lifecycle commands, not unrelated database DDL. Generated EQL migrations +contain the raw bundle and do not include these reinstall protections. diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a1d915d7d..27bf45bae 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -52,14 +52,23 @@ jobs: strategy: matrix: node-version: [22, 24] + postgres-version: [16, 17] + exclude: + - node-version: 22 + postgres-version: 17 + - node-version: 24 + postgres-version: 16 # Postgres + EQL for the integration tests. Official EQL image — - # PostgreSQL 17 with EQL pre-installed via /docker-entrypoint-initdb.d. + # PostgreSQL 16 and 17 with EQL pre-installed via + # /docker-entrypoint-initdb.d. Keeping one supported Node line on each + # server version exercises the pre-PG17 attstattarget representation + # without adding a third full test leg. # Pinned to eql-2.3.1 to match the EQL payload format the code emits # (protect-ffi 0.23.x); bump in lockstep with the protect-ffi upgrade. services: postgres: - image: ghcr.io/cipherstash/postgres-eql:17-2.3.1 + image: ghcr.io/cipherstash/postgres-eql:${{ matrix.postgres-version }}-2.3.1 env: POSTGRES_USER: cipherstash POSTGRES_PASSWORD: password @@ -333,11 +342,13 @@ jobs: # (`installer/__tests__/verify.live.test.ts`) would run in no CI # workflow at all: a routine `@cipherstash/eql` bump could then make # every `stash eql install` fail with phantom damage, on green CI. - # These suites need Postgres only, no CipherStash credentials; the - # verify suite installs EQL v3 into its own schemas, which coexists + # Most suites need Postgres only. The encrypted-index upgrade suite also + # loads this job's CipherStash credentials from packages/stack/.env and + # uses the binding built above to create genuine ciphertext. The verify + # suite installs EQL v3 into its own schemas, which coexists # with the image's pre-installed EQL v2 that the stack tests use. # They share that one database, so the CLI vitest config runs them - # serially (the `live` project sets `fileParallelism: false` — + # serially (the `live` project uses a single fork — # verify.live's bundle install opens with DROP SCHEMA … CASCADE, which # races destructively under the other suites in parallel forks). # (`supabase-push.live.test.ts` gates on different env vars and still diff --git a/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md new file mode 100644 index 000000000..8e62de2fd --- /dev/null +++ b/docs/adr/0001-eql-data-survives-disposable-schema-reinstall.md @@ -0,0 +1,25 @@ +--- +status: accepted +--- + +# Keep encrypted data durable and EQL schemas disposable + +EQL data-bearing domains live in `public` and must survive install, uninstall, +and reinstall, while the `eql_v3` and `eql_v3_internal` schemas remain +disposable and may be dropped with `CASCADE`. Search indexes are derived state: +tooling must capture, rebuild, and verify them around reinstall. Tooling must +refuse before mutation when it finds customer-owned dependencies such as +policies, constraints, or views that it cannot reconstruct safely. This follows +the EQL v2 persistence boundary and deliberately rejects brittle +object-by-object in-place upgrades and permanently versioned implementation +schemas. + +## Consequences + +- Losing an encrypted application column or stored encrypted value during any + EQL lifecycle operation is a correctness failure. +- Reinstall may incur an explicit, potentially expensive index rebuild. +- Index restoration failures are loud and actionable; they never degrade + silently to sequential scans. +- Changes that make an existing index definition invalid require operator + intervention rather than guessed migration semantics. diff --git a/packages/cli/README.md b/packages/cli/README.md index 019601d54..cf68abf9f 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -210,7 +210,16 @@ npx stash eql upgrade [options] | `--dry-run` | Show what would happen without making changes | | `--supabase` | Use Supabase-compatible upgrade | -The install SQL is idempotent and safe to re-run. If EQL is not installed, the command suggests running `npx stash eql install` instead. +Encrypted columns and rows live outside the disposable EQL schemas. Before +replacing those schemas, the CLI captures dependent functional indexes and then +restores their definitions and supported catalog properties in the same +transaction. It refuses unsupported dependencies before mutation and rolls back +if restoration fails. If EQL is absent, the command suggests +`npx stash eql install` instead. + +Run upgrade in a schema-migration maintenance window. Its advisory lock prevents +overlapping `stash` lifecycle commands, but unrelated sessions must not create, +alter, or drop EQL-backed indexes while replacement is running. --- @@ -305,6 +314,10 @@ Reads `databaseUrl` from `stash.config.ts`. Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema. +**Generated migrations contain the raw EQL bundle, not the CLI's reinstall +protocol.** A first install is safe. To replace an existing installation, use +`eql upgrade` or recreate every dependent object in the same migration. + ### Drizzle ```bash diff --git a/packages/cli/package.json b/packages/cli/package.json index 879cc0f09..881f69626 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -75,6 +75,8 @@ } }, "devDependencies": { + "@cipherstash/eql-upgrade-baseline": "npm:@cipherstash/eql@3.0.2", + "@cipherstash/protect-ffi": "workspace:*", "@cipherstash/stack": "workspace:*", "@types/pg": "^8.23.1", "node-pty": "^1.1.0", diff --git a/packages/cli/src/__tests__/installer.test.ts b/packages/cli/src/__tests__/installer.test.ts index 4b4191226..fbb5457ec 100644 --- a/packages/cli/src/__tests__/installer.test.ts +++ b/packages/cli/src/__tests__/installer.test.ts @@ -1,4 +1,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + RecordingRestorationDatabase, + searchIndexRestorationScenario, +} from '../installer/__tests__/restoration-scenarios.js' const mockConnect = vi.fn() const mockQuery = vi.fn() @@ -8,12 +12,43 @@ vi.mock('pg', () => ({ default: { Client: vi.fn(() => ({ connect: mockConnect, - query: mockQuery, + query: async (...args: unknown[]) => { + const result = await mockQuery(...args) + if ( + typeof args[0] === 'string' && + args[0].includes('pg_try_advisory') && + result?.rows?.[0]?.acquired === undefined + ) { + return { ...result, rows: [{ acquired: true }] } + } + return result + }, end: mockEnd, })), }, })) +/** + * Lets one test make the bundle parser throw the way a bundle the parser has + * outgrown would ({@link assertEveryStatementModelled}). Through `vi.hoisted` + * because the factory below is hoisted above every other top-level binding; + * everything else keeps the real parse. + */ +const { parseFailure } = vi.hoisted(() => ({ + parseFailure: { error: null as Error | null }, +})) + +vi.mock('../installer/verify.js', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + loadVerifiedEqlBundle: () => { + if (parseFailure.error) throw parseFailure.error + return actual.loadVerifiedEqlBundle() + }, + } +}) + /** A full preflight row with every capability present. */ const CAPABLE_ROW = { role_name: 'postgres', @@ -30,7 +65,10 @@ const CAPABLE_ROW = { } describe('EQLInstaller', () => { - beforeEach(() => vi.clearAllMocks()) + beforeEach(() => { + vi.clearAllMocks() + parseFailure.error = null + }) afterEach(() => vi.restoreAllMocks()) it('reports a fully-capable superuser with no gaps', async () => { @@ -212,25 +250,149 @@ describe('EQLInstaller', () => { const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) - mockQuery.mockResolvedValue({ rows: [{ found: 2 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v3')")) { + return Promise.resolve({ + rows: [{ installed: true }], + rowCount: 1, + }) + } + if (sql.includes('eql_v3.version()')) { + return Promise.resolve({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + } + return Promise.resolve({ + rows: [{ ore_opclass_present: true, poisoned_domains: 0 }], + rowCount: 1, + }) + }) await expect(installer.isInstalled()).resolves.toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [ - ['eql_v3', 'eql_v3_internal'], - ]) + expect(mockQuery).toHaveBeenCalledTimes(1) - mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v3')")) { + return Promise.resolve({ rows: [{ installed: false }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) await expect(installer.isInstalled()).resolves.toBe(false) }) it('retains read-only EQL v2 installation detection for status', async () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) - mockQuery.mockResolvedValue({ rows: [{ found: 1 }], rowCount: 1 }) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return Promise.resolve({ rows: [{ installed: true }], rowCount: 1 }) + } + if (sql.includes('eql_v2.version()')) { + return Promise.resolve({ rows: [{ version: '2.3.1' }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) const { EQLInstaller } = await import('@/installer/index.ts') const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) await expect(installer.isInstalled({ eqlVersion: 2 })).resolves.toBe(true) - expect(mockQuery).toHaveBeenCalledWith(expect.any(String), [['eql_v2']]) + expect(mockQuery).toHaveBeenCalledTimes(1) + expect(mockQuery).toHaveBeenCalledWith( + expect.stringContaining("to_regnamespace('eql_v2')"), + ) + }) + + it('reads a legacy installed version', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery + .mockResolvedValueOnce({ rows: [{ installed: true }], rowCount: 1 }) + .mockResolvedValueOnce({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.getInstalledVersion()).resolves.toBe('3.0.5') + }) + + it('reads the version when the schema exists but is hidden from information_schema', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('to_regnamespace')) { + return Promise.resolve({ rows: [{ installed: true }], rowCount: 1 }) + } + if (sql.includes('information_schema.schemata')) { + return Promise.resolve({ rows: [], rowCount: 0 }) + } + if (sql.includes('eql_v3.version()')) { + return Promise.resolve({ rows: [{ version: '3.0.5' }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ + databaseUrl: 'postgres://test', + }).getInstalledVersion(), + ).resolves.toBe('3.0.5') + }) + + it('reports unknown for an installed legacy schema without version()', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery + .mockResolvedValueOnce({ rows: [{ installed: true }], rowCount: 1 }) + .mockRejectedValueOnce( + Object.assign(new Error('undefined function'), { code: '42883' }), + ) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ + databaseUrl: 'postgres://test', + }).getInstalledVersion(), + ).resolves.toBe('unknown') + }) + + it('frames connection failures from legacy installation detection', async () => { + mockConnect.mockRejectedValue(new Error('connection refused')) + mockEnd.mockResolvedValue(undefined) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).isInstalled(), + ).rejects.toThrow('Failed to connect to database: connection refused') + }) + + it('preserves lifecycle lock timeout guidance without rollback narration', async () => { + vi.useFakeTimers() + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockImplementation((sql: string) => { + if (sql.includes('pg_try_advisory_xact_lock')) { + return Promise.resolve({ rows: [{ acquired: false }], rowCount: 1 }) + } + return Promise.resolve({ rows: [], rowCount: 0 }) + }) + try { + const { EQLInstaller } = await import('@/installer/index.ts') + const { EqlLifecycleLockTimeoutError } = await import( + '../installer/derived-search-index-restoration.js' + ) + const installing = new EQLInstaller({ + databaseUrl: 'postgres://test', + }).install() + const outcome = installing.catch((error: unknown) => error) + + await vi.advanceTimersByTimeAsync(300_000) + + const error = await outcome + expect(error).toBeInstanceOf(EqlLifecycleLockTimeoutError) + expect(error).not.toHaveProperty( + 'message', + expect.stringMatching(/Failed to install EQL/), + ) + } finally { + vi.useRealTimers() + } }) it('installs only the pinned EQL v3 bundle', async () => { @@ -246,14 +408,292 @@ describe('EQLInstaller', () => { const sqlCall = mockQuery.mock.calls.find( ([sql]) => - typeof sql === 'string' && - !['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql), + typeof sql === 'string' && sql.includes('CREATE SCHEMA eql_v3'), ) expect(sqlCall?.[0]).toContain('eql_v3') expect(sqlCall?.[0]).not.toContain('CREATE SCHEMA eql_v2') expect(mockQuery).toHaveBeenCalledWith('COMMIT') }) + it('captures, rebuilds, and verifies functional indexes around reinstall', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario(), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await installer.install() + + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + 'reconstruct', + 'analyze', + 'verify', + 'commit', + ]) + }) + + it('preserves the captured validity state when verifying rebuilt indexes', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + valid: false, + ready: false, + clustered: true, + clusterSql: 'ALTER TABLE app.users CLUSTER ON users_email_idx', + }), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).resolves.toEqual({ deferredGrantsSql: null }) + expect(database.events).toContain('cluster') + }) + + it('restores catalog state attached to a functional index', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + comment: 'Supports encrypted email equality searches', + commentSql: + "COMMENT ON INDEX app.users_email_idx IS 'Supports encrypted email equality searches'", + }), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + expect(database.events).toContain('comment') + expect(database.events.indexOf('comment')).toBeLessThan( + database.events.indexOf('verify'), + ) + }) + + it('restores explicit per-column index statistics targets', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ + statisticsTargets: [750], + statisticsSql: [ + 'ALTER INDEX app.users_email_idx ALTER COLUMN 1 SET STATISTICS 750', + ], + }), + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + expect(database.events).toContain('statistics') + expect(database.events.indexOf('statistics')).toBeLessThan( + database.events.indexOf('verify'), + ) + }) + + it('captures dependencies before destructive SQL in the protected transaction', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase() + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await new EQLInstaller({ databaseUrl: 'postgres://test' }).install() + + expect(database.events.slice(0, 5)).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + ]) + }) + + it('opens a transaction before setup failures use rollback narration', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase(undefined, { + failConfigurationWith: new Error('setting unavailable'), + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /Failed to install EQL: setting unavailable.*rolled back/s, + ) + expect(database.events).toEqual(['begin', 'lock', 'configure', 'rollback']) + }) + + it('reports a bundle the parser cannot model without a transaction narration', async () => { + parseFailure.error = new Error( + 'The EQL install SQL contains a statement the expected-surface parser does not model, at line 12: `CREATE PROCEDURE eql_v3.reindex()`.', + ) + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + mockQuery.mockResolvedValue({ rows: [], rowCount: 0 }) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + const error: unknown = await installer.install().then( + () => null, + (thrown: unknown) => thrown, + ) + + expect(error).toBeInstanceOf(Error) + const message = (error as Error).message + // The parser's own message names the statement and the remedy. Wrapping it + // in the install's "nothing was applied / rolled back" narration buries + // that behind a database story about a transaction that never opened. + expect(message).toContain('does not model') + expect(message).toContain('CREATE PROCEDURE eql_v3.reindex()') + expect(message).not.toContain('Failed to install EQL') + expect(message).not.toContain('rolled back') + expect(mockQuery).not.toHaveBeenCalledWith('BEGIN') + expect(mockQuery).not.toHaveBeenCalledWith('ROLLBACK') + // A bundle this CLI cannot read is a local defect, like a failed digest + // check: it must not reach the database at all. + expect(mockConnect).not.toHaveBeenCalled() + }) + + it('refuses before mutation when a dependency cannot be reconstructed', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase(undefined, { + unsafeIdentity: 'policy app.users_visible', + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + const installer = new EQLInstaller({ databaseUrl: 'postgres://test' }) + + await expect(installer.install()).rejects.toThrow( + /refused before making changes.*policy app\.users_visible/s, + ) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'rollback', + ]) + }) + + it('refuses before mutation when index catalog metadata is incomplete', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase(undefined, { + incompleteCaptureMetadata: true, + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /refused before making changes.*incomplete catalog metadata/s, + ) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'rollback', + ]) + }) + + it('rolls back schema replacement when index rebuild fails', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const scenario = searchIndexRestorationScenario({ + definition: + 'CREATE INDEX users_email_idx ON app.users (eql_v3.eq_term(email))', + }) + const database = new RecordingRestorationDatabase(scenario, { + failReconstructionWith: new Error('disk full'), + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow( + /transaction will restore.*Captured index SQL:\nCREATE INDEX users_email_idx/s, + ) + expect(database.events).toEqual([ + 'begin', + 'lock', + 'configure', + 'capture', + 'replace', + 'reconstruct', + 'rollback', + ]) + }) + + it('rolls back schema replacement when rebuilt index verification fails', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const scenario = searchIndexRestorationScenario() + const database = new RecordingRestorationDatabase(scenario, { + verificationOverrides: { valid: false }, + }) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + + it('rolls back when a rebuilt index has a different owner', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario(), + { verificationOverrides: { owner: 'unexpected_owner' } }, + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + + it('rolls back when a rebuilt index has different statistics targets', async () => { + mockConnect.mockResolvedValue(undefined) + mockEnd.mockResolvedValue(undefined) + const database = new RecordingRestorationDatabase( + searchIndexRestorationScenario({ statisticsTargets: [750] }), + { verificationOverrides: { statisticsTargets: [100] } }, + ) + mockQuery.mockImplementation(database.query) + const { EQLInstaller } = await import('@/installer/index.ts') + + await expect( + new EQLInstaller({ databaseUrl: 'postgres://test' }).install(), + ).rejects.toThrow(/missing, invalid, or changed.*app\.users_email_idx/s) + expect(database.events.slice(-2)).toEqual(['verify', 'rollback']) + expect(database.events).not.toContain('commit') + }) + it('grants both EQL v3 schemas to Supabase roles when the role is a member of postgres', async () => { mockConnect.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { @@ -326,7 +766,7 @@ describe('EQLInstaller', () => { mockConnect.mockResolvedValue(undefined) mockEnd.mockResolvedValue(undefined) mockQuery.mockImplementation((sql: string) => { - if (!['BEGIN', 'COMMIT', 'ROLLBACK'].includes(sql)) { + if (sql.includes('CREATE SCHEMA eql_v3')) { return Promise.reject(new Error('permission denied')) } return Promise.resolve({ rows: [], rowCount: 0 }) diff --git a/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts b/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts index c2871f80b..aff22e72c 100644 --- a/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts +++ b/packages/cli/src/commands/db/__tests__/install-verify-gate.test.ts @@ -41,11 +41,28 @@ vi.mock('@clack/prompts', () => ({ outro: clack.outro, })) -const verifier = vi.hoisted(() => ({ verifyEqlSurface: vi.fn() })) -vi.mock('@/installer/verify.js', () => ({ - verifyEqlSurface: verifier.verifyEqlSurface, +const assessment = vi.hoisted(() => ({ assessEqlInstallation: vi.fn() })) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assessment.assessEqlInstallation, })) +function assessed(report: VerifyReport) { + return { + v2: { status: 'absent' }, + v3: { status: 'installed', version: report.installedVersion ?? 'unknown' }, + ore: { status: 'absent' }, + surface: { + status: + report.status === 'version-mismatch' + ? 'not-comparable' + : report.ok + ? 'complete' + : 'damaged', + report, + }, + } +} + // Imported dynamically by the damage path for its findings renderer. const findingsReporter = vi.hoisted(() => ({ reportVerifyFindings: vi.fn() })) vi.mock('../../eql/verify.js', () => ({ @@ -77,21 +94,23 @@ describe('verifySurfaceOrExit', () => { }) it('returns without exiting on a complete surface', async () => { - verifier.verifyEqlSurface.mockResolvedValueOnce(report({})) + assessment.assessEqlInstallation.mockResolvedValueOnce(assessed(report({}))) await expect( verifySurfaceOrExit('postgres://db', spinner(), { remedy: 'r' }), ).resolves.toBeUndefined() }) it('exits 1 on damage, after reporting the findings and the remedy', async () => { - verifier.verifyEqlSurface.mockResolvedValueOnce( - report({ - status: 'incomplete', - ok: false, - findings: [ - { severity: 'damage', kind: 'operator', message: 'op missing' }, - ], - }), + assessment.assessEqlInstallation.mockResolvedValueOnce( + assessed( + report({ + status: 'incomplete', + ok: false, + findings: [ + { severity: 'damage', kind: 'operator', message: 'op missing' }, + ], + }), + ), ) const exit = vi .spyOn(process, 'exit') @@ -125,7 +144,7 @@ describe('verifySurfaceOrExit', () => { }, ], }) - verifier.verifyEqlSurface.mockResolvedValueOnce(mismatch) + assessment.assessEqlInstallation.mockResolvedValueOnce(assessed(mismatch)) const exit = vi.spyOn(process, 'exit').mockImplementation(() => { throw new Error('exit called') }) @@ -142,7 +161,7 @@ describe('verifySurfaceOrExit', () => { }) it('warns and continues when verification itself errors', async () => { - verifier.verifyEqlSurface.mockRejectedValueOnce( + assessment.assessEqlInstallation.mockRejectedValueOnce( new Error('connection terminated'), ) await expect( diff --git a/packages/cli/src/commands/db/__tests__/install.test.ts b/packages/cli/src/commands/db/__tests__/install.test.ts new file mode 100644 index 000000000..611048b6b --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/install.test.ts @@ -0,0 +1,93 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { CliExit } from '@/cli/exit.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' + +const install = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + spinner: () => spinner, + log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/database-url.js', () => ({ + resolveDatabaseUrl: ({ databaseUrlFlag }: { databaseUrlFlag?: string }) => + databaseUrlFlag ?? 'postgres://test', +})) +vi.mock('@/config/index.js', () => ({ + findConfigFile: () => null, + loadStashConfig: vi.fn(), +})) +vi.mock('../client-scaffold.js', () => ({ ensureEncryptionClient: vi.fn() })) +vi.mock('../config-scaffold.js', () => ({ offerStashConfig: vi.fn() })) +vi.mock('../grants-report.js', () => ({ + reportSupabaseGrantsOutcome: vi.fn(), +})) +vi.mock('@/installer/index.js', () => ({ + EQLInstaller: class { + install = install + }, +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: () => + Promise.resolve({ + v3: { status: 'absent' }, + capabilities: { + status: 'assessed', + preflight: { + ok: true, + currentUser: 'installer', + isSuperuser: true, + memberOfPostgres: false, + missing: [], + }, + }, + }), +})) +vi.mock('../detect.js', () => ({ + detectPrismaNext: () => null, + detectSupabase: () => false, +})) + +describe('installCommand', () => { + beforeEach(() => vi.clearAllMocks()) + + it('renders a reinstall refusal as an expected command failure', async () => { + const refusal = new EqlReinstallRefusalError('reinstall refused') + install.mockRejectedValueOnce(refusal) + + const { installCommand } = await import('../install.js') + await expect( + installCommand({ + databaseUrl: 'postgres://test', + force: true, + scaffoldConfig: 'skip', + }), + ).rejects.toEqual(new CliExit(1)) + + expect(spinner.stop).toHaveBeenLastCalledWith('EQL installation failed.') + expect( + vi.mocked((await import('@clack/prompts')).log.error), + ).toHaveBeenCalledWith('reinstall refused') + }) + + it('preserves an unexpected install error', async () => { + const error = new Error('database disappeared') + install.mockRejectedValueOnce(error) + + const { installCommand } = await import('../install.js') + await expect( + installCommand({ + databaseUrl: 'postgres://test', + force: true, + scaffoldConfig: 'skip', + }), + ).rejects.toBe(error) + }) +}) diff --git a/packages/cli/src/commands/db/__tests__/status.test.ts b/packages/cli/src/commands/db/__tests__/status.test.ts new file mode 100644 index 000000000..b17123dbd --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/status.test.ts @@ -0,0 +1,52 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const assess = vi.fn() +const logError = vi.fn() +const logInfo = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + spinner: () => spinner, + log: { error: logError, info: logInfo, success: vi.fn(), warn: vi.fn() }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: () => ({ databaseUrl: 'postgres://test' }), +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assess, +})) + +describe('statusCommand advisory sections', () => { + beforeEach(() => vi.clearAllMocks()) + + it('continues to ORE when the independent permission assessment fails', async () => { + assess + .mockResolvedValueOnce({ + v2: { status: 'absent' }, + v3: { status: 'installed', version: '3.0.5' }, + capabilities: { status: 'not-requested' }, + ore: { + status: 'observed', + state: 'indexable', + opclassPresent: true, + poisonedDomains: 0, + expectedPoisoned: 20, + }, + surface: { status: 'not-requested' }, + }) + .mockRejectedValueOnce(new Error('permission probe failed')) + + const { statusCommand } = await import('../status.js') + await statusCommand() + + expect(assess).toHaveBeenCalledTimes(2) + expect(logError).toHaveBeenCalledWith('permission probe failed') + expect(logInfo).toHaveBeenCalledWith(expect.stringContaining('usable')) + }) +}) diff --git a/packages/cli/src/commands/db/__tests__/upgrade.test.ts b/packages/cli/src/commands/db/__tests__/upgrade.test.ts new file mode 100644 index 000000000..8f0f109b2 --- /dev/null +++ b/packages/cli/src/commands/db/__tests__/upgrade.test.ts @@ -0,0 +1,74 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' +import { CliExit } from '@/cli/exit.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' + +const assess = vi.fn() +const install = vi.fn() +const logInfo = vi.fn() +const logError = vi.fn() +const spinner = { start: vi.fn(), stop: vi.fn() } + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + spinner: () => spinner, + log: { info: logInfo, warn: vi.fn(), error: logError }, +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: () => 'pnpm', + runnerCommand: (_pm: string, command: string) => command, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: () => ({ databaseUrl: 'postgres://test' }), +})) +vi.mock('@/installer/installation-state.js', () => ({ + assessEqlInstallation: assess, +})) +vi.mock('@/installer/index.js', () => ({ + EQLInstaller: class { + install = install + }, +})) + +describe('upgradeCommand version reporting', () => { + beforeEach(() => vi.clearAllMocks()) + + it('does not call two unknown versions unchanged', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: 'unknown' }, + }) + install.mockResolvedValue({ deferredGrantsSql: null }) + + const { upgradeCommand } = await import('../upgrade.js') + await upgradeCommand({}) + + expect(logInfo).not.toHaveBeenCalledWith( + 'Version unchanged — EQL was already up to date.', + ) + }) + + it('renders a reinstall refusal as an expected command failure', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: '3.0.4' }, + }) + install.mockRejectedValue(new EqlReinstallRefusalError('reinstall refused')) + + const { upgradeCommand } = await import('../upgrade.js') + await expect(upgradeCommand({})).rejects.toEqual(new CliExit(1)) + + expect(spinner.stop).toHaveBeenLastCalledWith('EQL upgrade failed.') + expect(logError).toHaveBeenCalledWith('reinstall refused') + }) + + it('preserves an unexpected upgrade error', async () => { + assess.mockResolvedValue({ + v3: { status: 'installed', version: '3.0.4' }, + }) + const error = new Error('database disappeared') + install.mockRejectedValue(error) + + const { upgradeCommand } = await import('../upgrade.js') + await expect(upgradeCommand({})).rejects.toBe(error) + }) +}) diff --git a/packages/cli/src/commands/db/install.ts b/packages/cli/src/commands/db/install.ts index cd8569ccc..5cd965595 100644 --- a/packages/cli/src/commands/db/install.ts +++ b/packages/cli/src/commands/db/install.ts @@ -1,11 +1,14 @@ import { installMigrationsSchema } from '@cipherstash/migrate' import * as p from '@clack/prompts' +import { CliExit } from '@/cli/exit.js' import { resolveDatabaseUrl } from '@/config/database-url.js' import { findConfigFile, loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' -import { type VerifyReport, verifyEqlSurface } from '@/installer/verify.js' +import type { VerifyReport } from '@/installer/verify.js' import { messages } from '@/messages.js' import { detectPackageManager, runnerCommand } from '../init/utils.js' import { ensureEncryptionClient } from './client-scaffold.js' @@ -141,7 +144,14 @@ export async function installCommand( const installer = new EQLInstaller({ databaseUrl }) s.start('Checking database permissions...') - const permissions = await installer.preflight() + const installation = await assessEqlInstallation({ + databaseUrl, + includeCapabilities: true, + }) + if (installation.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') + } + const permissions = installation.capabilities.preflight if (!permissions.ok) { s.stop('Insufficient database permissions.') p.log.error('The connected database role is missing required permissions:') @@ -163,7 +173,7 @@ export async function installCommand( if (!options.force) { s.start('Checking if EQL is already installed...') - const installed = await installer.isInstalled() + const installed = installation.v3.status === 'installed' s.stop(installed ? 'EQL is already installed.' : 'EQL is not installed.') if (installed) { // Re-apply the grants even when the bundle is present: since the bundle @@ -192,7 +202,18 @@ export async function installCommand( } s.start('Installing EQL v3 extensions (pinned bundle)...') - const installResult = await installer.install({ supabase }) + let installResult: Awaited> + try { + installResult = await installer.install({ supabase }) + } catch (error) { + s.stop('EQL installation failed.') + if (error instanceof EqlReinstallRefusalError) { + p.log.error(error.message) + p.outro('Installation aborted.') + throw new CliExit(1) + } + throw error + } s.stop('EQL extensions installed.') if (supabase) reportSupabaseGrantsOutcome(installResult) @@ -243,7 +264,14 @@ export async function verifySurfaceOrExit( s.start('Verifying the installed EQL surface...') let report: VerifyReport try { - report = await verifyEqlSurface(databaseUrl) + const installation = await assessEqlInstallation({ + databaseUrl, + depth: 'exhaustive', + }) + if (installation.surface.status === 'not-requested') { + throw new Error('Exhaustive EQL assessment returned no surface result') + } + report = installation.surface.report } catch (err) { s.stop('Could not verify the installed EQL surface.') p.log.warn( diff --git a/packages/cli/src/commands/db/status.ts b/packages/cli/src/commands/db/status.ts index 1fc3b25f1..c3eac84c1 100644 --- a/packages/cli/src/commands/db/status.ts +++ b/packages/cli/src/commands/db/status.ts @@ -2,9 +2,8 @@ import * as p from '@clack/prompts' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' -import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' -import { readOreState } from '@/installer/verify.js' export async function statusCommand(options: { databaseUrl?: string } = {}) { const pm = detectPackageManager() @@ -16,29 +15,18 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { const config = await loadStashConfig({ databaseUrlFlag: options.databaseUrl }) s.stop('Configuration loaded.') - const installer = new EQLInstaller({ - databaseUrl: config.databaseUrl, - }) - // 1. Check EQL installation status and version — both generations, so a // v3-only database is not misreported as "not installed" (the v2 check // only looks for the eql_v2 schema). s.start('Checking EQL installation...') - let installedV2: boolean - let installedV3: boolean - let versionV2: string | null - let versionV3: string | null + let installation: Awaited> try { - installedV2 = await installer.isInstalled({ eqlVersion: 2 }) - installedV3 = await installer.isInstalled({ eqlVersion: 3 }) - versionV2 = installedV2 - ? await installer.getInstalledVersion({ eqlVersion: 2 }) - : null - versionV3 = installedV3 - ? await installer.getInstalledVersion({ eqlVersion: 3 }) - : null + installation = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + includeOre: true, + }) } catch (error) { s.stop('Failed.') p.log.error( @@ -50,16 +38,18 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { process.exit(1) } + const installedV2 = installation.v2.status === 'installed' + const installedV3 = installation.v3.status === 'installed' if (installedV2 || installedV3) { s.stop('EQL is installed.') if (installedV2) { p.log.success( - `EQL v2 installed: yes (version: ${versionV2 ?? 'unknown'})`, + `EQL v2 installed: yes (version: ${installation.v2.status === 'installed' ? installation.v2.version : 'unknown'})`, ) } if (installedV3) { p.log.success( - `EQL v3 installed: yes (version: ${versionV3 ?? 'unknown'})`, + `EQL v3 installed: yes (version: ${installation.v3.status === 'installed' ? installation.v3.version : 'unknown'})`, ) } } else { @@ -75,7 +65,14 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { s.start('Checking database permissions...') try { - const permissions = await installer.preflight() + const capabilityAssessment = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + includeCapabilities: true, + }) + if (capabilityAssessment.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') + } + const permissions = capabilityAssessment.capabilities.preflight s.stop('Permissions checked.') if (permissions.ok) { @@ -103,41 +100,28 @@ export async function statusCommand(options: { databaseUrl?: string } = {}) { // half, and reads as 'fallback' on a database that has no EQL at all. if (installedV3) { s.start('Checking ORE operator class...') - const oreClient = createPgClient(config.databaseUrl) - try { - await oreClient.connect() - const ore = await readOreState(oreClient) - s.stop('ORE state checked.') - if (ore.comparable) { - const described = describeOreState(ore.state) - if (described.severity === 'damage') { - p.log.error(described.message) - } else { - p.log.info(described.message) - } + const ore = installation.ore + s.stop('ORE state checked.') + if (ore.status === 'observed') { + const described = describeOreState(ore.state) + if (described.severity === 'damage') { + p.log.error(described.message) } else { - // Version skew is not damage, and must not be rendered as any ORE - // answer at all: the domain list the poison CHECKs are counted over is - // the PINNED bundle's, so a perfectly healthy fallback install of an - // older EQL classifies as incoherent and would send this operator to - // `install --force` over nothing. Say the true thing instead. - p.log.info( - `ORE operator class: not compared — EQL ${ - ore.installedVersion ?? 'unknown' - } is installed and this CLI pins EQL ${ore.bundleVersion}, so the ORE state cannot be read against the pinned bundle. Run \`${runnerCommand(pm, 'stash eql upgrade')}\`, then check status again.`, - ) + p.log.info(described.message) } - } catch (error) { - // Advisory, not a gate: a status run that could not read one row should - // still report everything else it read. - s.stop('ORE state check failed.') - p.log.warn( - `Could not determine the ORE operator class state: ${ - error instanceof Error ? error.message : String(error) - }`, + } else if (ore.status === 'not-comparable') { + // Version skew is not damage, and must not be rendered as any ORE + // answer at all: the domain list the poison CHECKs are counted over is + // the PINNED bundle's, so a perfectly healthy fallback install of an + // older EQL classifies as incoherent and would send this operator to + // `install --force` over nothing. Say the true thing instead. + p.log.info( + `ORE operator class: not compared — EQL ${ + ore.installedVersion ?? 'unknown' + } is installed and this CLI pins EQL ${ore.bundleVersion}, so the ORE state cannot be read against the pinned bundle. Run \`${runnerCommand(pm, 'stash eql upgrade')}\`, then check status again.`, ) - } finally { - await oreClient.end().catch(() => {}) + } else if (ore.status === 'unavailable') { + p.log.warn(`Could not read the ORE operator class state: ${ore.message}`) } } diff --git a/packages/cli/src/commands/db/upgrade.ts b/packages/cli/src/commands/db/upgrade.ts index 74430604f..156923e6e 100644 --- a/packages/cli/src/commands/db/upgrade.ts +++ b/packages/cli/src/commands/db/upgrade.ts @@ -1,7 +1,10 @@ import * as p from '@clack/prompts' +import { CliExit } from '@/cli/exit.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' +import { EqlReinstallRefusalError } from '@/installer/derived-search-index-restoration.js' import { EQLInstaller } from '@/installer/index.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { reportSupabaseGrantsOutcome } from './grants-report.js' export async function upgradeCommand(options: { @@ -22,8 +25,10 @@ export async function upgradeCommand(options: { const installer = new EQLInstaller({ databaseUrl: config.databaseUrl }) s.start('Checking current EQL v3 installation...') - const installed = await installer.isInstalled() - if (!installed) { + const before = await assessEqlInstallation({ + databaseUrl: config.databaseUrl, + }) + if (before.v3.status === 'absent') { s.stop('EQL v3 is not installed.') p.log.warn( `EQL v3 is not currently installed. Run "${runnerCommand(pm, 'stash eql install')}" first.`, @@ -32,7 +37,8 @@ export async function upgradeCommand(options: { process.exit(1) } - const previousVersion = await installer.getInstalledVersion() + const previousVersion = + before.v3.version === 'unknown' ? null : before.v3.version s.stop(`Current version: ${previousVersion ?? 'unknown'}`) if (options.dryRun) { p.log.info('Dry run — no changes will be made.') @@ -45,12 +51,27 @@ export async function upgradeCommand(options: { } s.start('Upgrading EQL v3 extensions (pinned bundle)...') - const result = await installer.install({ supabase: options.supabase }) + let result: Awaited> + try { + result = await installer.install({ supabase: options.supabase }) + } catch (error) { + s.stop('EQL upgrade failed.') + if (error instanceof EqlReinstallRefusalError) { + p.log.error(error.message) + p.outro('Upgrade aborted.') + throw new CliExit(1) + } + throw error + } s.stop('EQL extensions upgraded.') if (options.supabase) reportSupabaseGrantsOutcome(result) s.start('Verifying new version...') - const newVersion = await installer.getInstalledVersion() + const after = await assessEqlInstallation({ databaseUrl: config.databaseUrl }) + const newVersion = + after.v3.status === 'installed' && after.v3.version !== 'unknown' + ? after.v3.version + : null s.stop(`New version: ${newVersion ?? 'unknown'}`) if (previousVersion && newVersion && previousVersion === newVersion) { p.log.info('Version unchanged — EQL was already up to date.') diff --git a/packages/cli/src/commands/eql/migration.ts b/packages/cli/src/commands/eql/migration.ts index 0e9cf3e69..8bec47090 100644 --- a/packages/cli/src/commands/eql/migration.ts +++ b/packages/cli/src/commands/eql/migration.ts @@ -27,8 +27,8 @@ import { tryResolveDatabaseUrl, } from '@/config/database-url.js' import { + emitSupabaseEqlAccessMigration, loadBundledEqlSql, - SUPABASE_MIGRATION_GRANTS_SQL_V3, } from '@/installer/index.js' import { messages } from '@/messages.js' @@ -227,7 +227,7 @@ export interface EqlMigrationOptions { export function buildEqlV3MigrationSql(opts: { supabase: boolean }): string { const eqlSql = loadBundledEqlSql() const grants = opts.supabase - ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${SUPABASE_MIGRATION_GRANTS_SQL_V3.trim()}` + ? `\n\n-- Supabase role grants: let anon/authenticated/service_role use the\n-- eql_v3 + eql_v3_internal schemas (required when tables are reached via\n-- PostgREST/RLS; harmless otherwise).\n${emitSupabaseEqlAccessMigration().trim()}` : '' return `${eqlSql.trim()}${grants}\n\n-- CipherStash encryption-migration tracking schema.\n-- Tracks per-column phase + backfill progress for \`stash encrypt\`.\n${MIGRATIONS_SCHEMA_SQL.trim()}\n` } diff --git a/packages/cli/src/commands/eql/validate.ts b/packages/cli/src/commands/eql/validate.ts index c9ecdbbfc..33d58c717 100644 --- a/packages/cli/src/commands/eql/validate.ts +++ b/packages/cli/src/commands/eql/validate.ts @@ -742,6 +742,43 @@ export async function readObservedState( } } +export interface DeclaredSchemaAssessment { + columns: DeclaredColumn[] + tableCount: number + fidelity: 'complete' | 'config-only' + database: + | { status: 'observed' } + | { + status: 'skipped' + reason: 'not-configured' | 'unreachable' + detail?: string + } + issues: ValidationIssue[] +} + +/** + * Assess the declared encryption schema through one result-oriented interface. + * Declaration normalization, catalogue observation, index parsing, and rule + * classification remain implementation details behind this seam. + */ +export async function assessDeclaredSchema(options: { + encryptConfig: EncryptConfig + schemas?: readonly AnyV3Table[] + databaseUrl?: string +}): Promise { + const columns = options.schemas + ? collectDeclaredColumns(options.schemas) + : collectDeclaredColumnsFromConfig(options.encryptConfig) + const observation = await tryReadObservedState(options.databaseUrl, columns) + return { + columns, + tableCount: new Set(columns.map((column) => column.table)).size, + fidelity: options.schemas ? 'complete' : 'config-only', + database: observation.database, + issues: validateSchemas(columns, observation.observed), + } +} + // --------------------------------------------------------------------------- // Reporting // --------------------------------------------------------------------------- @@ -815,32 +852,37 @@ export async function validateCommand(options: { ) s.stop('Encrypt client loaded.') - const columns = schemas - ? collectDeclaredColumns(schemas) - : collectDeclaredColumnsFromConfig(encryptConfig) + const assessment = await assessDeclaredSchema({ + encryptConfig, + schemas, + databaseUrl: config.databaseUrl, + }) - if (!schemas) { + if (assessment.fidelity === 'config-only') { p.log.warn( 'Your installed @cipherstash/stack does not expose `getSchemas()`, so the concrete EQL domain of each column is unavailable. Domain checks (ORE portability, database drift) were skipped — upgrade @cipherstash/stack to run them.', ) } - const tableCount = new Set(columns.map((column) => column.table)).size p.log.success( - `Schema loaded: ${tableCount} table${tableCount !== 1 ? 's' : ''}, ${columns.length} encrypted column${columns.length !== 1 ? 's' : ''}`, + `Schema loaded: ${assessment.tableCount} table${assessment.tableCount !== 1 ? 's' : ''}, ${assessment.columns.length} encrypted column${assessment.columns.length !== 1 ? 's' : ''}`, ) - const observed = await tryReadObservedState(config.databaseUrl, columns) - - const issues = validateSchemas(columns, observed) + if (assessment.database.status === 'skipped') { + p.log.info( + assessment.database.reason === 'not-configured' + ? 'No database URL resolved — skipping the database checks (drift, ORE availability, functional indexes). Pass --database-url or set DATABASE_URL to run them.' + : `Could not read the database (${assessment.database.detail}) — skipping the database checks (drift, ORE availability, functional indexes). The schema checks below still ran.`, + ) + } - if (issues.length === 0) { + if (assessment.issues.length === 0) { p.outro('No issues found.') return } console.log() // blank line before issues - const hasErrors = reportIssues(issues) + const hasErrors = reportIssues(assessment.issues) if (hasErrors) { process.exit(1) @@ -857,12 +899,12 @@ export async function validateCommand(options: { async function tryReadObservedState( databaseUrl: string | undefined, columns: DeclaredColumn[], -): Promise { +): Promise<{ + observed?: ObservedState + database: DeclaredSchemaAssessment['database'] +}> { if (!databaseUrl) { - p.log.info( - 'No database URL resolved — skipping the database checks (drift, ORE availability, functional indexes). Pass --database-url or set DATABASE_URL to run them.', - ) - return undefined + return { database: { status: 'skipped', reason: 'not-configured' } } } const tables = [...new Set(columns.map((column) => column.table))] @@ -870,13 +912,19 @@ async function tryReadObservedState( try { await client.connect() - return await readObservedState(client, tables) + return { + observed: await readObservedState(client, tables), + database: { status: 'observed' }, + } } catch (error) { const message = error instanceof Error ? error.message : String(error) - p.log.info( - `Could not read the database (${message}) — skipping the database checks (drift, ORE availability, functional indexes). The schema checks below still ran.`, - ) - return undefined + return { + database: { + status: 'skipped', + reason: 'unreachable', + detail: message, + }, + } } finally { await client.end().catch(() => {}) } diff --git a/packages/cli/src/commands/eql/verify.ts b/packages/cli/src/commands/eql/verify.ts index f17cfe0da..d26afad27 100644 --- a/packages/cli/src/commands/eql/verify.ts +++ b/packages/cli/src/commands/eql/verify.ts @@ -2,9 +2,9 @@ import * as p from '@clack/prompts' import { emitJsonError, emitJsonEvent } from '@/commands/auth/events.js' import { resolveDiagnosticDatabaseUrl } from '@/commands/db/resolve-diagnostic-url.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { assessEqlInstallation } from '@/installer/installation-state.js' import { describeOreState } from '@/installer/ore.js' import type { SurfaceFinding, VerifyReport } from '@/installer/verify.js' -import { verifyEqlSurface } from '@/installer/verify.js' /** * `stash eql verify` — assert the installed EQL surface is complete and @@ -46,7 +46,14 @@ export async function verifyCommand( s?.start('Comparing the installed EQL surface with the pinned bundle...') let report: VerifyReport try { - report = await verifyEqlSurface(databaseUrl) + const installation = await assessEqlInstallation({ + databaseUrl, + depth: 'exhaustive', + }) + if (installation.surface.status === 'not-requested') { + throw new Error('EQL surface was not assessed') + } + report = installation.surface.report } catch (error) { const message = error instanceof Error ? error.message : String(error) if (json) { diff --git a/packages/cli/src/installer/__tests__/bundle-digest.test.ts b/packages/cli/src/installer/__tests__/bundle-digest.test.ts index 4bf8477a7..430100699 100644 --- a/packages/cli/src/installer/__tests__/bundle-digest.test.ts +++ b/packages/cli/src/installer/__tests__/bundle-digest.test.ts @@ -59,6 +59,18 @@ describe('bundled EQL SQL digest verification', () => { ) }) + it('presents verified SQL and its derived surface as one artifact', async () => { + eqlSql.tampered = null + const { loadVerifiedEqlBundle } = await import('@/installer/verify.ts') + const { releaseManifest } = await import('@cipherstash/eql/sql') + + const bundle = loadVerifiedEqlBundle() + + expect(bundle.sql).toContain('CREATE SCHEMA eql_v3') + expect(bundle.expectedSurface.eqlVersion).toBe(releaseManifest.eqlVersion) + expect(bundle.expectedSurface.operators.length).toBeGreaterThan(2000) + }) + it('refuses SQL whose bytes do not hash to the manifest digest', async () => { eqlSql.tampered = TAMPERED const { loadBundledEqlSql } = await import('@/installer/index.ts') diff --git a/packages/cli/src/installer/__tests__/installation-state.test.ts b/packages/cli/src/installer/__tests__/installation-state.test.ts new file mode 100644 index 000000000..64975978f --- /dev/null +++ b/packages/cli/src/installer/__tests__/installation-state.test.ts @@ -0,0 +1,248 @@ +import { releaseManifest } from '@cipherstash/eql/sql' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const query = vi.fn() +const connect = vi.fn() +const end = vi.fn() + +vi.mock('@/db/client.js', () => ({ + createPgClient: () => ({ query, connect, end }), + TlsVerificationError: class extends Error {}, +})) + +describe('EQL installation state', () => { + beforeEach(() => { + vi.resetAllMocks() + connect.mockResolvedValue(undefined) + end.mockResolvedValue(undefined) + }) + + it('recovers from a missing version function before continuing its snapshot', async () => { + let aborted = false + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: true, + eql_v3_present: false, + eql_v3_internal_present: false, + }, + ], + } + } + if (sql.includes('eql_v2.version()')) { + aborted = true + throw Object.assign(new Error('undefined function'), { code: '42883' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT eql_version_probe') { + aborted = false + return { rows: [] } + } + if (aborted) { + throw Object.assign(new Error('transaction is aborted'), { + code: '25P02', + }) + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + }) + + expect(state.v2).toEqual({ status: 'installed', version: 'unknown' }) + expect(query).toHaveBeenCalledWith( + 'ROLLBACK TO SAVEPOINT eql_version_probe', + ) + expect(query).toHaveBeenCalledWith('COMMIT') + }) + + it('recovers when exhaustive surface observation finds a missing version function', async () => { + let versionReads = 0 + let aborted = false + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) { + versionReads += 1 + if (versionReads === 1) + return { rows: [{ version: releaseManifest.eqlVersion }] } + aborted = true + throw Object.assign(new Error('undefined function'), { code: '42883' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_version_probe') { + aborted = false + return { rows: [] } + } + if (aborted) + throw Object.assign(new Error('transaction is aborted'), { + code: '25P02', + }) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: true, + eql_v3_internal_present: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: true, poisoned_domains: 0 }] } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + depth: 'exhaustive', + }) + + expect(state.surface.status).toBe('damaged') + expect(query).toHaveBeenCalledWith( + 'ROLLBACK TO SAVEPOINT installed_eql_version_probe', + ) + expect(query).toHaveBeenCalledWith('COMMIT') + }) + + it('reports an unavailable advisory ORE observation without failing installation state', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) + return { rows: [{ version: releaseManifest.eqlVersion }] } + if (sql.includes('ore_opclass_present')) + throw new Error('catalog unavailable') + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + includeOre: true, + }) + + expect(state.ore).toEqual({ + status: 'unavailable', + message: 'catalog unavailable', + }) + expect(query).toHaveBeenCalledWith('COMMIT') + }) + + it('reports one authoritative schema-presence observation across installation and capabilities', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: true, + eql_v3_internal_present: true, + }, + ], + } + } + if (sql.includes('eql_v3.version()')) + return { rows: [{ version: releaseManifest.eqlVersion }] } + if (sql.includes('current_user AS role_name')) { + return { + rows: [ + { + role_name: 'restricted_role', + is_superuser: false, + member_of_postgres: false, + has_database_create: true, + has_public_create: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + // information_schema can hide schemas that to_regnamespace sees. + eql_v3_present: false, + eql_v3_internal_present: false, + can_drop_eql_v3: false, + can_drop_eql_v3_internal: false, + }, + ], + } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + includeCapabilities: true, + }) + + expect(state.v3.status).toBe('installed') + expect(state.capabilities).toMatchObject({ + status: 'assessed', + preflight: { + eqlV3SchemaPresent: true, + eqlV3InternalSchemaPresent: true, + }, + }) + }) + + it('exhaustively reports a missing EQL installation', async () => { + query.mockImplementation(async (sql: string) => { + if (sql.includes("to_regnamespace('eql_v2')")) { + return { + rows: [ + { + eql_v2_present: false, + eql_v3_present: false, + eql_v3_internal_present: false, + }, + ], + } + } + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + return { rows: [] } + }) + + const { assessEqlInstallation } = await import('../installation-state.js') + const state = await assessEqlInstallation({ + databaseUrl: 'postgres://test', + depth: 'exhaustive', + }) + + expect(state.surface.status).toBe('damaged') + if (state.surface.status !== 'damaged') return + expect(state.surface.report.status).toBe('not-installed') + }) +}) diff --git a/packages/cli/src/installer/__tests__/reinstall.live.test.ts b/packages/cli/src/installer/__tests__/reinstall.live.test.ts new file mode 100644 index 000000000..250dc880d --- /dev/null +++ b/packages/cli/src/installer/__tests__/reinstall.live.test.ts @@ -0,0 +1,638 @@ +/** + * Live-Postgres coverage for safe EQL schema replacement. + * + * The catalog dependency graph and pg_get_indexdef() are the public seam: a + * mock cannot prove PostgreSQL records an expression-index or policy dependency + * in the shape our classifier expects. Two of the checks below go further and + * are unreproducible anywhere else: `format_type()`'s search_path sensitivity + * and `ALTER INDEX … ATTACH PARTITION`'s effect on `indisvalid` are behaviours + * of the server, not properties of our SQL text. + */ + +import { afterAll, beforeEach, describe, expect, it } from 'vitest' +import { + derivedSearchIndexRestorationTestSeam, + EqlLifecycleLockTimeoutError, +} from '../derived-search-index-restoration.js' +import { EQLInstaller } from '../index.js' +import { + LiveRestorationDatabase, + searchIndexRestorationScenario, +} from './restoration-scenarios.js' + +const { acquireLifecycleLock } = derivedSearchIndexRestorationTestSeam + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const describeLive = DATABASE_URL ? describe : describe.skip +const postgres = new LiveRestorationDatabase(DATABASE_URL ?? '') + +async function query(sql: string): Promise { + return postgres.query(sql) +} + +/** + * Every index in the test schema, with the two things a partitioned rebuild + * can silently lose: its validity, and which partitioned index it is attached + * to. + */ +const INDEX_STATE_SQL = ` + SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, + c.relkind::text AS relkind, + i.indisvalid AS valid, + i.indisready AS ready, + pg_catalog.pg_get_indexdef(i.indexrelid) AS definition, + ( + SELECT pg_catalog.format('%I.%I', pn.nspname, pc.relname) + FROM pg_catalog.pg_inherits inh + JOIN pg_catalog.pg_class pc ON pc.oid = inh.inhparent + JOIN pg_catalog.pg_namespace pn ON pn.oid = pc.relnamespace + WHERE inh.inhrelid = c.oid + ) AS attached_to + FROM pg_catalog.pg_index i + JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + ORDER BY identity +` + +describeLive('EQLInstaller safe reinstall — live Postgres', () => { + beforeEach(async () => { + await query('DROP EVENT TRIGGER IF EXISTS stash_reinstall_pause_drop') + await query('DROP SCHEMA IF EXISTS stash_reinstall_test CASCADE') + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + await query(` + CREATE SCHEMA IF NOT EXISTS stash_reinstall_test; + DROP TABLE IF EXISTS stash_reinstall_test.records CASCADE; + CREATE TABLE stash_reinstall_test.records ( + id integer PRIMARY KEY, + encrypted public.eql_v3_text_eq NOT NULL + ); + INSERT INTO stash_reinstall_test.records VALUES + (1, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb); + `) + }, 180_000) + + afterAll(async () => { + await query('DROP SCHEMA IF EXISTS stash_reinstall_test CASCADE').catch( + () => undefined, + ) + }) + + it('preserves data and rebuilds a functional index', async () => { + const scenario = searchIndexRestorationScenario({ + identity: 'stash_reinstall_test.records_encrypted_idx', + tableIdentity: 'stash_reinstall_test.records', + definition: + 'CREATE INDEX records_encrypted_idx ON stash_reinstall_test.records (eql_v3.eq_term(encrypted))', + }) + await query(` + ${scenario.definition}; + CREATE UNIQUE INDEX "Records encrypted complex" + ON stash_reinstall_test.records USING btree (eql_v3.eq_term(encrypted)) + INCLUDE (id) WITH (fillfactor = 80) WHERE id > 0; + `) + const identityBefore = await query<{ + table_oid: string + column_number: number + column_type_oid: string + value: unknown + }>(` + SELECT c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c + ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `) + const definitionsBefore = await query<{ + identity: string + definition: string + }>(` + SELECT c.relname AS identity, pg_catalog.pg_get_indexdef(c.oid) AS definition + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + AND c.relname IN ('records_encrypted_idx', 'Records encrypted complex') + ORDER BY c.relname + `) + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + const rows = await query<{ + value: unknown + index_valid: boolean + index_ready: boolean + }>(` + SELECT r.encrypted::jsonb AS value, + i.indisvalid AS index_valid, + i.indisready AS index_ready + FROM stash_reinstall_test.records r + CROSS JOIN pg_catalog.pg_index i + WHERE r.id = 1 + AND i.indexrelid = 'stash_reinstall_test.records_encrypted_idx'::regclass + `) + expect(rows).toEqual([ + { + value: { v: 3, i: {}, c: 'ciphertext', hm: 'term' }, + index_valid: true, + index_ready: true, + }, + ]) + expect( + await query<{ + table_oid: string + column_number: number + column_type_oid: string + value: unknown + }>(` + SELECT c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c + ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `), + ).toEqual(identityBefore) + expect( + await query<{ identity: string; definition: string }>(` + SELECT c.relname AS identity, pg_catalog.pg_get_indexdef(c.oid) AS definition + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'stash_reinstall_test' + AND c.relname IN ('records_encrypted_idx', 'Records encrypted complex') + ORDER BY c.relname + `), + ).toEqual(definitionsBefore) + expect( + await query<{ definition: string }>(` + SELECT pg_catalog.pg_get_indexdef( + 'stash_reinstall_test."Records encrypted complex"'::regclass + ) AS definition + `), + ).toEqual([ + { + definition: expect.stringMatching( + /CREATE UNIQUE INDEX.*INCLUDE \(id\).*fillfactor='80'.*WHERE \(id > 0\)/, + ), + }, + ]) + }, 180_000) + + it('preserves index ownership and explicit statistics targets', async () => { + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + ALTER INDEX stash_reinstall_test.records_encrypted_idx + ALTER COLUMN 1 SET STATISTICS 750; + `) + + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + expect( + await query<{ owner: string; statistics_target: number }>(` + SELECT pg_catalog.pg_get_userbyid(index_class.relowner) AS owner, + attribute.attstattarget AS statistics_target + FROM pg_catalog.pg_class index_class + JOIN pg_catalog.pg_attribute attribute + ON attribute.attrelid = index_class.oid AND attribute.attnum = 1 + WHERE index_class.oid = + 'stash_reinstall_test.records_encrypted_idx'::regclass + `), + ).toEqual([{ owner: 'cipherstash', statistics_target: 750 }]) + }, 180_000) + + it('preserves the default pre-Postgres-17 statistics target representation', async () => { + const [{ server_version_num: serverVersion }] = await query<{ + server_version_num: string + }>('SHOW server_version_num') + if (Number(serverVersion) >= 170000) return + + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + `) + const statisticsTarget = async () => + query<{ statistics_target: number }>(` + SELECT attribute.attstattarget AS statistics_target + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = + 'stash_reinstall_test.records_encrypted_idx'::regclass + AND attribute.attnum = 1 + `) + + await expect(statisticsTarget()).resolves.toEqual([ + { statistics_target: -1 }, + ]) + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + await expect(statisticsTarget()).resolves.toEqual([ + { statistics_target: -1 }, + ]) + }, 180_000) + + /** + * `format_type()` omits the schema whenever the type is visible on the + * current search_path, while the identities parsed out of the bundle always + * carry the qualification the bundle wrote. Read the catalogue with + * `format_type()` and every bundle-owned operator and cast falls out of its + * ownership exemption the moment a connection names `eql_v3` — and the + * installer refuses on a healthy database, listing EQL's own operators as + * customer-owned objects. Only a live server shows this: the sensitivity is + * in `format_type()`, not in our SQL. + */ + it("exempts the bundle's own operators and casts when the EQL schemas are on the search_path", async () => { + const rows = await postgres.withEqlSearchPath().dependencyInventory<{ + dependency_kind: string + identity: string + }>() + const unsafe = rows + .filter((row) => row.dependency_kind !== 'index') + .map((row) => row.identity) + + // Reported as a count plus a sample: the pre-fix failure is ~600 rows, and + // a bare array comparison buries the count that identifies the bug. + expect({ count: unsafe.length, sample: unsafe.slice(0, 3) }).toEqual({ + count: 0, + sample: [], + }) + }, 60_000) + + it('exempts installed EQL operators independently of the pinned bundle', async () => { + const rows = await postgres + .withEqlSearchPath() + .dependencyInventory<{ dependency_kind: string; identity: string }>() + + expect(rows.filter((row) => row.dependency_kind !== 'index')).toEqual([]) + }, 60_000) + + it('exempts installed EQL casts independently of the pinned bundle', async () => { + await query(` + CREATE DOMAIN stash_reinstall_test.legacy_encrypted AS jsonb; + CREATE FUNCTION eql_v3.legacy_query_cast(stash_reinstall_test.legacy_encrypted) + RETURNS eql_v3.query_text_eq + LANGUAGE sql IMMUTABLE STRICT + AS 'SELECT jsonb_build_object(''v'', $1)::eql_v3.query_text_eq'; + CREATE CAST ( + stash_reinstall_test.legacy_encrypted AS eql_v3.query_text_eq + ) WITH FUNCTION eql_v3.legacy_query_cast(stash_reinstall_test.legacy_encrypted); + `) + + const rows = await postgres.dependencyInventory<{ + dependency_kind: string + identity: string + }>() + + expect(rows.filter((row) => row.dependency_kind !== 'index')).toEqual([]) + }, 60_000) + + it('keeps a customer operator unsafe when it duplicates an EQL signature', async () => { + await query(`CREATE OPERATOR stash_reinstall_test.= ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`) + const rows = await postgres.dependencyInventory<{ + dependency_kind: string + identity: string + }>() + const unsafe = rows + .filter((row) => row.dependency_kind !== 'index') + .map((row) => row.identity) + + expect(unsafe).toEqual([ + 'stash_reinstall_test.=(public.eql_v3_text_eq,public.eql_v3_text_eq)', + ]) + }, 60_000) + + it('installs over a connection whose search_path names the EQL schemas', async () => { + await query(` + CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted)); + `) + await expect( + new EQLInstaller({ + databaseUrl: postgres.withEqlSearchPath().url, + }).install(), + ).resolves.toEqual({ deferredGrantsSql: null }) + }, 180_000) + + /** + * A partitioned index is `relkind = 'I'`, its per-partition children are + * separate `relkind = 'i'` rows, and the parent only becomes `indisvalid` + * once every child has been ATTACHed. Recreating the captured definitions + * and stopping there leaves the parent invalid forever. + * + * Two levels deep on purpose: an intermediate partitioned index is both a + * parent and a child. `INDEX_STATE_SQL` proves refusal preserves every name, + * definition, validity flag and attachment rather than partially rebuilding + * the tree. + */ + it('refuses a partitioned index tree before mutation', async () => { + await query(` + CREATE TABLE stash_reinstall_test.partitioned_records ( + id integer, encrypted public.eql_v3_text_eq NOT NULL + ) PARTITION BY RANGE (id); + CREATE TABLE stash_reinstall_test.partitioned_records_a + PARTITION OF stash_reinstall_test.partitioned_records + FOR VALUES FROM (0) TO (100) PARTITION BY RANGE (id); + CREATE TABLE stash_reinstall_test.partitioned_records_a1 + PARTITION OF stash_reinstall_test.partitioned_records_a + FOR VALUES FROM (0) TO (50); + CREATE TABLE stash_reinstall_test.partitioned_records_b + PARTITION OF stash_reinstall_test.partitioned_records + FOR VALUES FROM (100) TO (200); + CREATE INDEX partitioned_encrypted_idx + ON stash_reinstall_test.partitioned_records (eql_v3.eq_term(encrypted)); + INSERT INTO stash_reinstall_test.partitioned_records VALUES + (1, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb), + (150, '{"v":3,"i":{},"c":"ciphertext","hm":"term"}'::jsonb); + `) + const before = await query>(INDEX_STATE_SQL) + // The fixture only means anything if Postgres really built the tree. + expect( + before.filter((row) => row.relkind === 'I').map((row) => row.identity), + ).toEqual([ + 'stash_reinstall_test.partitioned_encrypted_idx', + 'stash_reinstall_test.partitioned_records_a_eq_term_idx', + ]) + + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow(/reinstall refused.*partitioned_encrypted_idx/is) + + expect(await query>(INDEX_STATE_SQL)).toEqual( + before, + ) + expect( + await query<{ count: number }>( + 'SELECT count(*)::int AS count FROM stash_reinstall_test.partitioned_records', + ), + ).toEqual([{ count: 2 }]) + }, 180_000) + + /** + * `pg_advisory_xact_lock` waits forever. A concurrent install, or a session that + * died holding the lock, then makes the command hang with no output — + * indistinguishable from a network stall. + */ + it('refuses instead of hanging when another session holds the lifecycle lock', async () => { + const { default: pg } = await import('pg') + const holder = new pg.Client({ connectionString: DATABASE_URL }) + const blocked = new pg.Client({ connectionString: DATABASE_URL }) + await holder.connect() + await blocked.connect() + try { + // A regression here is a HANG, not a failure — a blocking + // `pg_advisory_xact_lock` never returns, the `finally` below never runs, and + // the leaked lock then blocks every later `install()` in this file. The + // timeout turns that into a failed assertion. It is inert once the + // acquire polls with `pg_try_advisory_xact_lock`, which never waits. + await blocked.query("SET statement_timeout = '10s'") + await holder.query('BEGIN') + await holder.query( + "SELECT pg_catalog.pg_advisory_xact_lock(pg_catalog.hashtext('cipherstash.eql.lifecycle'))", + ) + await blocked.query('BEGIN') + + // An explicit short budget: the behaviour under test is "refuses rather + // than hangs", which does not depend on how long the wait is, and the + // production default is deliberately five minutes. The default's own + // property — that an ordinary queued install still succeeds — is the + // next test. + await expect(acquireLifecycleLock(blocked, 1_000)).rejects.toBeInstanceOf( + EqlLifecycleLockTimeoutError, + ) + await blocked.query('ROLLBACK') + await holder.query('COMMIT') + await blocked.query('BEGIN') + await expect(acquireLifecycleLock(blocked)).resolves.toBeUndefined() + await blocked.query('COMMIT') + } finally { + await blocked.query('ROLLBACK').catch(() => {}) + await holder.query('ROLLBACK').catch(() => {}) + await blocked.end().catch(() => undefined) + await holder.end().catch(() => undefined) + } + }, 30_000) + + /** + * The other half of the lock change, and the one that bites. Bounding the + * wait converts "hangs forever" into a message — but bound it too tightly and + * ordinary queueing becomes a failure, because what holds the lock is a whole + * install (`DROP SCHEMA … CASCADE` plus ~3,000 objects, 10-30s here). A + * five-second budget looked generous and broke exactly this: two installs + * back to back, the second refused. The waiter must still win. + */ + it('queues a second concurrent install instead of refusing it', async () => { + const results = await Promise.all([ + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ]) + expect(results).toEqual([ + { deferredGrantsSql: null }, + { deferredGrantsSql: null }, + ]) + }, 180_000) + + it('restores schemas, data, column identity, and indexes when install fails after DROP SCHEMA', async () => { + await query(`CREATE INDEX records_encrypted_idx + ON stash_reinstall_test.records (eql_v3.eq_term(encrypted))`) + const before = await query<{ + version: string + table_oid: string + column_number: number + column_type_oid: string + value: unknown + index_definition: string + }>(` + SELECT eql_v3.version() AS version, + c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value, + pg_catalog.pg_get_indexdef('stash_reinstall_test.records_encrypted_idx'::regclass) + AS index_definition + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `) + await query(` + CREATE FUNCTION stash_reinstall_test.reject_schema_create() + RETURNS event_trigger LANGUAGE plpgsql AS + 'BEGIN RAISE EXCEPTION ''forced installer failure after DROP SCHEMA''; END'; + CREATE EVENT TRIGGER stash_reinstall_reject_create + ON ddl_command_start WHEN TAG IN ('CREATE SCHEMA') + EXECUTE FUNCTION stash_reinstall_test.reject_schema_create(); + `) + try { + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow(/forced installer failure after DROP SCHEMA/) + } finally { + await query('DROP EVENT TRIGGER IF EXISTS stash_reinstall_reject_create') + } + expect( + await query<{ + version: string + table_oid: string + column_number: number + column_type_oid: string + value: unknown + index_definition: string + }>(` + SELECT eql_v3.version() AS version, + c.oid::text AS table_oid, + a.attnum AS column_number, + a.atttypid::text AS column_type_oid, + r.encrypted::jsonb AS value, + pg_catalog.pg_get_indexdef('stash_reinstall_test.records_encrypted_idx'::regclass) + AS index_definition + FROM stash_reinstall_test.records r + JOIN pg_catalog.pg_class c ON c.oid = 'stash_reinstall_test.records'::regclass + JOIN pg_catalog.pg_attribute a + ON a.attrelid = c.oid AND a.attname = 'encrypted' + WHERE r.id = 1 + `), + ).toEqual(before) + }, 180_000) + + it.each([ + { + name: 'RLS policy', + identity: 'encrypted_visible', + setup: ` + ALTER TABLE stash_reinstall_test.records ENABLE ROW LEVEL SECURITY; + CREATE POLICY encrypted_visible ON stash_reinstall_test.records + USING (eql_v3.eq_term(encrypted) IS NOT NULL); + `, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_policy WHERE polname = 'encrypted_visible'`, + }, + { + name: 'view', + identity: 'encrypted_terms', + setup: `CREATE VIEW stash_reinstall_test.encrypted_terms AS + SELECT eql_v3.eq_term(encrypted) AS term FROM stash_reinstall_test.records`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_views + WHERE schemaname = 'stash_reinstall_test' AND viewname = 'encrypted_terms'`, + }, + { + name: 'check constraint', + identity: 'encrypted_has_term', + setup: `ALTER TABLE stash_reinstall_test.records + ADD CONSTRAINT encrypted_has_term CHECK (eql_v3.eq_term(encrypted) IS NOT NULL)`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_constraint + WHERE conname = 'encrypted_has_term'`, + }, + { + name: 'generated column', + identity: 'generated_term', + setup: `ALTER TABLE stash_reinstall_test.records ADD COLUMN generated_term text + GENERATED ALWAYS AS (eql_v3.eq_term(encrypted)::text) STORED`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_attribute + WHERE attrelid = 'stash_reinstall_test.records'::regclass + AND attname = 'generated_term' AND NOT attisdropped`, + }, + { + name: 'trigger predicate', + identity: 'encrypted_trigger', + setup: ` + CREATE FUNCTION stash_reinstall_test.noop_trigger() RETURNS trigger + LANGUAGE plpgsql AS 'BEGIN RETURN NEW; END'; + CREATE TRIGGER encrypted_trigger BEFORE UPDATE ON stash_reinstall_test.records + FOR EACH ROW WHEN (eql_v3.eq_term(NEW.encrypted) IS NOT NULL) + EXECUTE FUNCTION stash_reinstall_test.noop_trigger(); + `, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_trigger + WHERE tgname = 'encrypted_trigger'`, + }, + { + name: 'customer operator', + identity: '===', + setup: `CREATE OPERATOR stash_reinstall_test.=== ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname = 'stash_reinstall_test' AND o.oprname = '==='`, + }, + { + name: 'operator duplicating a bundle signature in another schema', + identity: 'stash_reinstall_test.=(', + setup: `CREATE OPERATOR stash_reinstall_test.= ( + FUNCTION = eql_v3.eq, + LEFTARG = public.eql_v3_text_eq, + RIGHTARG = public.eql_v3_text_eq + )`, + remains: `SELECT count(*)::int AS count FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname = 'stash_reinstall_test' AND o.oprname = '=' + AND o.oprleft = 'public.eql_v3_text_eq'::regtype + AND o.oprright = 'public.eql_v3_text_eq'::regtype`, + }, + ])( + 'refuses before mutation for a customer-owned $name', + async ({ setup, identity, remains }) => { + await query(setup) + const versionBefore = await query<{ version: string }>( + 'SELECT eql_v3.version() AS version', + ) + + await expect( + new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install(), + ).rejects.toThrow( + new RegExp( + `refused before making changes.*${identity.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}`, + 's', + ), + ) + + expect( + await query<{ version: string }>('SELECT eql_v3.version() AS version'), + ).toEqual(versionBefore) + expect(await query<{ count: number }>(remains)).toEqual([{ count: 1 }]) + }, + 180_000, + ) + + it('names every view in a dependent chain, not the rewrite rule', async () => { + // Only chain_1 references EQL. chain_2 and chain_3 reach it through the + // view above them, and CASCADE takes all three -- but a view's dependency + // on an EQL function is recorded against its _RETURN rule, and nothing in + // pg_depend leads back out of a rule to its view. Before the walk carried + // that edge, the refusal named `"_RETURN" on ...chain_1` and mentioned no + // view at all, understating what was about to be destroyed by two. + await query(`CREATE VIEW stash_reinstall_test.chain_1 AS + SELECT eql_v3.eq_term(encrypted) AS term FROM stash_reinstall_test.records`) + await query( + 'CREATE VIEW stash_reinstall_test.chain_2 AS SELECT term FROM stash_reinstall_test.chain_1', + ) + await query( + 'CREATE VIEW stash_reinstall_test.chain_3 AS SELECT term FROM stash_reinstall_test.chain_2', + ) + + const failure = await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }) + .install() + .then(() => null) + .catch((error: unknown) => + error instanceof Error ? error.message : String(error), + ) + + expect(failure).toMatch(/refused before making changes/) + expect(failure).toContain('stash_reinstall_test.chain_1') + expect(failure).toContain('stash_reinstall_test.chain_2') + expect(failure).toContain('stash_reinstall_test.chain_3') + // The rule and the row/array types name the same casualties the views do. + expect(failure).not.toContain('_RETURN') + expect(failure).not.toContain('chain_1[]') + }, 180_000) +}) diff --git a/packages/cli/src/installer/__tests__/restoration-scenarios.ts b/packages/cli/src/installer/__tests__/restoration-scenarios.ts new file mode 100644 index 000000000..bd2cfa930 --- /dev/null +++ b/packages/cli/src/installer/__tests__/restoration-scenarios.ts @@ -0,0 +1,215 @@ +export interface SearchIndexRestorationScenario { + identity: string + tableIdentity: string + definition: string + valid: boolean + ready: boolean + clustered: boolean + clusterSql: string | null + replicaIdentity: boolean + replicaIdentitySql: string | null + comment: string | null + commentSql: string | null + owner: string + statisticsTargets: Array + statisticsSql: string[] +} + +export type RestorationEvent = + | 'begin' + | 'lock' + | 'configure' + | 'capture' + | 'replace' + | 'reconstruct' + | 'cluster' + | 'replica-identity' + | 'comment' + | 'statistics' + | 'analyze' + | 'verify' + | 'commit' + | 'rollback' + +export function searchIndexRestorationScenario( + overrides: Partial = {}, +): SearchIndexRestorationScenario { + return { + identity: 'app.users_email_idx', + tableIdentity: 'app.users', + definition: + 'CREATE INDEX users_email_idx ON app.users USING btree (eql_v3.eq_term(email))', + valid: true, + ready: true, + clustered: false, + clusterSql: null, + replicaIdentity: false, + replicaIdentitySql: null, + comment: null, + commentSql: null, + owner: 'app_owner', + statisticsTargets: [null], + statisticsSql: [], + ...overrides, + } +} + +/** Recording PostgreSQL adapter for deterministic restoration protocol tests. */ +export class RecordingRestorationDatabase { + readonly events: RestorationEvent[] = [] + + constructor( + private readonly scenario?: SearchIndexRestorationScenario, + private readonly options: { + unsafeIdentity?: string + failReconstructionWith?: Error + failConfigurationWith?: Error + verificationOverrides?: Partial + incompleteCaptureMetadata?: boolean + } = {}, + ) {} + + query = async ( + sql: string, + ): Promise<{ rows: unknown[]; rowCount: number }> => { + const event = restorationEvent(sql, this.scenario) + if (event !== null) this.events.push(event) + + if (event === 'lock') return { rows: [{ acquired: true }], rowCount: 1 } + if (event === 'configure' && this.options.failConfigurationWith) { + throw this.options.failConfigurationWith + } + if (event === 'capture') { + if (this.options.incompleteCaptureMetadata) { + return { + rows: [ + { + ...captureRow(searchIndexRestorationScenario()), + identity: null, + definition: null, + }, + ], + rowCount: 1, + } + } + if (this.options.unsafeIdentity) { + return { + rows: [ + { + dependency_kind: 'unsafe', + identity: this.options.unsafeIdentity, + }, + ], + rowCount: 1, + } + } + return this.scenario + ? { rows: [captureRow(this.scenario)], rowCount: 1 } + : { rows: [], rowCount: 0 } + } + if (event === 'reconstruct' && this.options.failReconstructionWith) { + throw this.options.failReconstructionWith + } + if (event === 'verify' && this.scenario) { + return { + rows: [ + verificationRow({ + ...this.scenario, + ...this.options.verificationOverrides, + }), + ], + rowCount: 1, + } + } + return { rows: [], rowCount: 0 } + } +} + +/** Live PostgreSQL adapter for catalog behavior that a recording cannot prove. */ +export class LiveRestorationDatabase { + constructor(readonly url: string) {} + + withEqlSearchPath(): LiveRestorationDatabase { + const parsed = new URL(this.url) + parsed.searchParams.set( + 'options', + '-c search_path=public,eql_v3,eql_v3_internal', + ) + return new LiveRestorationDatabase(parsed.toString()) + } + + async query(sql: string, params: unknown[] = []): Promise { + const { default: pg } = await import('pg') + const client = new pg.Client({ connectionString: this.url }) + await client.connect() + try { + return (await client.query(sql, params)).rows as T[] + } finally { + await client.end().catch(() => undefined) + } + } + + dependencyInventory(): Promise { + return this.query( + derivedSearchIndexRestorationTestSeam.lifecycleDependenciesSql, + ) + } +} + +function captureRow(scenario: SearchIndexRestorationScenario) { + return { + dependency_kind: 'index', + identity: scenario.identity, + definition: scenario.definition, + table_identity: scenario.tableIdentity, + valid: scenario.valid, + ready: scenario.ready, + clustered: scenario.clustered, + cluster_sql: scenario.clusterSql, + replica_identity: scenario.replicaIdentity, + replica_identity_sql: scenario.replicaIdentitySql, + comment: scenario.comment, + comment_sql: scenario.commentSql, + owner: scenario.owner, + statistics_targets: scenario.statisticsTargets, + statistics_sql: scenario.statisticsSql, + } +} + +function verificationRow(scenario: SearchIndexRestorationScenario) { + return { + identity: scenario.identity, + definition: scenario.definition, + valid: scenario.valid, + ready: scenario.ready, + clustered: scenario.clustered, + replica_identity: scenario.replicaIdentity, + comment: scenario.comment, + owner: scenario.owner, + statistics_targets: scenario.statisticsTargets, + } +} + +function restorationEvent( + sql: string, + scenario?: SearchIndexRestorationScenario, +): RestorationEvent | null { + if (sql === 'BEGIN') return 'begin' + if (sql.includes('pg_try_advisory_xact_lock')) return 'lock' + if (sql === 'SET LOCAL jit = off') return 'configure' + if (sql.includes('stash_eql_lifecycle_dependencies')) return 'capture' + if (sql.includes('CREATE SCHEMA eql_v3')) return 'replace' + if (scenario && sql === scenario.definition) return 'reconstruct' + if (scenario?.clusterSql && sql === scenario.clusterSql) return 'cluster' + if (scenario?.replicaIdentitySql && sql === scenario.replicaIdentitySql) + return 'replica-identity' + if (scenario?.commentSql && sql === scenario.commentSql) return 'comment' + if (scenario?.statisticsSql.includes(sql)) return 'statistics' + if (sql.startsWith('ANALYZE ')) return 'analyze' + if (sql.includes('stash_eql_verify_rebuilt_indexes')) return 'verify' + if (sql === 'COMMIT') return 'commit' + if (sql === 'ROLLBACK') return 'rollback' + return null +} + +import { derivedSearchIndexRestorationTestSeam } from '../derived-search-index-restoration.js' diff --git a/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts new file mode 100644 index 000000000..52ac614fe --- /dev/null +++ b/packages/cli/src/installer/__tests__/upgrade-encrypted-indexes.live.test.ts @@ -0,0 +1,173 @@ +/** + * Credentialed upgrade coverage: a released EQL bundle owns the original + * database objects, protect-ffi writes genuine ciphertext, and the public CLI + * installer replaces that bundle without breaking the encrypted indexes. + */ + +import { readInstallSql as readBaselineInstallSql } from '@cipherstash/eql-upgrade-baseline/sql' +import type { + EncryptConfig, + EncryptedPayload, + newClient, +} from '@cipherstash/protect-ffi' +import { config as loadEnv } from 'dotenv' +import pg from 'pg' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { EQLInstaller } from '../index.js' + +loadEnv({ + path: new URL('../../../../stack/.env', import.meta.url), + quiet: true, +}) + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const hasCredentials = [ + 'CS_WORKSPACE_CRN', + 'CS_CLIENT_ID', + 'CS_CLIENT_KEY', + 'CS_CLIENT_ACCESS_KEY', +].every((name) => process.env[name]) +const describeLive = DATABASE_URL && hasCredentials ? describe : describe.skip + +const BASELINE_VERSION = '3.0.2' +const encryptConfig: EncryptConfig = { + v: 1, + tables: { + eql_upgrade_records: { + email: { cast_as: 'text', indexes: { unique: {} } }, + score: { cast_as: 'int', indexes: { ore: {} } }, + }, + }, +} + +describeLive('EQLInstaller upgrade — genuine encrypted indexes', () => { + const client = new pg.Client({ connectionString: DATABASE_URL }) + let protectFfi: typeof import('@cipherstash/protect-ffi') + let protectClient: Awaited> + + beforeAll(async () => { + protectFfi = await import('@cipherstash/protect-ffi') + await client.connect() + await client.query('DROP TABLE IF EXISTS eql_upgrade_records') + await client.query(readBaselineInstallSql()) + expect( + ( + await client.query<{ version: string }>( + 'SELECT eql_v3.version() AS version', + ) + ).rows[0].version, + ).toBe(BASELINE_VERSION) + + protectClient = await protectFfi.newClient({ encryptConfig, eqlVersion: 3 }) + const rows = await protectFfi.encryptBulk(protectClient, { + plaintexts: [ + { + plaintext: 'alice@example.com', + column: 'email', + table: 'eql_upgrade_records', + }, + { plaintext: 10, column: 'score', table: 'eql_upgrade_records' }, + { + plaintext: 'bob@example.com', + column: 'email', + table: 'eql_upgrade_records', + }, + { plaintext: 20, column: 'score', table: 'eql_upgrade_records' }, + ], + }) + await client.query(` + CREATE TABLE eql_upgrade_records ( + id integer PRIMARY KEY, + email public.eql_v3_text_eq NOT NULL, + score public.eql_v3_integer_ord_ore NOT NULL + ); + CREATE INDEX eql_upgrade_email_idx + ON eql_upgrade_records (eql_v3.eq_term(email)); + CREATE INDEX eql_upgrade_score_idx + ON eql_upgrade_records (eql_v3.ord_term_ore(score)); + `) + await client.query( + `INSERT INTO eql_upgrade_records VALUES + (1, $1::jsonb, $2::jsonb), (2, $3::jsonb, $4::jsonb)`, + rows, + ) + }, 180_000) + + afterAll(async () => { + await client + .query('DROP TABLE IF EXISTS eql_upgrade_records') + .catch(() => undefined) + await client.end().catch(() => undefined) + }) + + it('upgrades a released installation while preserving usable encrypted indexes', async () => { + await new EQLInstaller({ databaseUrl: DATABASE_URL ?? '' }).install() + + const emailOperand = await protectFfi.encryptQuery(protectClient, { + plaintext: 'bob@example.com', + column: 'email', + table: 'eql_upgrade_records', + indexType: 'unique', + }) + const scoreOperand = await protectFfi.encryptQuery(protectClient, { + plaintext: 15, + column: 'score', + table: 'eql_upgrade_records', + indexType: 'ore', + }) + const result = await client.query<{ + email: EncryptedPayload + score: EncryptedPayload + }>( + `SELECT email::jsonb, score::jsonb FROM eql_upgrade_records + WHERE email = $1::jsonb::eql_v3.query_text_eq + AND score > $2::jsonb::eql_v3.query_integer_ord_ore`, + [emailOperand, scoreOperand], + ) + expect( + await protectFfi.decryptBulk(protectClient, { + ciphertexts: result.rows.flatMap(({ email, score }) => [ + { ciphertext: email }, + { ciphertext: score }, + ]), + }), + ).toEqual(['bob@example.com', 20]) + + const indexes = await client.query<{ relname: string; valid: boolean }>(` + SELECT c.relname, i.indisvalid AS valid + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname IN ('eql_upgrade_email_idx', 'eql_upgrade_score_idx') + ORDER BY c.relname + `) + expect(indexes.rows).toEqual([ + { relname: 'eql_upgrade_email_idx', valid: true }, + { relname: 'eql_upgrade_score_idx', valid: true }, + ]) + + await client.query('SET enable_seqscan = off') + try { + const emailPlan = await client.query<{ 'QUERY PLAN': string }>( + `EXPLAIN (COSTS OFF) + SELECT id FROM eql_upgrade_records + WHERE eql_v3.eq_term(email) = + eql_v3.eq_term($1::jsonb::eql_v3.query_text_eq)`, + [emailOperand], + ) + const scorePlan = await client.query<{ 'QUERY PLAN': string }>( + `EXPLAIN (COSTS OFF) + SELECT id FROM eql_upgrade_records + WHERE eql_v3.ord_term_ore(score) > + eql_v3.ord_term_ore($1::jsonb::eql_v3.query_integer_ord_ore)`, + [scoreOperand], + ) + expect( + emailPlan.rows.map((row) => row['QUERY PLAN']).join('\n'), + ).toContain('eql_upgrade_email_idx') + expect( + scorePlan.rows.map((row) => row['QUERY PLAN']).join('\n'), + ).toContain('eql_upgrade_score_idx') + } finally { + await client.query('RESET enable_seqscan') + } + }, 180_000) +}) diff --git a/packages/cli/src/installer/__tests__/verify.live.test.ts b/packages/cli/src/installer/__tests__/verify.live.test.ts index cf575f4ad..a8eebe234 100644 --- a/packages/cli/src/installer/__tests__/verify.live.test.ts +++ b/packages/cli/src/installer/__tests__/verify.live.test.ts @@ -20,9 +20,9 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { EQLInstaller } from '../index.js' import { + assessEqlSurface, bundledExpectedSurface, readInstalledSurface, - readOreState, verifyEqlSurface, } from '../verify.js' @@ -78,7 +78,7 @@ describeLive('verifyEqlSurface — live Postgres', () => { }, 60_000) /** - * `eql status` reads the ORE half through {@link readOreState} rather than + * `eql status` reads the ORE half through the summary assessment rather than * the full surface diff (#891). Both must answer the same question the same * way against the same database — a cheap read that disagreed with `verify` * would be worse than no read at all. @@ -90,7 +90,8 @@ describeLive('verifyEqlSurface — live Postgres', () => { const client = new pg.Client({ connectionString: url }) await client.connect() try { - const ore = await readOreState(client) + await client.query('BEGIN READ ONLY') + const { ore } = await assessEqlSurface(client, 'summary') // The database runs the pinned bundle, so the probe is comparable — // it declines to answer only on a version skew. expect(ore.comparable).toBe(true) @@ -100,6 +101,7 @@ describeLive('verifyEqlSurface — live Postgres', () => { expect(ore.poisonedDomains).toBe(report.ore?.poisonedDomains) expect(ore.expectedPoisoned).toBe(report.ore?.expectedPoisoned) } finally { + await client.query('ROLLBACK').catch(() => undefined) await client.end().catch(() => undefined) } }, 60_000) diff --git a/packages/cli/src/installer/__tests__/verify.test.ts b/packages/cli/src/installer/__tests__/verify.test.ts index a51dcbcd9..4ad79888e 100644 --- a/packages/cli/src/installer/__tests__/verify.test.ts +++ b/packages/cli/src/installer/__tests__/verify.test.ts @@ -2,11 +2,12 @@ import { readInstallSql } from '@cipherstash/eql/sql' import type pg from 'pg' import { describe, expect, it } from 'vitest' import { + assessEqlSurface, diffSurface, type ExpectedSurface, type InstalledSurface, parseExpectedSurface, - readOreState, + readInstalledSurface, } from '../verify.js' /** @@ -387,7 +388,7 @@ describe('diffSurface', () => { }) /** - * A client that answers just the two queries `readOreState` issues, so the + * A client that answers the catalogue queries the summary assessment issues, so the * version gate can be exercised without a database. `undefinedFunction` * spells the 42883 an absent `eql_v3.version()` raises. */ @@ -423,10 +424,10 @@ function fakeOreClient(answers: { return { client: client as unknown as pg.ClientBase, queries } } -describe('readOreState', () => { +describe('assessEqlSurface summary', () => { it('classifies the ORE state when the installed version is the pinned one', async () => { const { client } = fakeOreClient({ version: expected.eqlVersion }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(true) if (!reading.comparable) return expect(reading.state).toBe('indexable') @@ -439,7 +440,7 @@ describe('readOreState', () => { opclassPresent: false, poisonedDomains: expected.oreDomains.length, }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable && reading.state).toBe('fallback') }) @@ -454,7 +455,7 @@ describe('readOreState', () => { opclassPresent: false, poisonedDomains: expected.oreDomains.length - 2, }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(false) if (reading.comparable) return expect(reading.installedVersion).toBe('3.0.0') @@ -467,9 +468,150 @@ describe('readOreState', () => { it('reports a missing version() as not comparable', async () => { const { client } = fakeOreClient({ version: null }) - const reading = await readOreState(client) + const { ore: reading } = await assessEqlSurface(client, 'summary') expect(reading.comparable).toBe(false) if (reading.comparable) return expect(reading.installedVersion).toBeNull() }) }) + +describe('readInstalledSurface in a caller transaction', () => { + it('restores transaction-local settings before returning', async () => { + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: false, poisoned_domains: 0 }] } + } + return { rows: [] } + }, + } + + await readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }) + + expect(queries[0]).toBe('SAVEPOINT installed_eql_surface_read') + expect(queries.at(-2)).toBe( + 'ROLLBACK TO SAVEPOINT installed_eql_surface_read', + ) + expect(queries.at(-1)).toBe('RELEASE SAVEPOINT installed_eql_surface_read') + }) + + it('preserves the version-read error when probe cleanup also fails', async () => { + const client = { + async query(sql: string) { + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: true, + eql_v3_internal_present: true, + pgcrypto_installed: true, + pgcrypto_schema: 'public', + }, + ], + } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: true, poisoned_domains: 0 }] } + } + if (sql.includes('eql_v3.version()')) { + throw Object.assign(new Error('permission denied'), { code: '42501' }) + } + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_version_probe') { + throw new Error('cleanup failed') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toThrow('Could not read eql_v3.version(): permission denied') + }) + + it('preserves result-construction errors after restoring the caller transaction', async () => { + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('pgcrypto_installed')) { + return { + rows: [ + { + eql_v3_present: false, + eql_v3_internal_present: false, + pgcrypto_installed: false, + pgcrypto_schema: null, + }, + ], + } + } + if (sql.includes('FROM pg_catalog.pg_proc')) { + return { rows: [{ name: null, signature: '' }] } + } + if (sql.includes('ore_opclass_present')) { + return { rows: [{ ore_opclass_present: false, poisoned_domains: 0 }] } + } + if ( + sql === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read' && + queries.filter((query) => query === sql).length > 1 + ) { + throw new Error('savepoint no longer exists') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toThrow(/null|toLowerCase/) + + expect( + queries.filter( + (query) => query === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read', + ), + ).toHaveLength(1) + }) + + it('preserves a surface-read error when caller-savepoint cleanup also fails', async () => { + const primary = new Error('catalog read failed') + const queries: string[] = [] + const client = { + async query(sql: string) { + queries.push(sql) + if (sql.includes('FROM pg_catalog.pg_proc')) throw primary + if (sql === 'ROLLBACK TO SAVEPOINT installed_eql_surface_read') { + throw new Error('rollback failed') + } + return { rows: [] } + }, + } + + await expect( + readInstalledSurface(client as unknown as pg.ClientBase, expected, { + manageTransaction: false, + }), + ).rejects.toBe(primary) + + expect(queries).toContain('RELEASE SAVEPOINT installed_eql_surface_read') + }) +}) diff --git a/packages/cli/src/installer/derived-search-index-restoration.ts b/packages/cli/src/installer/derived-search-index-restoration.ts new file mode 100644 index 000000000..96f2d1fa8 --- /dev/null +++ b/packages/cli/src/installer/derived-search-index-restoration.ts @@ -0,0 +1,701 @@ +import type pg from 'pg' + +import { createPgClient, TlsVerificationError } from '@/db/client.js' + +const LIFECYCLE_LOCK = 'cipherstash.eql.lifecycle' + +/** + * How long to keep trying for the lifecycle lock before giving up. + * + * The budget is sized off the INSTALL, which is what actually holds the lock: + * `DROP SCHEMA ... CASCADE` plus ~3,000 object creations, 10-30s against a + * local container and longer on managed Postgres. The dependency capture + * ({@link LIFECYCLE_DEPENDENCIES_SQL}) also runs under the lock but is not the + * cost. Its recursive shape inflates PostgreSQL's estimate enough to trigger + * JIT compilation of hundreds of expressions: measured on an idle local + * container with EQL installed, the query itself is ~35ms with JIT disabled + * versus ~880ms with JIT enabled. `install()` disables JIT for its transaction. + * + * Five minutes is deliberate margin over that, not a measurement: a waiter + * should never be refused for ordinary queueing, only for a holder that will + * never release. Earlier revisions of this comment justified the number with + * capture timings of 17-68s. Those were measured on a server concurrently + * running the test suite and are contention, not query cost; do not reinstate + * them as evidence. + */ +const LOCK_WAIT_MS = 300_000 +const LOCK_RETRY_INTERVAL_MS = 250 + +export class EqlReinstallRefusalError extends Error {} + +export class EqlReinstallConnectionError extends Error {} + +export class EqlLifecycleLockTimeoutError extends Error {} + +export class DerivedSearchIndexReconstructionError extends Error {} + +export class DerivedSearchIndexVerificationError extends Error {} + +export interface RestorationSummary { + restoredIndexes: number + analyzedTables: number +} + +interface RestoreAroundEqlReplacementOptions { + databaseUrl: string + bundledSql: string +} + +export interface ReinstallIndex { + identity: string + definition: string + tableIdentity: string + valid: boolean + ready: boolean + clustered: boolean + clusterSql: string | null + replicaIdentity: boolean + replicaIdentitySql: string | null + comment: string | null + commentSql: string | null + owner: string + statisticsTargets: Array + statisticsSql: string[] +} + +interface DependencyRow { + dependency_kind?: unknown + identity?: unknown + definition?: unknown + table_identity?: unknown + valid?: unknown + ready?: unknown + clustered?: unknown + cluster_sql?: unknown + replica_identity?: unknown + replica_identity_sql?: unknown + comment?: unknown + comment_sql?: unknown + owner?: unknown + statistics_targets?: unknown + statistics_sql?: unknown +} + +const LIFECYCLE_DEPENDENCIES_SQL = ` +/* stash_eql_lifecycle_dependencies */ +WITH RECURSIVE +eql_roots(classid, objid, objsubid) AS ( + SELECT 'pg_catalog.pg_namespace'::regclass, n.oid, 0 + FROM pg_catalog.pg_namespace n + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_proc'::regclass, p.oid, 0 + FROM pg_catalog.pg_proc p + JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_type'::regclass, t.oid, 0 + FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_operator'::regclass, o.oid, 0 + FROM pg_catalog.pg_operator o + JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_opclass'::regclass, o.oid, 0 + FROM pg_catalog.pg_opclass o + JOIN pg_catalog.pg_namespace n ON n.oid = o.opcnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') + UNION ALL + SELECT 'pg_catalog.pg_opfamily'::regclass, o.oid, 0 + FROM pg_catalog.pg_opfamily o + JOIN pg_catalog.pg_namespace n ON n.oid = o.opfnamespace + WHERE n.nspname IN ('eql_v3', 'eql_v3_internal') +), +/* + * Everything reachable from the bundle, transitively. The lateral is one + * recursive self-reference carrying two kinds of edge, because PostgreSQL + * permits only one. + * + * The first is the ordinary one: whatever depends on the object we are + * standing on. + * + * The second exists because a view's dependency on an EQL function is recorded + * against its rewrite RULE, not against the view, and nothing in pg_depend + * leads from the rule back out to the view — the rule's own edge to it points + * the wrong way. Walking edge one alone therefore stops at + * \`"_RETURN" on public.v1\` and never learns that v1 is a view, that v2 selects + * from v1, or that v3 selects from v2. The refusal was still correct (a rule is + * not a rebuildable index, so the install refused) but it named a rule where + * three views were at stake, and under-reported the blast radius to whoever had + * to decide what to do about it. + * + * Restricted to \`relkind IN ('v', 'm')\` deliberately. A view cannot outlive its + * _RETURN rule, so the view really is destroyed and really is the better name. + * A rule on an ordinary TABLE is different: dropping the rule leaves the table + * standing, so hopping there would name a table that survives, and the rule + * stays the honest answer. + */ +dependency_edges(refclassid, refobjid, refobjsubid, classid, objid, objsubid) AS ( + SELECT d.refclassid, d.refobjid, d.refobjsubid, d.classid, d.objid, d.objsubid + FROM pg_catalog.pg_depend d + UNION ALL + SELECT 'pg_catalog.pg_rewrite'::regclass, rewrite_rule.oid, 0, + 'pg_catalog.pg_class'::regclass, view_class.oid, 0 + FROM pg_catalog.pg_rewrite rewrite_rule + JOIN pg_catalog.pg_class view_class ON view_class.oid = rewrite_rule.ev_class + WHERE view_class.relkind IN ('v', 'm') +), +dependants(classid, objid, objsubid) AS ( + SELECT classid, objid, objsubid FROM eql_roots + UNION + SELECT e.classid, e.objid, e.objsubid + FROM dependency_edges e + JOIN dependants parent + ON e.refclassid = parent.classid + AND e.refobjid = parent.objid + AND (parent.objsubid = 0 OR e.refobjsubid = parent.objsubid) +), +external_dependants AS ( + SELECT d.classid, d.objid, d.objsubid + FROM dependants d + WHERE NOT EXISTS ( + SELECT 1 FROM eql_roots r + WHERE r.classid = d.classid + AND r.objid = d.objid + AND r.objsubid = d.objsubid + ) + -- Objects whose own namespace is disposable are bundle contents, even when + -- they were reached indirectly through an operator family or row type. + AND COALESCE( + (pg_catalog.pg_identify_object(d.classid, d.objid, d.objsubid)).schema, + '' + ) NOT IN ('eql_v3', 'eql_v3_internal') + -- A relation drags its composite row type and that type's array type along. + -- Both name the same casualty the relation already names (\`public.v1\`, + -- \`public.v1[]\`), so report the relation once and drop the two types. + AND NOT ( + d.classid = 'pg_catalog.pg_type'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_type shadow + LEFT JOIN pg_catalog.pg_type element ON element.oid = shadow.typelem + WHERE shadow.oid = d.objid + AND COALESCE(element.typrelid, shadow.typrelid) <> 0 + ) + ) + -- The view this rule belongs to is now in the walk and is the better name for + -- the same casualty, so report the view and drop the rule rather than both. + AND NOT ( + d.classid = 'pg_catalog.pg_rewrite'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_rewrite rewrite_rule + JOIN pg_catalog.pg_class view_class ON view_class.oid = rewrite_rule.ev_class + WHERE rewrite_rule.oid = d.objid + AND view_class.relkind IN ('v', 'm') + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_constraint'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_constraint con + JOIN pg_catalog.pg_type typ ON typ.oid = con.contypid + JOIN pg_catalog.pg_namespace n ON n.oid = typ.typnamespace + WHERE con.oid = d.objid + AND ( + n.nspname IN ('eql_v3', 'eql_v3_internal') + OR ( + con.conname = 'eql_ore_unavailable' + AND n.nspname = 'public' + AND typ.typname LIKE 'eql\\_v3\\_%' ESCAPE '\\' + ) + ) + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_proc'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_proc proc + JOIN pg_catalog.pg_namespace n ON n.oid = proc.pronamespace + WHERE proc.oid = d.objid + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + AND NOT ( + d.classid = 'pg_catalog.pg_class'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_class class + JOIN pg_catalog.pg_namespace n ON n.oid = class.relnamespace + WHERE class.oid = d.objid + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + -- EQL installs its operators in public and implements them with functions in + -- a disposable EQL schema. Classify the installed catalog, not the incoming + -- bundle: an upgrade may legitimately remove an old operator. + AND NOT ( + d.classid = 'pg_catalog.pg_operator'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_operator operator + JOIN pg_catalog.pg_namespace operator_namespace + ON operator_namespace.oid = operator.oprnamespace + JOIN pg_catalog.pg_proc implementation + ON implementation.oid = operator.oprcode + JOIN pg_catalog.pg_namespace implementation_namespace + ON implementation_namespace.oid = implementation.pronamespace + WHERE operator.oid = d.objid + AND operator_namespace.nspname = 'public' + AND implementation_namespace.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + -- Likewise, EQL's casts are implemented by functions in the disposable EQL + -- schemas. Classify the installed catalog rather than the incoming bundle: + -- an upgrade may legitimately remove an old cast. + AND NOT ( + d.classid = 'pg_catalog.pg_cast'::regclass + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_cast cast_row + JOIN pg_catalog.pg_proc implementation + ON implementation.oid = cast_row.castfunc + JOIN pg_catalog.pg_namespace implementation_namespace + ON implementation_namespace.oid = implementation.pronamespace + WHERE cast_row.oid = d.objid + AND implementation_namespace.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) + -- pg_amop/pg_amproc have no namespace of their own; their owning family does. + AND NOT ( + d.classid IN ( + 'pg_catalog.pg_amop'::regclass, + 'pg_catalog.pg_amproc'::regclass + ) + AND EXISTS ( + SELECT 1 + FROM pg_catalog.pg_opfamily family + JOIN pg_catalog.pg_namespace n ON n.oid = family.opfnamespace + WHERE family.oid = CASE + WHEN d.classid = 'pg_catalog.pg_amop'::regclass + THEN (SELECT amopfamily FROM pg_catalog.pg_amop WHERE oid = d.objid) + ELSE (SELECT amprocfamily FROM pg_catalog.pg_amproc WHERE oid = d.objid) + END + AND n.nspname IN ('eql_v3', 'eql_v3_internal') + ) + ) +) +/* + * Only standalone ordinary indexes are reconstructed. Partitioned index parents + * (\`I\`) and attached child indexes are classified as unsafe because recreating + * their attachment graph requires metadata beyond \`pg_get_indexdef\`. + */ +SELECT DISTINCT + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN 'index' + ELSE 'unsafe' + END AS dependency_kind, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.format('%I.%I', index_namespace.nspname, index_class.relname) + ELSE (pg_catalog.pg_identify_object(e.classid, e.objid, e.objsubid)).identity + END AS identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN pg_catalog.pg_get_indexdef(e.objid) + END AS definition, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.format('%I.%I', table_namespace.nspname, table_class.relname) + END AS table_identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisvalid + END AS valid, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisready + END AS ready, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisclustered + END AS clustered, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL THEN index_meta.indisreplident + END AS replica_identity, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN pg_catalog.obj_description(index_class.oid, 'pg_class') + END AS comment, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND index_meta.indisclustered + THEN pg_catalog.format( + 'ALTER TABLE %I.%I CLUSTER ON %I', + table_namespace.nspname, + table_class.relname, + index_class.relname + ) + END AS cluster_sql, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND index_meta.indisreplident + THEN pg_catalog.format( + 'ALTER TABLE %I.%I REPLICA IDENTITY USING INDEX %I', + table_namespace.nspname, + table_class.relname, + index_class.relname + ) + END AS replica_identity_sql, + CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + AND pg_catalog.obj_description(index_class.oid, 'pg_class') IS NOT NULL + THEN pg_catalog.format( + 'COMMENT ON INDEX %I.%I IS %L', + index_namespace.nspname, + index_class.relname, + pg_catalog.obj_description(index_class.oid, 'pg_class') + ) + END AS comment_sql + , pg_catalog.pg_get_userbyid(index_class.relowner) AS owner + , CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN ARRAY( + SELECT attribute.attstattarget + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = index_class.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + ORDER BY attribute.attnum + ) + END AS statistics_targets + , CASE + WHEN e.classid = 'pg_catalog.pg_class'::regclass + AND index_class.relkind = 'i' + AND index_partition.inhrelid IS NULL + THEN ARRAY( + SELECT pg_catalog.format( + 'ALTER INDEX %I.%I ALTER COLUMN %s SET STATISTICS %s', + index_namespace.nspname, + index_class.relname, + attribute.attnum, + attribute.attstattarget + ) + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = index_class.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + AND attribute.attstattarget >= 0 + ORDER BY attribute.attnum + ) + END AS statistics_sql +FROM external_dependants e +LEFT JOIN pg_catalog.pg_class index_class + ON e.classid = 'pg_catalog.pg_class'::regclass AND index_class.oid = e.objid +LEFT JOIN pg_catalog.pg_inherits index_partition + ON index_partition.inhrelid = index_class.oid +LEFT JOIN pg_catalog.pg_namespace index_namespace ON index_namespace.oid = index_class.relnamespace +LEFT JOIN pg_catalog.pg_index index_meta ON index_meta.indexrelid = index_class.oid +LEFT JOIN pg_catalog.pg_class table_class ON table_class.oid = index_meta.indrelid +LEFT JOIN pg_catalog.pg_namespace table_namespace ON table_namespace.oid = table_class.relnamespace +ORDER BY dependency_kind, identity +` + +const VERIFY_REBUILT_INDEXES_SQL = ` +/* stash_eql_verify_rebuilt_indexes */ +SELECT pg_catalog.format('%I.%I', n.nspname, c.relname) AS identity, + i.indisvalid AS valid, + i.indisready AS ready, + i.indisclustered AS clustered, + i.indisreplident AS replica_identity, + pg_catalog.obj_description(c.oid, 'pg_class') AS comment, + pg_catalog.pg_get_userbyid(c.relowner) AS owner, + ARRAY( + SELECT attribute.attstattarget + FROM pg_catalog.pg_attribute attribute + WHERE attribute.attrelid = c.oid + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + ORDER BY attribute.attnum + ) AS statistics_targets, + pg_catalog.pg_get_indexdef(i.indexrelid) AS definition +FROM pg_catalog.pg_index i +JOIN pg_catalog.pg_class c ON c.oid = i.indexrelid +JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace +WHERE pg_catalog.format('%I.%I', n.nspname, c.relname) = ANY($1::text[]) +` + +/** + * Take the installer's advisory lock, or say why not. + * + * `pg_advisory_xact_lock` waits forever. A concurrent `stash eql install`, or a + * session that died holding the lock, then makes the command hang with no + * output and no timeout — indistinguishable from a network stall, and the one + * failure a user cannot diagnose. Polling `pg_try_advisory_xact_lock` against a + * bounded budget keeps the ordinary case working — a queued install still wins + * the lock and runs — while turning the pathological one into a sentence. See + * {@link LOCK_WAIT_MS} for why the five-minute budget is not the handful of + * seconds it first was. (#959) + */ +async function acquireLifecycleLock( + client: pg.ClientBase, + // Optional so `acquireLifecycleLock(client)` keeps meaning what it did. The + // budget is a policy, not a constant of nature, and the live suite drives it + // short to assert the refusal without waiting out the production default. + waitMs: number = LOCK_WAIT_MS, +) { + const deadline = Date.now() + waitMs + for (;;) { + const result = await client.query<{ acquired: boolean }>( + 'SELECT pg_catalog.pg_try_advisory_xact_lock(pg_catalog.hashtext($1)) AS acquired', + [LIFECYCLE_LOCK], + ) + if (result.rows[0]?.acquired === true) { + return + } + if (Date.now() >= deadline) { + throw new EqlLifecycleLockTimeoutError( + `Another EQL lifecycle operation is in progress on this database — it has held the installer's advisory lock for more than ${Math.round(waitMs / 1000)} seconds. Nothing was changed. Wait for the other \`stash eql install\`/\`eql upgrade\` to finish and re-run. If no other command is running, an earlier one may have died holding the lock: find its session in \`pg_stat_activity\` and close it, then retry.`, + ) + } + // Deliberately NOT unref'd: the retry is the only pending work between + // polls, and letting Node drop it would end the process mid-install. + await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_INTERVAL_MS)) + } +} + +async function inspectReinstallDependencies( + client: pg.ClientBase, +): Promise { + const result = await client.query(LIFECYCLE_DEPENDENCIES_SQL) + const unsafe: string[] = [] + const indexes: ReinstallIndex[] = [] + for (const row of result.rows as DependencyRow[]) { + if (row.dependency_kind !== 'index') { + unsafe.push(String(row.identity ?? 'unknown database object')) + continue + } + if ( + typeof row.identity !== 'string' || + typeof row.definition !== 'string' || + typeof row.table_identity !== 'string' || + typeof row.valid !== 'boolean' || + typeof row.ready !== 'boolean' || + typeof row.clustered !== 'boolean' || + typeof row.replica_identity !== 'boolean' || + (row.cluster_sql !== null && typeof row.cluster_sql !== 'string') || + (row.clustered && typeof row.cluster_sql !== 'string') || + (row.replica_identity_sql !== null && + typeof row.replica_identity_sql !== 'string') || + (row.replica_identity && typeof row.replica_identity_sql !== 'string') || + (row.comment !== null && typeof row.comment !== 'string') || + (row.comment_sql !== null && typeof row.comment_sql !== 'string') || + (typeof row.comment === 'string' && + typeof row.comment_sql !== 'string') || + typeof row.owner !== 'string' || + !Array.isArray(row.statistics_targets) || + !row.statistics_targets.every( + (target) => target === null || typeof target === 'number', + ) || + !Array.isArray(row.statistics_sql) || + !row.statistics_sql.every((sql) => typeof sql === 'string') + ) { + unsafe.push( + String(row.identity ?? 'index with incomplete catalog metadata'), + ) + continue + } + indexes.push({ + identity: row.identity, + definition: row.definition, + tableIdentity: row.table_identity, + valid: row.valid, + ready: row.ready, + clustered: row.clustered, + clusterSql: row.cluster_sql, + replicaIdentity: row.replica_identity, + replicaIdentitySql: row.replica_identity_sql, + comment: row.comment, + commentSql: row.comment_sql, + owner: row.owner, + statisticsTargets: row.statistics_targets, + statisticsSql: row.statistics_sql, + }) + } + if (unsafe.length > 0) { + throw new EqlReinstallRefusalError( + `EQL reinstall refused before making changes because customer-owned database objects depend on disposable EQL machinery and cannot be reconstructed safely:\n${unsafe.map((identity) => ` - ${identity}`).join('\n')}`, + ) + } + return indexes +} + +/** + * Recreate captured ordinary indexes and verify their exact catalog shape. + * + * `pg_get_indexdef` is itself search_path-sensitive: on a connection whose + * search_path names `eql_v3` the captured definition reads `eq_term(encrypted)` + * rather than `eql_v3.eq_term(encrypted)`. That is safe here and deliberately + * not "fixed" — the same session rebuilds it, and Postgres qualifies a name + * exactly when leaving it bare would resolve to something else, so the + * rebuilt expression binds to the same function the original did. + * + */ +async function rebuildIndexes( + client: pg.ClientBase, + indexes: ReinstallIndex[], +) { + for (const index of indexes) { + try { + await client.query(index.definition) + } catch (error) { + const detail = error instanceof Error ? error.message : String(error) + throw new DerivedSearchIndexReconstructionError( + `EQL reinstall could not rebuild search index ${index.identity}: ${detail}\nThe transaction will restore the previous EQL installation and index. Captured index SQL:\n${index.definition}`, + { cause: error }, + ) + } + } + for (const index of indexes) { + if (index.clusterSql !== null) await client.query(index.clusterSql) + if (index.replicaIdentitySql !== null) + await client.query(index.replicaIdentitySql) + if (index.commentSql !== null) await client.query(index.commentSql) + for (const statisticsSql of index.statisticsSql) { + await client.query(statisticsSql) + } + } + const tableIdentities = new Set(indexes.map((index) => index.tableIdentity)) + for (const tableIdentity of tableIdentities) { + await client.query(`ANALYZE ${tableIdentity}`) + } + if (indexes.length === 0) { + return { restoredIndexes: 0, analyzedTables: 0 } + } + const result = await client.query(VERIFY_REBUILT_INDEXES_SQL, [ + indexes.map((index) => index.identity), + ]) + const healthy = new Set( + result.rows + .filter((row) => { + const expected = indexes.find( + (index) => index.identity === row.identity, + ) + if (expected === undefined) return false + // The definition must match EXACTLY: it is the only evidence that the + // index which came back is the index that went away. + // + // Validity is compared against what was CAPTURED rather than against + // `true`, because an index can already be invalid going in and the + // reinstall does not promise to repair one it did not break — a + // partitioned parent created `ON ONLY` while its per-partition indexes + // are still being built, or an interrupted `CREATE INDEX CONCURRENTLY`. + // Demanding `true` there fails the whole install and the rollback says + // nothing about why. A VALID index coming back invalid is still caught; + // that is the regression this check exists for. + return ( + expected.definition === row.definition && + (row.valid === true || expected.valid === false) && + (row.ready === true || expected.ready === false) && + row.clustered === expected.clustered && + row.replica_identity === expected.replicaIdentity && + (row.comment ?? null) === expected.comment && + row.owner === expected.owner && + Array.isArray(row.statistics_targets) && + row.statistics_targets.length === expected.statisticsTargets.length && + row.statistics_targets.every( + (target: unknown, position: number) => + target === expected.statisticsTargets[position], + ) + ) + }) + .map((row) => String(row.identity)), + ) + const unhealthy = indexes.filter((index) => !healthy.has(index.identity)) + if (unhealthy.length > 0) { + throw new DerivedSearchIndexVerificationError( + `EQL reinstall produced missing, invalid, or changed search indexes; the transaction will restore the previous installation:\n${unhealthy.map((index) => ` - ${index.identity}\n ${index.definition}`).join('\n')}`, + ) + } + return { + restoredIndexes: indexes.length, + analyzedTables: tableIdentities.size, + } +} + +/** + * Replace EQL machinery while preserving every reconstructable derived search + * index in one transaction. Catalog discovery, reconstruction details, and + * verification remain private so callers cannot execute the protocol out of + * order or retain stale captured state. + */ +export async function restoreDerivedSearchIndexesAroundEqlReplacement({ + databaseUrl, + bundledSql, +}: RestoreAroundEqlReplacementOptions): Promise { + const client = createPgClient(databaseUrl) + try { + await client.connect() + } catch (error) { + await client.end().catch(() => {}) + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new EqlReinstallConnectionError( + `Failed to connect to database: ${detail}`, + { + cause: error, + }, + ) + } + + try { + await client.query('BEGIN') + try { + await acquireLifecycleLock(client) + await client.query('SET LOCAL jit = off') + const indexes = await inspectReinstallDependencies(client) + await client.query(bundledSql) + const summary = await rebuildIndexes(client, indexes) + await client.query('COMMIT') + return summary + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error + } + } finally { + await client.end() + } +} + +/** @internal Direct catalog probes retained only for live PostgreSQL evidence. */ +export const derivedSearchIndexRestorationTestSeam = { + acquireLifecycleLock, + lifecycleDependenciesSql: LIFECYCLE_DEPENDENCIES_SQL, +} diff --git a/packages/cli/src/installer/eql-bundle.ts b/packages/cli/src/installer/eql-bundle.ts new file mode 100644 index 000000000..ead2fd6d6 --- /dev/null +++ b/packages/cli/src/installer/eql-bundle.ts @@ -0,0 +1,23 @@ +import { readInstallSql } from '@cipherstash/eql/sql' +import { assertBundledEqlSqlDigest } from './bundle-digest.js' + +/** Schemas in which the pinned EQL bundle can resolve pgcrypto safely. */ +export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] + +/** + * Read the pinned EQL installer and prove its bytes match the resolved release. + * Keep this file free of installer and verifier imports: both consume the same + * artifact, and neither should become the other's dependency. + */ +export function loadBundledEqlSql(): string { + let sql: string + try { + sql = readInstallSql() + } catch (error) { + throw new Error( + 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', + { cause: error }, + ) + } + return assertBundledEqlSqlDigest(sql) +} diff --git a/packages/cli/src/installer/grants.ts b/packages/cli/src/installer/grants.ts index 352b57474..0594aab1d 100644 --- a/packages/cli/src/installer/grants.ts +++ b/packages/cli/src/installer/grants.ts @@ -189,3 +189,41 @@ export const SUPABASE_MIGRATION_GRANTS_SQL_V3 = `${SUPABASE_IMMEDIATE_GRANTS_SQL -- runs as a member of \`postgres\` (they cover EQL objects \`postgres\` might -- later create outside stash tooling; stash re-grants on every install). ${SUPABASE_GUARDED_DEFAULT_PRIVILEGES_SQL_V3}` + +export type SupabaseEqlAccessOutcome = + | { status: 'applied' } + | { status: 'applied-with-deferred-defaults'; deferredSql: string } + +interface SqlExecutor { + query(sql: string): Promise<{ rows: Record[] }> +} + +/** + * Apply the Supabase EQL access policy through a PostgreSQL adapter. + * + * Role membership, the immediate/default split, statement ordering, and the + * operator-facing deferred SQL are implementation details of this module. + */ +export async function applySupabaseEqlAccess( + database: SqlExecutor, +): Promise { + const membership = await database.query(` + SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') + THEN pg_has_role(current_user, 'postgres', 'MEMBER') + END AS member_of_postgres + `) + if (membership.rows[0]?.member_of_postgres === true) { + await database.query(SUPABASE_PERMISSIONS_SQL_V3) + return { status: 'applied' } + } + await database.query(SUPABASE_IMMEDIATE_GRANTS_SQL_V3) + return { + status: 'applied-with-deferred-defaults', + deferredSql: DEFERRED_GRANTS_HEADER + SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, + } +} + +/** Emit the same access policy through a migration-file adapter. */ +export function emitSupabaseEqlAccessMigration(): string { + return SUPABASE_MIGRATION_GRANTS_SQL_V3 +} diff --git a/packages/cli/src/installer/index.ts b/packages/cli/src/installer/index.ts index 21ac65440..8113bde2b 100644 --- a/packages/cli/src/installer/index.ts +++ b/packages/cli/src/installer/index.ts @@ -1,20 +1,31 @@ -import { readInstallSql } from '@cipherstash/eql/sql' -import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' -import { assertBundledEqlSqlDigest } from './bundle-digest.js' import { - DEFERRED_GRANTS_HEADER, - EQL_V3_INTERNAL_SCHEMA_NAME, - EQL_V3_SCHEMA_NAME, - SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, - SUPABASE_IMMEDIATE_GRANTS_SQL_V3, + EqlLifecycleLockTimeoutError, + EqlReinstallConnectionError, + EqlReinstallRefusalError, + restoreDerivedSearchIndexesAroundEqlReplacement, +} from './derived-search-index-restoration.js' +import { + applySupabaseEqlAccess, SUPABASE_PERMISSIONS_SQL_V3, } from './grants.js' +import { + assessEqlInstallation, + type PreflightResult, +} from './installation-state.js' +import { loadVerifiedEqlBundle } from './verify.js' export { + loadBundledEqlSql, + SUPPORTED_PGCRYPTO_SCHEMAS, +} from './eql-bundle.js' + +export { + applySupabaseEqlAccess, DEFERRED_GRANTS_HEADER, EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME, + emitSupabaseEqlAccessMigration, SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, SUPABASE_GUARDED_DEFAULT_PRIVILEGES_SQL_V3, SUPABASE_IMMEDIATE_GRANTS_SQL_V3, @@ -27,35 +38,6 @@ export { /** EQL generations recognised by read-only installation diagnostics. */ export type EqlVersion = 2 | 3 -const EQL_V2_SCHEMA_NAME = 'eql_v2' - -/** - * The pinned EQL v3 install SQL, verified against the resolved release's - * `installSqlSha256` before it is handed to anything that executes or emits it. - * - * This is the CLI's single choke point for the bundle — `install()`, - * `stash eql migration`'s emitter and `bundledExpectedSurface()` all come - * through here — which is why the digest check lives in the wrapper rather than - * at each call site. `readInstallSql()` itself is in the frozen - * `@cipherstash/eql` subtree, published from another repository, so a check - * added there would be dead code for every consumer installing from npm. - * - * @throws if the bundle cannot be read, or if its bytes are not the ones the - * resolved release attests to (see {@link assertBundledEqlSqlDigest}). - */ -export function loadBundledEqlSql(): string { - let sql: string - try { - sql = readInstallSql() - } catch (error) { - throw new Error( - 'Failed to read the EQL v3 install SQL from `@cipherstash/eql`. Reinstall dependencies (the package ships the bundle in `dist/sql/`).', - { cause: error }, - ) - } - return assertBundledEqlSqlDigest(sql) -} - /** Supabase grants for the sole installable generation, EQL v3. */ export function supabaseGrantsFor(): string { return SUPABASE_PERMISSIONS_SQL_V3 @@ -70,48 +52,7 @@ export function supabaseGrantsFor(): string { * fine; the installer defers the owner-scoped Supabase default-privilege * statements instead (see {@link InstallResult.deferredGrantsSql}). */ -export interface PreflightResult { - currentUser: string - isSuperuser: boolean - /** - * Whether `current_user` can run `ALTER DEFAULT PRIVILEGES FOR ROLE - * postgres`. `null` when the database has no `postgres` role at all. - */ - memberOfPostgres: boolean | null - hasDatabaseCreate: boolean - hasPublicCreate: boolean - pgcryptoInstalled: boolean - /** - * The schema `pgcrypto` lives in, or `null` when not installed. The pinned - * bundle accepts `extensions` and `public` (its functions' search_path) and - * ABORTS for any other schema — so an unsupported placement blocks even a - * superuser. - */ - pgcryptoSchema: string | null - eqlV3SchemaPresent: boolean - eqlV3InternalSchemaPresent: boolean - /** - * Whether `current_user` may drop the existing `eql_v3` / `eql_v3_internal` - * schemas (owner, member of the owning role, or superuser). `null` when the - * schema is absent. Matters because a reinstall begins with - * `DROP SCHEMA ... CASCADE`. - */ - canDropEqlV3Schema: boolean | null - canDropEqlV3InternalSchema: boolean | null - /** - * Whether this role can create the ORE btree operator class the `_ord_ore` - * domains need (#891). `null` when the probe could not answer. - * - * Never blocks: the bundle skips the class and installs its loud-failure - * fallback instead, which is a supported configuration. It is reported so - * the trade is known before a schema is written, not after a query fails. - * See {@link probeOperatorClassCreate} for why this is probed rather than - * inferred from `isSuperuser`. - */ - canCreateOperatorClass: boolean | null - missing: string[] - ok: boolean -} +export type { PreflightResult } from './installation-state.js' /** * The legacy permission-check shape. @@ -138,93 +79,6 @@ export interface InstallResult { deferredGrantsSql: string | null } -/** - * One query answering every preflight question. Two guard patterns are - * load-bearing: `pg_has_role` raises on a nonexistent role name (not every - * database has a `postgres` role), and `has_schema_privilege` raises 3F000 on - * a nonexistent schema (hardened databases drop `public`) — each probe that - * can raise is wrapped so a missing object reads as a capability answer, not - * a query failure. The scalar subqueries against `pg_namespace` return NULL - * (not an error) when the schema is absent, which maps to the `null` arms of - * {@link PreflightResult}. - */ -const PREFLIGHT_SQL = ` - SELECT - current_user AS role_name, - (SELECT rolsuper FROM pg_roles WHERE rolname = current_user) AS is_superuser, - CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') - THEN pg_has_role(current_user, 'postgres', 'MEMBER') - END AS member_of_postgres, - has_database_privilege(current_user, current_database(), 'CREATE') AS has_database_create, - CASE WHEN EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') - THEN has_schema_privilege(current_user, 'public', 'CREATE') - ELSE false - END AS has_public_create, - EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') AS pgcrypto_installed, - (SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace - WHERE e.extname = 'pgcrypto') AS pgcrypto_schema, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_SCHEMA_NAME}') AS eql_v3_present, - EXISTS (SELECT 1 FROM information_schema.schemata WHERE schema_name = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS eql_v3_internal_present, - (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n - WHERE n.nspname = '${EQL_V3_SCHEMA_NAME}') AS can_drop_eql_v3, - (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n - WHERE n.nspname = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS can_drop_eql_v3_internal -` - -/** The schemas the pinned bundle accepts `pgcrypto` in (its search_path). */ -export const SUPPORTED_PGCRYPTO_SCHEMAS = ['extensions', 'public'] - -/** - * Can this role create the ORE btree operator class? (#891) - * - * Asked of the server rather than inferred, because `rolsuper` is the wrong - * question. `CREATE OPERATOR CLASS` is superuser-gated in stock PostgreSQL, - * but managed platforms differ on whether their admin role clears that gate: - * AWS RDS and Aurora do (with `rolsuper = f`), cloud-hosted Supabase does not. - * Predicting from `rolsuper` would tell an RDS operator their ORE domains are - * unavailable when they work — exactly the blanket claim about "managed - * Postgres" this whole change exists to stop making. - * - * `CREATE OPERATOR FAMILY` shares the privilege gate with `CREATE OPERATOR - * CLASS` and needs no member operators, so it is the cheapest statement that - * tests it. The whole probe runs in a transaction that is always rolled back, - * so preflight stays observably read-only. - * - * Returns `null` when the attempt could not answer the question — a read-only - * replica (`25006`), a statement timeout, no `public` schema to create into. - * Callers must render that as unknown, never as either answer. - */ -async function probeOperatorClassCreate( - client: pg.ClientBase, -): Promise { - // A name no bundle uses, so a probe that somehow escaped its rollback is - // recognisable rather than mistaken for an EQL object. - const probeName = 'public.stash_preflight_opclass_probe' - try { - await client.query('BEGIN') - } catch { - return null - } - try { - await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) - return true - } catch (error) { - // 42501 insufficient_privilege is the gate itself — a real "no". Anything - // else (no CREATE on public, read-only transaction, timeout) is a probe - // that failed to ask the question. - const code = - typeof error === 'object' && error !== null && 'code' in error - ? String((error as { code?: unknown }).code) - : undefined - return code === '42501' ? false : null - } finally { - // Always: on the success path this is what keeps preflight read-only, and - // on the failure path it clears the aborted transaction. A rollback that - // itself fails leaves nothing behind — the connection is closed next. - await client.query('ROLLBACK').catch(() => {}) - } -} - export class EQLInstaller { private readonly databaseUrl: string @@ -233,103 +87,14 @@ export class EQLInstaller { } async preflight(): Promise { - const client = createPgClient(this.databaseUrl) - try { - await client.connect() - } catch (error) { - await client.end().catch(() => {}) - // Already shaped centrally by createPgClient's connect wrapper — the - // message is self-contained; adding framing would bury the remedy. - if (error instanceof TlsVerificationError) throw error - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) - } - try { - const result = await client.query(PREFLIGHT_SQL) - const row = result.rows[0] ?? {} - const isSuperuser = row.is_superuser === true - const hasDatabaseCreate = row.has_database_create === true - const pgcryptoInstalled = row.pgcrypto_installed === true - const pgcryptoSchema = - typeof row.pgcrypto_schema === 'string' ? row.pgcrypto_schema : null - const asBoolOrNull = (value: unknown) => - typeof value === 'boolean' ? value : null - const canDropEqlV3Schema = asBoolOrNull(row.can_drop_eql_v3) - const canDropEqlV3InternalSchema = asBoolOrNull( - row.can_drop_eql_v3_internal, - ) - const missing: string[] = [] - if (!isSuperuser) { - if (!hasDatabaseCreate) { - missing.push( - 'CREATE on database (required for CREATE SCHEMA and CREATE EXTENSION)', - ) - } - if (row.has_public_create !== true) { - missing.push( - 'CREATE on public schema (required for CREATE DOMAIN public.eql_v3_*)', - ) - } - if (!pgcryptoInstalled && !hasDatabaseCreate) { - missing.push( - 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', - ) - } - } - // Not gated on superuser: the bundle itself raises for a pgcrypto - // outside its functions' search_path, whoever runs it. - if ( - pgcryptoInstalled && - pgcryptoSchema !== null && - !SUPPORTED_PGCRYPTO_SCHEMAS.includes(pgcryptoSchema) - ) { - missing.push( - `pgcrypto relocated (it is in schema "${pgcryptoSchema}", which is not on the EQL search_path — the install aborts; fix with: ALTER EXTENSION pgcrypto SET SCHEMA extensions)`, - ) - } - // pg_has_role is true for superusers and for the owner, so this only - // fires for a role that genuinely cannot run the bundle's opening - // DROP SCHEMA ... CASCADE against someone else's install. - if ( - canDropEqlV3Schema === false || - canDropEqlV3InternalSchema === false - ) { - missing.push( - 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', - ) - } - // After the capability read, so a probe that somehow poisons the session - // cannot affect any of the answers above. - const canCreateOperatorClass = await probeOperatorClassCreate(client) - return { - currentUser: String(row.role_name ?? 'unknown'), - isSuperuser, - memberOfPostgres: asBoolOrNull(row.member_of_postgres), - hasDatabaseCreate, - hasPublicCreate: row.has_public_create === true, - pgcryptoInstalled, - pgcryptoSchema, - eqlV3SchemaPresent: row.eql_v3_present === true, - eqlV3InternalSchemaPresent: row.eql_v3_internal_present === true, - canDropEqlV3Schema, - canDropEqlV3InternalSchema, - canCreateOperatorClass, - missing, - // Deliberately not folded into `missing`: the bundle's ORE fallback - // means an install without the operator class is complete, not - // blocked. - ok: missing.length === 0, - } - } catch (error) { - const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Database preflight query failed: ${detail}`, { - cause: error, - }) - } finally { - await client.end() + const installation = await assessEqlInstallation({ + databaseUrl: this.databaseUrl, + includeCapabilities: true, + }) + if (installation.capabilities.status !== 'assessed') { + throw new Error('Database capabilities were not assessed') } + return installation.capabilities.preflight } /** @@ -349,18 +114,16 @@ export class EQLInstaller { /** Generation-aware read-only detection retained for legacy diagnostics. */ async isInstalled(options?: { eqlVersion?: EqlVersion }): Promise { + const generation = options?.eqlVersion ?? 3 const client = createPgClient(this.databaseUrl) - const requiredSchemas = - (options?.eqlVersion ?? 3) === 3 - ? [EQL_V3_SCHEMA_NAME, EQL_V3_INTERNAL_SCHEMA_NAME] - : [EQL_V2_SCHEMA_NAME] try { await client.connect() - const result = await client.query( - 'SELECT count(*)::int AS found FROM information_schema.schemata WHERE schema_name = ANY($1)', - [requiredSchemas], + const result = await client.query<{ installed: boolean }>( + generation === 2 + ? "SELECT to_regnamespace('eql_v2') IS NOT NULL AS installed" + : "SELECT to_regnamespace('eql_v3') IS NOT NULL AND to_regnamespace('eql_v3_internal') IS NOT NULL AS installed", ) - return result.rows[0]?.found === requiredSchemas.length + return result.rows[0]?.installed === true } catch (error) { if (error instanceof TlsVerificationError) throw error const detail = error instanceof Error ? error.message : String(error) @@ -376,29 +139,26 @@ export class EQLInstaller { async getInstalledVersion(options?: { eqlVersion?: EqlVersion }): Promise { - const schemaName = - (options?.eqlVersion ?? 3) === 3 ? EQL_V3_SCHEMA_NAME : EQL_V2_SCHEMA_NAME + const generation = options?.eqlVersion ?? 3 const client = createPgClient(this.databaseUrl) try { await client.connect() - const schemaResult = await client.query( - 'SELECT schema_name FROM information_schema.schemata WHERE schema_name = $1', + const schemaName = `eql_v${generation}` + const schema = await client.query<{ installed: boolean }>( + 'SELECT to_regnamespace($1) IS NOT NULL AS installed', [schemaName], ) - if (schemaResult.rowCount === null || schemaResult.rowCount === 0) { - return null - } + if (schema.rows[0]?.installed !== true) return null try { - const versionResult = await client.query( + const result = await client.query<{ version: string }>( `SELECT ${schemaName}.version() AS version`, ) - if (versionResult.rows[0]?.version) { - return String(versionResult.rows[0].version) - } + return result.rows[0]?.version + ? String(result.rows[0].version) + : 'unknown' } catch { - // Older installs may not expose version(). + return 'unknown' } - return 'unknown' } catch (error) { if (error instanceof TlsVerificationError) throw error const detail = error instanceof Error ? error.message : String(error) @@ -431,45 +191,38 @@ export class EQLInstaller { // was attempted and rolled back". It also keeps the digest message the // whole error, rather than a `detail` interpolated into the install // wrapper's transaction narration below. - const bundledSql = loadBundledEqlSql() - const client = createPgClient(this.databaseUrl) + const bundle = loadVerifiedEqlBundle() try { - await client.connect() + await restoreDerivedSearchIndexesAroundEqlReplacement({ + databaseUrl: this.databaseUrl, + bundledSql: bundle.sql, + }) } catch (error) { - if (error instanceof TlsVerificationError) throw error + if ( + error instanceof TlsVerificationError || + error instanceof EqlLifecycleLockTimeoutError || + error instanceof EqlReinstallConnectionError || + error instanceof EqlReinstallRefusalError + ) { + throw error + } const detail = error instanceof Error ? error.message : String(error) - throw new Error(`Failed to connect to database: ${detail}`, { - cause: error, - }) + throw new Error( + `Failed to install EQL: ${detail}. Nothing was applied — the install runs in a transaction and was rolled back.`, + { cause: error }, + ) } - try { - try { - await client.query('BEGIN') - await client.query(bundledSql) - await client.query('COMMIT') - } catch (error) { - await client.query('ROLLBACK').catch(() => {}) - const detail = error instanceof Error ? error.message : String(error) - throw new Error( - `Failed to install EQL: ${detail}. Nothing was applied — the install runs in a transaction and was rolled back.`, - { cause: error }, - ) - } - - if (!options?.supabase) return { deferredGrantsSql: null } + if (!options?.supabase) return { deferredGrantsSql: null } - try { - return await this.runSupabaseGrants(client) - } catch (error) { - const detail = error instanceof Error ? error.message : String(error) - throw new Error( - `EQL v3 is installed, but granting the Supabase roles failed: ${detail}. The install itself was NOT rolled back — re-run \`stash eql install --force\` (or plain \`stash eql install\`, which re-applies the grants on an already-installed database).`, - { cause: error }, - ) - } - } finally { - await client.end() + try { + return await this.applySupabaseGrants() + } catch (error) { + const detail = error instanceof Error ? error.message : String(error) + throw new Error( + `EQL v3 is installed, but granting the Supabase roles failed: ${detail}. The install itself was NOT rolled back — re-run \`stash eql install --force\` (or plain \`stash eql install\`, which re-applies the grants on an already-installed database).`, + { cause: error }, + ) } } @@ -494,7 +247,11 @@ export class EQLInstaller { }) } try { - return await this.runSupabaseGrants(client) + const outcome = await applySupabaseEqlAccess(client) + return { + deferredGrantsSql: + outcome.status === 'applied' ? null : outcome.deferredSql, + } } catch (error) { const detail = error instanceof Error ? error.message : String(error) throw new Error(`Failed to apply the Supabase role grants: ${detail}`, { @@ -504,22 +261,4 @@ export class EQLInstaller { await client.end() } } - - /** The shared grants phase: full block for members, immediate half + deferred tail otherwise. */ - private async runSupabaseGrants(client: pg.Client): Promise { - const memberResult = await client.query(` - SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') - THEN pg_has_role(current_user, 'postgres', 'MEMBER') - END AS member_of_postgres - `) - if (memberResult.rows[0]?.member_of_postgres === true) { - await client.query(SUPABASE_PERMISSIONS_SQL_V3) - return { deferredGrantsSql: null } - } - await client.query(SUPABASE_IMMEDIATE_GRANTS_SQL_V3) - return { - deferredGrantsSql: - DEFERRED_GRANTS_HEADER + SUPABASE_DEFAULT_PRIVILEGES_SQL_V3, - } - } } diff --git a/packages/cli/src/installer/installation-state.ts b/packages/cli/src/installer/installation-state.ts new file mode 100644 index 000000000..c4a0d9f53 --- /dev/null +++ b/packages/cli/src/installer/installation-state.ts @@ -0,0 +1,297 @@ +import type pg from 'pg' +import { createPgClient, TlsVerificationError } from '@/db/client.js' +import { SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' +import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' +import type { OreSurfaceState } from './ore.js' +import { + assessEqlSurface, + type OreStateReading, + type VerifyReport, +} from './verify.js' + +export type InstalledEqlGeneration = + | { status: 'absent' } + | { status: 'installed'; version: string | 'unknown' } + +export type AssessedOreState = + | { status: 'absent' } + | { status: 'not-requested' } + | { status: 'unavailable'; message: string } + | { + status: 'not-comparable' + bundleVersion: string + installedVersion: string | null + } + | { + status: 'observed' + state: OreSurfaceState + opclassPresent: boolean + poisonedDomains: number + expectedPoisoned: number + } + +export type AssessedEqlSurface = + | { status: 'not-requested' } + | { status: 'not-comparable'; report: VerifyReport } + | { status: 'complete'; report: VerifyReport } + | { status: 'damaged'; report: VerifyReport } + +export interface EqlInstallationState { + v2: InstalledEqlGeneration + v3: InstalledEqlGeneration + ore: AssessedOreState + surface: AssessedEqlSurface + capabilities: + | { status: 'not-requested' } + | { status: 'assessed'; preflight: PreflightResult } +} + +export interface PreflightResult { + currentUser: string + isSuperuser: boolean + memberOfPostgres: boolean | null + hasDatabaseCreate: boolean + hasPublicCreate: boolean + pgcryptoInstalled: boolean + pgcryptoSchema: string | null + eqlV3SchemaPresent: boolean + eqlV3InternalSchemaPresent: boolean + canDropEqlV3Schema: boolean | null + canDropEqlV3InternalSchema: boolean | null + canCreateOperatorClass: boolean | null + missing: string[] + ok: boolean +} + +const CAPABILITIES_SQL = ` + SELECT current_user AS role_name, + (SELECT rolsuper FROM pg_roles WHERE rolname = current_user) AS is_superuser, + CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'postgres') THEN pg_has_role(current_user, 'postgres', 'MEMBER') END AS member_of_postgres, + has_database_privilege(current_user, current_database(), 'CREATE') AS has_database_create, + CASE WHEN EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') THEN has_schema_privilege(current_user, 'public', 'CREATE') ELSE false END AS has_public_create, + EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcrypto') AS pgcrypto_installed, + (SELECT n.nspname FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'pgcrypto') AS pgcrypto_schema, + (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_SCHEMA_NAME}') AS can_drop_eql_v3, + (SELECT pg_has_role(current_user, n.nspowner, 'MEMBER') FROM pg_namespace n WHERE n.nspname = '${EQL_V3_INTERNAL_SCHEMA_NAME}') AS can_drop_eql_v3_internal +` + +export async function assessEqlInstallation(options: { + databaseUrl: string + depth?: 'summary' | 'exhaustive' + includeCapabilities?: boolean + includeOre?: boolean +}): Promise { + const client = createPgClient(options.databaseUrl) + try { + await client.connect() + } catch (error) { + await client.end().catch(() => {}) + if (error instanceof TlsVerificationError) throw error + const detail = error instanceof Error ? error.message : String(error) + throw new Error(`Failed to connect to database: ${detail}`, { + cause: error, + }) + } + + try { + await client.query('BEGIN READ ONLY') + const presence = await client.query<{ + eql_v2_present: boolean + eql_v3_present: boolean + eql_v3_internal_present: boolean + }>(` + SELECT + to_regnamespace('eql_v2') IS NOT NULL AS eql_v2_present, + to_regnamespace('eql_v3') IS NOT NULL AS eql_v3_present, + to_regnamespace('eql_v3_internal') IS NOT NULL AS eql_v3_internal_present + `) + const row = presence.rows[0] + const v2Present = row?.eql_v2_present === true + const v3Present = + row?.eql_v3_present === true && row.eql_v3_internal_present === true + const v2 = v2Present + ? { status: 'installed' as const, version: await readVersion(client, 2) } + : { status: 'absent' as const } + const v3 = v3Present + ? { status: 'installed' as const, version: await readVersion(client, 3) } + : { status: 'absent' as const } + + let verification = null + let unavailableOre: AssessedOreState | null = null + if (options.depth === 'exhaustive') { + verification = await assessEqlSurface(client, 'exhaustive') + } else if (v3Present && options.includeOre === true) { + await client.query('SAVEPOINT eql_ore_assessment') + try { + verification = await assessEqlSurface(client, 'summary') + await client.query('RELEASE SAVEPOINT eql_ore_assessment') + } catch (error) { + await client.query('ROLLBACK TO SAVEPOINT eql_ore_assessment') + await client.query('RELEASE SAVEPOINT eql_ore_assessment') + unavailableOre = { + status: 'unavailable', + message: error instanceof Error ? error.message : String(error), + } + } + } + const ore = + verification?.depth === 'summary' + ? assessOre(verification.ore) + : verification?.report.ore + ? { status: 'observed' as const, ...verification.report.ore } + : v3Present && verification?.report.status === 'version-mismatch' + ? { + status: 'not-comparable' as const, + bundleVersion: verification.report.bundleVersion, + installedVersion: verification.report.installedVersion, + } + : (unavailableOre ?? + (v3Present + ? { status: 'not-requested' as const } + : { status: 'absent' as const })) + let surface: AssessedEqlSurface = { status: 'not-requested' } + if (verification?.depth === 'exhaustive') { + const report = verification.report + surface = + report.status === 'version-mismatch' + ? { status: 'not-comparable', report } + : report.ok + ? { status: 'complete', report } + : { status: 'damaged', report } + } + const capabilityRow = options.includeCapabilities + ? ((await client.query(CAPABILITIES_SQL)).rows[0] ?? {}) + : null + await client.query('COMMIT') + const capabilities = capabilityRow + ? { + status: 'assessed' as const, + preflight: buildPreflight( + capabilityRow, + await probeOperatorClassCreate(client), + { + eqlV3SchemaPresent: row?.eql_v3_present === true, + eqlV3InternalSchemaPresent: row?.eql_v3_internal_present === true, + }, + ), + } + : { status: 'not-requested' as const } + return { v2, v3, ore, surface, capabilities } + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error + } finally { + await client.end() + } +} + +function buildPreflight( + row: Record, + canCreateOperatorClass: boolean | null, + presence: { + eqlV3SchemaPresent: boolean + eqlV3InternalSchemaPresent: boolean + }, +): PreflightResult { + const asBoolOrNull = (value: unknown) => + typeof value === 'boolean' ? value : null + const isSuperuser = row.is_superuser === true + const hasDatabaseCreate = row.has_database_create === true + const pgcryptoInstalled = row.pgcrypto_installed === true + const pgcryptoSchema = + typeof row.pgcrypto_schema === 'string' ? row.pgcrypto_schema : null + const canDropEqlV3Schema = asBoolOrNull(row.can_drop_eql_v3) + const canDropEqlV3InternalSchema = asBoolOrNull(row.can_drop_eql_v3_internal) + const missing: string[] = [] + if (!isSuperuser) { + if (!hasDatabaseCreate) + missing.push( + 'CREATE on database (required for CREATE SCHEMA and CREATE EXTENSION)', + ) + if (row.has_public_create !== true) + missing.push( + 'CREATE on public schema (required for CREATE DOMAIN public.eql_v3_*)', + ) + if (!pgcryptoInstalled && !hasDatabaseCreate) + missing.push( + 'SUPERUSER or extension owner (required for CREATE EXTENSION pgcrypto)', + ) + } + if ( + pgcryptoInstalled && + pgcryptoSchema !== null && + !SUPPORTED_PGCRYPTO_SCHEMAS.includes(pgcryptoSchema) + ) + missing.push( + `pgcrypto relocated (it is in schema "${pgcryptoSchema}", which is not on the EQL search_path — the install aborts; fix with: ALTER EXTENSION pgcrypto SET SCHEMA extensions)`, + ) + if (canDropEqlV3Schema === false || canDropEqlV3InternalSchema === false) + missing.push( + 'ownership of the existing EQL schemas (a reinstall begins with DROP SCHEMA eql_v3 / eql_v3_internal CASCADE, which needs the owner, a member of the owning role, or a superuser)', + ) + return { + currentUser: String(row.role_name ?? 'unknown'), + isSuperuser, + memberOfPostgres: asBoolOrNull(row.member_of_postgres), + hasDatabaseCreate, + hasPublicCreate: row.has_public_create === true, + pgcryptoInstalled, + pgcryptoSchema, + eqlV3SchemaPresent: presence.eqlV3SchemaPresent, + eqlV3InternalSchemaPresent: presence.eqlV3InternalSchemaPresent, + canDropEqlV3Schema, + canDropEqlV3InternalSchema, + canCreateOperatorClass, + missing, + ok: missing.length === 0, + } +} + +async function probeOperatorClassCreate( + client: pg.ClientBase, +): Promise { + const probeName = 'public.stash_preflight_opclass_probe' + try { + await client.query('BEGIN') + } catch { + return null + } + try { + await client.query(`CREATE OPERATOR FAMILY ${probeName} USING btree`) + return true + } catch (error) { + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : undefined + return code === '42501' ? false : null + } finally { + await client.query('ROLLBACK').catch(() => {}) + } +} + +async function readVersion( + client: { + query: (sql: string) => Promise<{ rows: Array<{ version?: unknown }> }> + }, + generation: 2 | 3, +): Promise { + await client.query('SAVEPOINT eql_version_probe') + try { + const result = await client.query( + `SELECT eql_v${generation}.version() AS version`, + ) + await client.query('RELEASE SAVEPOINT eql_version_probe') + return result.rows[0]?.version ? String(result.rows[0].version) : 'unknown' + } catch { + await client.query('ROLLBACK TO SAVEPOINT eql_version_probe') + await client.query('RELEASE SAVEPOINT eql_version_probe') + return 'unknown' + } +} + +function assessOre(ore: OreStateReading): AssessedOreState { + return ore.comparable + ? { status: 'observed', ...ore } + : { status: 'not-comparable', ...ore } +} diff --git a/packages/cli/src/installer/verify.ts b/packages/cli/src/installer/verify.ts index e0c69bd1d..37ab7a784 100644 --- a/packages/cli/src/installer/verify.ts +++ b/packages/cli/src/installer/verify.ts @@ -1,8 +1,8 @@ import { releaseManifest } from '@cipherstash/eql/sql' import type pg from 'pg' import { createPgClient, TlsVerificationError } from '@/db/client.js' +import { loadBundledEqlSql, SUPPORTED_PGCRYPTO_SCHEMAS } from './eql-bundle.js' import { EQL_V3_INTERNAL_SCHEMA_NAME, EQL_V3_SCHEMA_NAME } from './grants.js' -import { loadBundledEqlSql, SUPPORTED_PGCRYPTO_SCHEMAS } from './index.js' import { classifyOreState, describeOreState, @@ -61,6 +61,12 @@ export interface ExpectedSurface { oreDomains: string[] } +/** The digest-verified installer bytes and the complete surface derived from them. */ +export interface VerifiedEqlBundle { + sql: string + expectedSurface: ExpectedSurface +} + /** * How the ORE half of the install reads. Defined in `./ore.js`, which owns the * whole ORE model — the catalogue probe, the state machine, and the copy every @@ -353,6 +359,16 @@ export function parseExpectedSurface(sql: string): ExpectedSurface { } } +/** + * Load the one verified artifact consumed by installation, restoration, and + * verification. Completeness parsing happens here so callers cannot combine + * SQL bytes with metadata derived from a different bundle. + */ +export function loadVerifiedEqlBundle(): VerifiedEqlBundle { + const sql = loadBundledEqlSql() + return { sql, expectedSurface: parseExpectedSurface(sql) } +} + /** The expected surface of the pinned bundle this CLI installs. */ export function bundledExpectedSurface(): ExpectedSurface { // Through `loadBundledEqlSql()` rather than `readInstallSql()` directly, so @@ -360,12 +376,7 @@ export function bundledExpectedSurface(): ExpectedSurface { // database against this expectation — derived from an unverified bundle it // would answer a different question than the one asked, and could report a // healthy install as broken (or the reverse) from tampered bytes alone. - const sql = loadBundledEqlSql() - // Deliberately outside any try: a parse failure is a bundle the parser has - // outgrown ({@link assertEveryStatementModelled}), and its message names the - // statement. Wrapping it in "reinstall dependencies" would send whoever hits - // it to the one remedy that cannot help. - return parseExpectedSurface(sql) + return loadVerifiedEqlBundle().expectedSurface } // --------------------------------------------------------------------------- @@ -517,17 +528,32 @@ const ORE_STATE_SQL = ` */ async function readInstalledEqlVersion( client: pg.ClientBase, + insideCallerTransaction = false, ): Promise { + if (insideCallerTransaction) { + await client.query('SAVEPOINT installed_eql_version_probe') + } try { const version = await client.query<{ version: string }>( `SELECT ${EQL_V3_SCHEMA_NAME}.version() AS version`, ) + if (insideCallerTransaction) { + await client.query('RELEASE SAVEPOINT installed_eql_version_probe') + } return version.rows[0]?.version ?? null } catch (error) { const code = error !== null && typeof error === 'object' && 'code' in error ? (error as { code?: string }).code : undefined + if (insideCallerTransaction) { + await client + .query('ROLLBACK TO SAVEPOINT installed_eql_version_probe') + .catch(() => {}) + await client + .query('RELEASE SAVEPOINT installed_eql_version_probe') + .catch(() => {}) + } if (code === '42883') return null const detail = error instanceof Error ? error.message : String(error) throw new Error( @@ -546,9 +572,10 @@ async function readInstalledEqlVersion( export async function readInstalledSurface( client: pg.ClientBase, expected: ExpectedSurface, + options: { manageTransaction?: boolean } = {}, ): Promise { // Sequential on purpose: a single pg.Client serialises concurrent query() - // calls anyway (and deprecates them); these are six fast catalogue reads. + // calls anyway (and deprecates them); these are seven fast catalogue reads. // // The read-only transaction exists for `SET LOCAL search_path = ''`: // `format_type` qualifies a name exactly when the type is not visible on @@ -559,78 +586,95 @@ export async function readInstalledSurface( // (`integer`, `text[]`). That is precisely the spelling the bundle parser // produces; without the pin the output would vary with the connection's // search_path. SET LOCAL dies with the transaction, so the caller's - // session is untouched (the version() probe below runs after COMMIT and - // needs the default path restored — `eql_v3.version` is qualified, but its - // body's search_path is its own SET clause either way). - await client.query('BEGIN READ ONLY') - await client.query(`SET LOCAL search_path = ''`) - const schemas = await client.query<{ - eql_v3_present: boolean - eql_v3_internal_present: boolean - pgcrypto_installed: boolean - pgcrypto_schema: string | null - }>(SCHEMAS_SQL) - const types = await client.query<{ name: string }>(TYPES_SQL, [ - [...expected.domains, ...expected.types], - ]) - const functions = await client.query<{ name: string; signature: string }>( - FUNCTION_SIGNATURES_SQL, - [[...expected.functions.keys()]], - ) - const operators = await client.query<{ - name: string - leftarg: string - rightarg: string - }>(OPERATORS_SQL) - const casts = await client.query<{ source: string; target: string }>( - CASTS_SQL, - ) - const ore = await client.query<{ - ore_opclass_present: boolean - poisoned_domains: number - }>(ORE_STATE_SQL, [expected.oreDomains]) - // Ends the SET LOCAL scope. On a mid-transaction error the caller's - // client.end() discards the aborted transaction with the connection. - await client.query('COMMIT') - - const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true - const installedVersion = eqlV3SchemaPresent - ? await readInstalledEqlVersion(client) - : null - - const functionSignatures = new Map>() - for (const row of functions.rows) { - const name = row.name.toLowerCase() - const existing = functionSignatures.get(name) ?? new Set() - existing.add(row.signature.toLowerCase()) - functionSignatures.set(name, existing) - } + // session is untouched. The qualified version() probe is protected by a + // savepoint because a missing function must not abort the caller's snapshot. + const manageTransaction = options.manageTransaction !== false + if (manageTransaction) await client.query('BEGIN READ ONLY') + else await client.query('SAVEPOINT installed_eql_surface_read') + try { + await client.query(`SET LOCAL search_path = ''`) + const schemas = await client.query<{ + eql_v3_present: boolean + eql_v3_internal_present: boolean + pgcrypto_installed: boolean + pgcrypto_schema: string | null + }>(SCHEMAS_SQL) + const types = await client.query<{ name: string }>(TYPES_SQL, [ + [...expected.domains, ...expected.types], + ]) + const functions = await client.query<{ name: string; signature: string }>( + FUNCTION_SIGNATURES_SQL, + [[...expected.functions.keys()]], + ) + const operators = await client.query<{ + name: string + leftarg: string + rightarg: string + }>(OPERATORS_SQL) + const casts = await client.query<{ source: string; target: string }>( + CASTS_SQL, + ) + const ore = await client.query<{ + ore_opclass_present: boolean + poisoned_domains: number + }>(ORE_STATE_SQL, [expected.oreDomains]) + const eqlV3SchemaPresent = schemas.rows[0]?.eql_v3_present === true + const installedVersion = eqlV3SchemaPresent + ? await readInstalledEqlVersion(client, true) + : null + const functionSignatures = new Map>() + for (const row of functions.rows) { + const name = row.name.toLowerCase() + const existing = functionSignatures.get(name) ?? new Set() + existing.add(row.signature.toLowerCase()) + functionSignatures.set(name, existing) + } - return { - eqlV3SchemaPresent, - eqlV3InternalSchemaPresent: - schemas.rows[0]?.eql_v3_internal_present === true, - pgcryptoInstalled: schemas.rows[0]?.pgcrypto_installed === true, - pgcryptoSchema: - typeof schemas.rows[0]?.pgcrypto_schema === 'string' - ? schemas.rows[0].pgcrypto_schema - : null, - installedVersion, - presentTypes: new Set(types.rows.map((row) => row.name.toLowerCase())), - functionSignatures, - presentOperators: new Set( - operators.rows.map( - (row) => - `${row.name.toLowerCase()} (${row.leftarg.toLowerCase()}, ${row.rightarg.toLowerCase()})`, + const installedSurface: InstalledSurface = { + eqlV3SchemaPresent, + eqlV3InternalSchemaPresent: + schemas.rows[0]?.eql_v3_internal_present === true, + pgcryptoInstalled: schemas.rows[0]?.pgcrypto_installed === true, + pgcryptoSchema: + typeof schemas.rows[0]?.pgcrypto_schema === 'string' + ? schemas.rows[0].pgcrypto_schema + : null, + installedVersion, + presentTypes: new Set(types.rows.map((row) => row.name.toLowerCase())), + functionSignatures, + presentOperators: new Set( + operators.rows.map( + (row) => + `${row.name.toLowerCase()} (${row.leftarg.toLowerCase()}, ${row.rightarg.toLowerCase()})`, + ), ), - ), - presentCasts: new Set( - casts.rows.map( - (row) => `${row.source.toLowerCase()} AS ${row.target.toLowerCase()}`, + presentCasts: new Set( + casts.rows.map( + (row) => `${row.source.toLowerCase()} AS ${row.target.toLowerCase()}`, + ), ), - ), - oreOpclassPresent: ore.rows[0]?.ore_opclass_present === true, - poisonedDomains: ore.rows[0]?.poisoned_domains ?? 0, + oreOpclassPresent: ore.rows[0]?.ore_opclass_present === true, + poisonedDomains: ore.rows[0]?.poisoned_domains ?? 0, + } + // Keep the savepoint alive until every fallible conversion above has + // completed. Otherwise the catch path attempts to clean up a savepoint + // already released and can replace the original error with 25P01. + if (manageTransaction) await client.query('COMMIT') + else { + await client.query('ROLLBACK TO SAVEPOINT installed_eql_surface_read') + await client.query('RELEASE SAVEPOINT installed_eql_surface_read') + } + return installedSurface + } catch (error) { + if (!manageTransaction) { + await client + .query('ROLLBACK TO SAVEPOINT installed_eql_surface_read') + .catch(() => {}) + await client + .query('RELEASE SAVEPOINT installed_eql_surface_read') + .catch(() => {}) + } + throw error } } @@ -932,7 +976,7 @@ export async function verifyEqlSurface( } /** - * The ORE half of an install as {@link readOreState} could read it. + * The ORE half of an install returned by a summary surface assessment. * * `comparable: false` means the installed EQL is not the pinned one, so there * is no honest ORE answer to give — not that anything is wrong. Callers must @@ -952,29 +996,41 @@ export type OreStateReading = installedVersion: string | null } +export type EqlSurfaceAssessment = + | { depth: 'summary'; ore: OreStateReading } + | { depth: 'exhaustive'; report: VerifyReport } + /** - * Read just the ORE half of an install — the two catalogue values and the - * state they classify to (#891). + * The report-oriented verification interface used by installation assessment. + * Parsing, catalogue observation, version gating, ORE classification, and + * surface diffing remain implementation details behind this seam. * - * `eql status` wants the ORE answer and nothing else. Routing it through - * {@link verifyEqlSurface} would work but would read the whole 3,000-operator - * surface to render one row. - * - * It still needs {@link diffSurface}'s version gate, though, because the ORE - * state is NOT a pure catalogue fact: `expectedPoisoned` is the pinned - * bundle's ORE-domain count, and {@link ORE_STATE_SQL} counts poisoned domains - * only among that same pinned list. So a healthy fallback install of a - * DIFFERENT EQL — the ordinary "CLI upgraded, database not yet" case — poisons - * ITS domains, of which the pinned list sees only some, and - * {@link classifyOreState} reads the shortfall as `incoherent-unpoisoned` - * damage. Reporting a version skew as `comparable: false` is what stops - * `eql status` telling that operator to reinstall `--force` over nothing. + * The caller owns the connection and transaction so installation presence, + * versions, and this result can describe one database snapshot. */ -export async function readOreState( +export async function assessEqlSurface( client: pg.ClientBase, -): Promise { + depth: 'summary' | 'exhaustive', +): Promise { const expected = bundledExpectedSurface() - const installedVersion = await readInstalledEqlVersion(client) + if (depth === 'summary') { + return { depth, ore: await readOreStateAgainst(client, expected, true) } + } + const installed = await readInstalledSurface(client, expected, { + manageTransaction: false, + }) + return { depth, report: diffSurface(expected, installed) } +} + +async function readOreStateAgainst( + client: pg.ClientBase, + expected: ExpectedSurface, + insideCallerTransaction = false, +): Promise { + const installedVersion = await readInstalledEqlVersion( + client, + insideCallerTransaction, + ) if (installedVersion !== expected.eqlVersion) { return { comparable: false, diff --git a/packages/cli/vitest.config.ts b/packages/cli/vitest.config.ts index c65ccc12f..eae72f539 100644 --- a/packages/cli/vitest.config.ts +++ b/packages/cli/vitest.config.ts @@ -5,13 +5,13 @@ export default defineConfig({ test: { globals: true, exclude: ['**/node_modules/**', '**/dist/**', 'tests/e2e/**'], - // Two projects so ONLY the live suites are serialised. Four of them gate + // Two projects so ONLY the live suites are serialised. Several of them gate // on STASH_TEST_DATABASE_URL and share one database and one // eql_v3/eql_v3_internal pair — and verify.live's beforeAll installs the // full bundle, which opens with `DROP SCHEMA … CASCADE`, destroying the // schemas (and their ACLs/OIDs) under a concurrently running - // guarded-grants.live. Run in parallel forks they race; run serially each - // suite sees the database state its comments already assume. The unit + // guarded-grants.live. Run in parallel forks they race; one fork lets each + // suite see the database state its comments already assume. The unit // project keeps default file parallelism — serialising all ~1300 tests // for the sake of four files is the `packages/migrate` fix at the wrong // scale. @@ -33,7 +33,7 @@ export default defineConfig({ test: { name: 'live', include: ['src/**/*.live.test.ts'], - fileParallelism: false, + poolOptions: { forks: { singleFork: true } }, }, }, ], diff --git a/packages/eql/CONTEXT.md b/packages/eql/CONTEXT.md index 72efae290..d5c2298a5 100644 --- a/packages/eql/CONTEXT.md +++ b/packages/eql/CONTEXT.md @@ -29,3 +29,10 @@ _Avoid_: Encrypted data, durable data A customer-owned constraint, policy, view, or other database object whose meaning cannot be safely inferred and recreated by the EQL installer. _Avoid_: Derived search index + +**EQL installation state**: +A consistent observation of installed EQL generations, their versions, the +health of comparable EQL machinery, and the ORE state. When the installed EQL +version differs from the observing tool's pinned bundle, health is not +comparable; version skew is not evidence of damage. +_Avoid_: Installation status, database state diff --git a/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs b/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs index aef936754..52f76cc12 100644 --- a/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs +++ b/packages/eql/tests/sqlx/tests/v3_uninstall_tests.rs @@ -189,7 +189,7 @@ async fn shipped_installer_can_run_over_existing_public_domains(pool: PgPool) -> Ok(()) } -#[sqlx::test] +#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_integer", "v3_ste_vec")))] async fn uninstaller_preserves_application_tables_with_public_domain_columns( pool: PgPool, ) -> Result<()> { @@ -199,10 +199,23 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( "expected both eql_v3 schemas installed before uninstall" ); - let scalar_payload = r#"{"v":3,"i":{},"c":"scalar-42","hm":"hm-42"}"#; - // SteVec entries carry no `hm`: an op path entry / term-less entry. - let json_payload = r#"{"i":{},"v":3,"h":"kh","sv":[{"s":"age","c":"cipher-age","op":"ab"}]}"#; - let entry_payload = r#"{"s":"age","c":"cipher-age","op":"ab"}"#; + // These are generated by cipherstash-client for every test run. Preservation + // must be proved with real ciphertext and index terms, not JSON shaped by + // the test to happen to satisfy today's domain constraints. + let scalar_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.eql_v3_integer ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let json_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.v3_ste_vec ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let entry_payload = json_payload + .get("sv") + .and_then(serde_json::Value::as_array) + .and_then(|entries| entries.first()) + .cloned() + .expect("generated SteVec fixture must contain at least one entry"); sqlx::query( r#" @@ -230,9 +243,9 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( ) "#, ) - .bind(scalar_payload) - .bind(json_payload) - .bind(entry_payload) + .bind(&scalar_payload) + .bind(&json_payload) + .bind(&entry_payload) .execute(&pool) .await?; @@ -311,18 +324,9 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( ) .fetch_one(&pool) .await?; - assert_eq!( - values.0, - serde_json::from_str::(scalar_payload)? - ); - assert_eq!( - values.1, - serde_json::from_str::(json_payload)? - ); - assert_eq!( - values.2, - serde_json::from_str::(entry_payload)? - ); + assert_eq!(values.0, scalar_payload); + assert_eq!(values.1, json_payload); + assert_eq!(values.2, entry_payload); // The misuse table survives, but its query-operand column went down with // the eql_v3 schema (DROP SCHEMA ... CASCADE drops the domain, which @@ -352,5 +356,158 @@ async fn uninstaller_preserves_application_tables_with_public_domain_columns( "a column typed as an eql_v3 query-operand domain is dropped with the schema" ); + run_shipped_installer(&pool).await?; + + assert_eq!( + schema_count(&pool).await?, + 2, + "reinstall must recreate both EQL-owned schemas without replacing application data" + ); + let values_after_reinstall: (serde_json::Value, serde_json::Value, serde_json::Value) = + sqlx::query_as( + r#" + SELECT + scalar_value::jsonb, + doc_value::jsonb, + entry_value::jsonb + FROM public.eql_v3_uninstall_preserve + WHERE id = 1 + "#, + ) + .fetch_one(&pool) + .await?; + assert_eq!( + values_after_reinstall, values, + "install -> uninstall -> reinstall must preserve every stored byte" + ); + + Ok(()) +} + +#[sqlx::test(fixtures(path = "../fixtures", scripts("eql_v3_text", "v3_ste_vec")))] +async fn uninstall_and_reinstall_preserve_rows_for_every_public_eql_domain( + pool: PgPool, +) -> Result<()> { + let scalar_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.eql_v3_text ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let document_payload: serde_json::Value = + sqlx::query_scalar("SELECT payload FROM fixtures.v3_ste_vec ORDER BY id LIMIT 1") + .fetch_one(&pool) + .await?; + let entry_payload = document_payload + .get("sv") + .and_then(serde_json::Value::as_array) + .and_then(|entries| entries.first()) + .cloned() + .expect("generated SteVec fixture must contain at least one entry"); + + let domains: Vec = sqlx::query_scalar( + r#" + SELECT t.typname + FROM pg_catalog.pg_type t + JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname = 'public' + AND t.typtype = 'd' + AND t.typname LIKE 'eql_v3\_%' ESCAPE '\' + ORDER BY t.typname + "#, + ) + .fetch_all(&pool) + .await?; + assert!( + !domains.is_empty(), + "installer must expose public data domains" + ); + + let mut expected = Vec::with_capacity(domains.len()); + for (index, domain) in domains.iter().enumerate() { + // Names come from pg_type and are quoted before interpolation; values + // remain bind parameters. One independent table per domain makes a + // CASCADE-dropped column observable as missing data, not just a catalog + // discrepancy. + let quoted_domain = domain.replace('"', "\"\""); + let table = format!("eql_v3_preserve_{index}"); + let payload = match domain.as_str() { + "eql_v3_json_search" => &document_payload, + "eql_v3_json_entry" => &entry_payload, + _ => &scalar_payload, + }; + sqlx::raw_sql(&format!( + "CREATE TABLE public.{table} (id integer PRIMARY KEY, value public.\"{quoted_domain}\" NOT NULL)" + )) + .execute(&pool) + .await?; + sqlx::query(&format!( + "INSERT INTO public.{table} VALUES (1, $1::jsonb::public.\"{quoted_domain}\")" + )) + .bind(payload) + .execute(&pool) + .await?; + let identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + expected.push((table, payload.clone(), identity)); + } + + run_shipped_uninstaller(&pool).await?; + for (table, payload, identity) in &expected { + let actual: serde_json::Value = sqlx::query_scalar(&format!( + "SELECT value::jsonb FROM public.{table} WHERE id = 1" + )) + .fetch_one(&pool) + .await?; + assert_eq!(&actual, payload, "uninstall changed data in {table}"); + let actual_identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + assert_eq!( + &actual_identity, identity, + "uninstall changed column identity in {table}" + ); + } + + run_shipped_installer(&pool).await?; + for (table, payload, identity) in &expected { + let actual: serde_json::Value = sqlx::query_scalar(&format!( + "SELECT value::jsonb FROM public.{table} WHERE id = 1" + )) + .fetch_one(&pool) + .await?; + assert_eq!(&actual, payload, "reinstall changed data in {table}"); + let actual_identity: (i64, i16, i64, String) = sqlx::query_as(&format!( + r#" + SELECT c.oid::bigint, a.attnum, a.atttypid::bigint, a.attname::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = 'public' AND c.relname = '{table}' AND a.attname = 'value' + "# + )) + .fetch_one(&pool) + .await?; + assert_eq!( + &actual_identity, identity, + "reinstall changed column identity in {table}" + ); + } + Ok(()) } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4ae3e15b7..5db220f73 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -248,6 +248,12 @@ importers: specifier: ^3.25.76 version: 3.25.76 devDependencies: + '@cipherstash/eql-upgrade-baseline': + specifier: npm:@cipherstash/eql@3.0.2 + version: '@cipherstash/eql@3.0.2' + '@cipherstash/protect-ffi': + specifier: workspace:* + version: link:../protect-ffi '@cipherstash/stack': specifier: workspace:* version: link:../stack @@ -1038,6 +1044,9 @@ packages: '@cipherstash/auth-win32-x64-msvc': optional: true + '@cipherstash/eql@3.0.2': + resolution: {integrity: sha512-E85o0aoOqgCW6RReLtJ0YLh/ExRlmDJo7LlJGpWPoMTVaw+CW8o11DJ4oJIF1vFtuxSVxNULuPzzBuVmpTvvcA==} + '@clack/core@1.4.3': resolution: {integrity: sha512-/kr3UWNtdJfxZtPgDqUOmG2pvwlmcLGheex5yiZKdwbzZJxhV+HMNR9QNmyY5cGwTNV6LrR7Jtp+KjhUAP1qBQ==} engines: {node: '>= 20.12.0'} @@ -3990,6 +3999,8 @@ snapshots: '@cipherstash/auth-linux-x64-musl': 0.44.0 '@cipherstash/auth-win32-x64-msvc': 0.44.0 + '@cipherstash/eql@3.0.2': {} + '@clack/core@1.4.3': dependencies: fast-wrap-ansi: 0.2.0 diff --git a/scripts/__tests__/cli-live-postgres-ci.test.mjs b/scripts/__tests__/cli-live-postgres-ci.test.mjs new file mode 100644 index 000000000..644cf0355 --- /dev/null +++ b/scripts/__tests__/cli-live-postgres-ci.test.mjs @@ -0,0 +1,51 @@ +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { readJsonc } from './lib/read-jsonc.mjs' +import { REPO_ROOT } from './lib/repo-root.mjs' +import { readWorkflow } from './lib/workflows.mjs' + +describe('CLI live-Postgres CI contract', () => { + it('forwards the live database URL through Turbo test tasks', () => { + const turbo = readJsonc(join(REPO_ROOT, 'turbo.json')) + + expect(turbo.tasks.test.env).toContain('STASH_TEST_DATABASE_URL') + }) + + it('supplies the live database URL from the test workflow', () => { + const workflow = readWorkflow('.github/workflows/tests.yml') + const runTests = workflow.jobs['run-tests'].steps.find( + (step) => step.name === 'Run tests', + ) + + expect(runTests.env.STASH_TEST_DATABASE_URL).toMatch( + /^postgres:\/\/[^/]+\/cipherstash$/, + ) + }) + + it('runs the live reinstall suite against both pre-17 and current Postgres catalogs', () => { + const workflow = readWorkflow('.github/workflows/tests.yml') + const runTests = workflow.jobs['run-tests'] + const versions = runTests.strategy.matrix['postgres-version'] + + expect(versions).toEqual([16, 17]) + expect(runTests.strategy.matrix['node-version']).toEqual([22, 24]) + expect(runTests.strategy.matrix.exclude).toEqual([ + { 'node-version': 22, 'postgres-version': 17 }, + { 'node-version': 24, 'postgres-version': 16 }, + ]) + expect(runTests.services.postgres.image).toContain( + '$' + '{{ matrix.postgres-version }}', + ) + }) + + it('serializes live suites that share the EQL schemas', async () => { + const config = await import( + '../../packages/cli/vitest.config.ts?cli-live-ci-contract' + ) + const live = config.default.test.projects.find( + (project) => project.test.name === 'live', + ) + + expect(live.test.poolOptions.forks.singleFork).toBe(true) + }) +}) diff --git a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs index a248f333a..9b581fc01 100644 --- a/scripts/__tests__/lint-no-eql-registry-pins.test.mjs +++ b/scripts/__tests__/lint-no-eql-registry-pins.test.mjs @@ -79,14 +79,12 @@ const cargoForms = (body) => cargoDeclarations('Cargo.toml', body).map((d) => d.form) describe('the tree it actually guards', () => { - it('passes: every EQL dependency resolves in-tree, with nothing exempt', () => { + it('passes with only the immutable CLI upgrade baseline exempt', () => { const { exitCode, output } = run() expect(output).toContain('resolves in-tree') - // No `(N exempt: …)` suffix. The exemption list is empty as of CIP-3744 - // and the success line reports what it excused, so this is the assertion - // that the tree needs no standing permission at all — not merely that the - // one it had is still described accurately. - expect(output).not.toContain('exempt') + expect(output).toContain( + '(1 exempt: packages/cli/package.json :: @cipherstash/eql-upgrade-baseline)', + ) expect(exitCode).toBe(0) }) @@ -802,6 +800,30 @@ describe('the scan reports what it finds', () => { 'b/package.json :: @cipherstash/eql', ]) }) + + it('exempts a test alias without exempting the runtime dependency beside it', () => { + const root = tree({ + 'a/package.json': JSON.stringify({ + dependencies: { '@cipherstash/eql': '3.0.4' }, + devDependencies: { + '@cipherstash/eql-upgrade-baseline': 'npm:@cipherstash/eql@3.0.2', + }, + }), + }) + const result = lint({ + root, + expected: [], + exemptions: new Map([ + ['a/package.json :: @cipherstash/eql-upgrade-baseline', 'test fixture'], + ]), + }) + expect(result.exempted.map((entry) => entry.key)).toEqual([ + '@cipherstash/eql-upgrade-baseline', + ]) + expect(result.offenders.map((entry) => entry.key)).toEqual([ + '@cipherstash/eql', + ]) + }) }) describe('the linter fails when its own configuration goes stale', () => { diff --git a/scripts/lint-no-eql-registry-pins.mjs b/scripts/lint-no-eql-registry-pins.mjs index 830993e28..0bdc21174 100644 --- a/scripts/lint-no-eql-registry-pins.mjs +++ b/scripts/lint-no-eql-registry-pins.mjs @@ -167,7 +167,7 @@ export const EXPECTED_SOURCES = [WORKSPACE_FILE] /** * Declarations allowed to name a registry version, each with the reason. * - * EMPTY, and that is the goal state rather than an oversight. Every entry is a + * Keep this list as short as possible. Every entry is a * place the two halves of EQL can drift apart again, and the reason is what a * later reader needs in order to decide whether it is still true. * @@ -179,9 +179,19 @@ export const EXPECTED_SOURCES = [WORKSPACE_FILE] * DECLARES `@cipherstash/eql` and an existence-based check would have gone on * passing over a standing permission nothing needed. * - * Adding one back means writing the reason down here. Prefer not to. + * The remaining entry is an immutable test fixture rather than a runtime + * dependency. Adding another means writing the reason down here. Prefer not + * to. */ -export const EXEMPT_DECLARATIONS = new Map([]) +export const EXEMPT_DECLARATIONS = new Map([ + [ + 'packages/cli/package.json :: @cipherstash/eql-upgrade-baseline', + 'Test-only immutable upgrade origin: the credentialed live installer test ' + + 'must install a real previously released bundle before the workspace ' + + 'installer upgrades it. Runtime `@cipherstash/eql` and the payload-emitting ' + + 'Rust remain workspace-linked; this alias is never packed for consumers.', + ], +]) /** Files this scan reads, by name. */ const SCANNED_FILES = new Set(['Cargo.toml', 'package.json', WORKSPACE_FILE]) @@ -458,9 +468,9 @@ function collectNpmEntries(file, table, entries, found) { if (named || NPM_ALIAS.test(scalar)) { found.push({ file, - // Always the package, never the key it was found under: both - // hand-maintained lists are keyed ` :: `, so an - // alias filed under its alias name could never be exempted. + // Expected declarations stay keyed by the real package. Exemptions + // use the alias key when present so they cannot excuse a runtime + // declaration of the same package in the same manifest. dependency: NPM_DEPENDENCY, table, key, @@ -614,6 +624,12 @@ export function scanTree(root) { export const declarationId = (declaration) => `${declaration.file} :: ${declaration.dependency}` +/** A renamed npm dependency can be exempted without exempting its runtime twin. */ +const exemptionId = (declaration) => + declaration.form === 'alias' && declaration.key !== declaration.dependency + ? `${declaration.file} :: ${declaration.key}` + : declarationId(declaration) + /** * The whole check, as data. Separated from the reporting below so the tests can * drive every branch — including the two exit-2 ones — by passing a different @@ -627,17 +643,15 @@ export function lint({ } = {}) { const { declarations, sources: read } = scanTree(root) const ids = declarations.map(declarationId) - const registryPinned = declarations - .filter((d) => !d.inTree) - .map(declarationId) + const registryPinned = declarations.filter((d) => !d.inTree).map(exemptionId) return { declarations, ids, offenders: declarations.filter( - (d) => !d.inTree && !exemptions.has(declarationId(d)), + (d) => !d.inTree && !exemptions.has(exemptionId(d)), ), exempted: declarations.filter( - (d) => !d.inTree && exemptions.has(declarationId(d)), + (d) => !d.inTree && exemptions.has(exemptionId(d)), ), // An exemption that is not excusing anything, and an exemption whose reason // was emptied out. Both are the configuration going stale, and both must @@ -745,7 +759,7 @@ export function report(result) { if (result.offenders.length === 0) { const suffix = result.exempted.length ? ` (${result.exempted.length} exempt: ${result.exempted - .map(declarationId) + .map(exemptionId) .join(', ')})` : '' return { code: 0, out: `Every EQL dependency resolves in-tree${suffix}.` } diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index ace23e94a..d9a3a0567 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -419,6 +419,10 @@ Run it whenever query-time behaviour looks inconsistent with a "successful" inst Generates an **EQL v3 install migration**, instead of running SQL directly against the database (`eql install`). Migration-first is the preferred path: the install lands in your migration history and ships to every environment through the same migrate step as the rest of your schema. On Supabase it is the *only* durable path — `supabase db reset` replays the migrations directory, so a direct install is wiped by the next reset. v3 only — there is no `--eql-version` here. +**Generated migrations contain the raw bundle, not the reinstall protocol.** Use +one for a first install. For replacement, use `eql upgrade` or recreate every +dependent object in the same migration; see `eql upgrade` below. + ```bash stash eql migration --drizzle # Drizzle custom migration in drizzle/ stash eql migration --drizzle --supabase # also grant eql_v3 to anon/authenticated/service_role @@ -495,7 +499,17 @@ An applied migration carrying a statement the sweep would have skipped anyway #### `eql upgrade` -The install SQL is safe to re-run — columns and data survive — but it cascade-drops functional indexes that depend on `eql_v3`; recreate them afterward. `upgrade` is v3-only and accepts `--supabase`, `--dry-run`, and `--database-url`. +Encrypted columns and rows live outside the disposable EQL schemas. `upgrade` +captures dependent functional indexes, replaces the schemas, then restores and +verifies the indexes in one transaction. Unsupported dependencies are refused +before mutation; restoration failure rolls back the replacement. Completion +means every definition and restorable catalog property matches, with no +validity/readiness regression. `upgrade` is v3-only and accepts `--supabase`, +`--dry-run`, and `--database-url`. + +Run it in a schema-migration maintenance window. The advisory lock serializes +other `stash` lifecycle commands, not arbitrary DDL from unrelated sessions; +do not create, alter, or drop EQL-backed indexes concurrently. #### `eql status` diff --git a/turbo.json b/turbo.json index a2448e541..ac38cf670 100644 --- a/turbo.json +++ b/turbo.json @@ -18,6 +18,7 @@ "test": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$", ".env*"], + "env": ["STASH_TEST_DATABASE_URL"], "cache": false }, // `packages/bench`'s "build" is `tsc --noEmit` — a typecheck, not a bundle —