Skip to content

feat(ci): native CI regression gate over registry verdicts - #7

Merged
HarperZ9 merged 2 commits into
mainfrom
feat/ci-regression-gate
Jul 3, 2026
Merged

feat(ci): native CI regression gate over registry verdicts#7
HarperZ9 merged 2 commits into
mainfrom
feat/ci-regression-gate

Conversation

@HarperZ9

@HarperZ9 HarperZ9 commented Jul 3, 2026

Copy link
Copy Markdown
Owner

What this ships

A native CI regression gate: crucible ci REGISTRY.

  • --write-baseline FILE captures the current verified-latest verdict per (thesis, claim) as a sealed baseline snapshot (crucible.ci-baseline/v1).
  • --baseline FILE re-derives the current verdicts, compares them against the baseline, exits nonzero on regression, and emits a deterministic PR-comment-ready Markdown claims-by-rounds verdict matrix.
  • A regression is a claim moving MATCH -> DRIFT, becoming UNVERIFIABLE, or dropping out of the verified-latest set (fail-closed: a claim that lost its witnessed standing is never read as held). New claims and improvements do not fail the gate.
  • Every summary cell references the re-derivable assessment packet (the assessment seal); a reviewer re-checks any cell with crucible verdicts REGISTRY --verify.
  • --out FILE writes the Markdown while preserving the gate exit code; --json emits the full gate report.
  • The baseline file seals its own cells, so a hand-edited baseline (a downgraded status) is rejected on load.

The feature rides on the existing accountability layer: it re-reads the same verified-latest state registry stats uses and opens no second verdict path. The gate's own output is a pure function of two snapshots, so it recomputes from the record and cannot be asserted.

Structure

  • crucible/ci_gate.py (255 lines) - the pure core: Cell / Snapshot (sealed) / GateReport, status ranking, gate(), cells_from_latest().
  • crucible/ci_report.py (67 lines) - deterministic Markdown rendering (split from logic per the repo's report.py vs assess.py convention, keeping every file under the 300-line budget).
  • crucible/ci_cmd.py (85 lines) - the crucible ci command handler.
  • cli.py / __init__.py - wiring and public API exports.

Docs

  • README gains a CI regression gate section with the command and a ready-to-copy GitHub Action snippet (assess -> gate -> post the Markdown as a PR comment, if: always() on the comment step so the exit code still gates).
  • CHANGELOG Unreleased entry.

Tests (23 new; suite 306 -> 329 passed, 1 skipped)

The three contract behaviors are asserted directly:

  • no-regression exits 0 with a MATCH summary (test_no_regression_exits_zero_with_match_summary)
  • injected regression exits nonzero and names the drifted claim (test_injected_regression_exits_nonzero_and_names_the_drifted_claim, asserting the drifted claim id appears in both stderr and the Markdown)
  • the Markdown summary is deterministic (test_markdown_summary_is_deterministic_across_runs and a unit-level determinism test)

Plus: dropped-claim regression, drift->unverifiable regression, improvement/new not failing, tampered-baseline rejection, wrong-kind rejection, order-independent seal, JSON output, --out file behavior, and cells_from_latest skipping unknown statuses rather than guessing.

Test evidence

$ python -m pytest -q
329 passed, 1 skipped in 8.46s

$ python -m mypy src/crucible
Success: no issues found in 42 source files

$ python -m ruff check src/crucible/ci_gate.py src/crucible/ci_report.py src/crucible/ci_cmd.py tests/test_ci_gate.py tests/test_cli_ci.py
All checks passed!

Shipped vs specced

Fully shipped and tested; nothing deferred to a spec. Zero-dependency and Fair-Source posture preserved (stdlib only). Zero em-dashes / en-dashes. Experimental nothing; all behavior is exercised by tests.

Honest limitations

  • A baseline is committed and refreshed by a maintainer when the posture legitimately advances; the gate does not auto-advance the baseline (by design: an auto-advancing baseline cannot catch a slow regression).
  • The gate compares verdict status (MATCH/DRIFT/UNVERIFIABLE), not margin deltas within a status; a claim degrading while still MATCH reads as held. crucible drift remains the margin-delta view.
  • The gate keys on (thesis_id, claim_id); a claim whose id changes (e.g. its content hash shifts) is seen as a drop + a new claim, which is the fail-closed reading.
  • Pre-existing, out-of-scope: tests/test_browser_evidence.py has an unrelated ruff I001 import-order warning that exists on main; not touched here.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

🤖 Generated with Claude Code

Zain Dana Harper and others added 2 commits July 2, 2026 18:31
Add `crucible ci REGISTRY`: capture a sealed baseline of the current
verified-latest verdict per (thesis, claim), then gate a later commit by
re-deriving the current verdicts and failing nonzero on regression (a
claim moving MATCH -> DRIFT, becoming UNVERIFIABLE, or dropping out of
the verified-latest set, fail-closed). Emit a deterministic
PR-comment-ready Markdown matrix whose every cell references the
re-derivable assessment packet seal.

The gate rides on the existing witnessed assessments: it re-reads the
same verified-latest state `registry stats` uses and opens no second
verdict path. The pure core (ci_gate) ranks statuses over two snapshots;
the render layer (ci_report) emits the Markdown; the baseline seals its
own cells so a hand-edited baseline is rejected on load.

Tests assert the three contract behaviors: no-regression exits 0 with a
MATCH summary, an injected regression exits nonzero and names the drifted
claim, and the Markdown summary is deterministic. README documents the
command and ships a ready-to-copy GitHub Action snippet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the CI lint gate on this branch; the import fix already landed on
main but this branch predated it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@HarperZ9
HarperZ9 merged commit e73e9e3 into main Jul 3, 2026
2 checks passed
@HarperZ9
HarperZ9 deleted the feat/ci-regression-gate branch July 3, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant