Skip local-input tests exactly when the input is not the pinned one

[MaxGhenis]

Ten tests failed locally on the maintainer's Mac but skip or pass in CI. CI Tests run 36239836717 on 672e677 was green. Each test reads a local input that has drifted from its pin, while its skip condition checked only whether the input was present. This PR makes each skip exact: skip when the local input is absent or is not the pinned one. It changes tests only, and no committed runs/*.json, gates.yaml, engine/* or src/ file changes.

Per test

Test	Local cause	Change
cola_track_a/test_statutory_capture_oracle.py::test_capture_script_reproduces_the_committed_capture	The capture records the checkout's revision (a03e82e503), but the skip checked only the file digests.	The capture script still runs, with its own checks. Then the test skips unless git log -1 --format=%h in the checkout gives exactly the captured revision, and only then compares bytes.
…::test_oracle_parameters_pass_every_statutory_check	Same	statutory_value_checks still runs on the pinned files at any commit. Only the final revision-prefix assert skips unless git's revision starts with the captured one.
min_benefit_track_m/test_quarter_of_coverage_recapture.py::test_the_recapture_equals_the_committed_file	The skip checked only that the checkout directory exists. The capture records the QC file's sha256, the wage index's sha256 and the revision.	Skip first if either file is not the captured one. build() still runs its statute comparison. Then skip unless git reports the captured revision, and only then compare bytes.
data/test_psid_codebook_inventory_reproduction.py::test_all_codebook_eras_and_adjudication_reproduce_byte_for_byte	Homebrew moved poppler to 26.09.0; the inventory pins 26.04.0 (DictionaryDriftError).	Skip when inventory._pdftotext_version() raises DictionaryDriftError.
test_rebuild_amendment11_missing_reason_authority.py × 3 (fresh_47_source_build, r05_direct_cli, successful_write_normalizes_output_mode)	Same poppler drift. The builder and replay stop with "Poppler version drift before semantic parsing"; the replay reports this as "production source authentication failed".	Skip unless extraction.pdftotext_version() equals the literal 26.04.0 that both scripts compare against.
test_axiom_benefit_bridge.py::test_actual_engine_reproduces_published_case_{a,b}…	The 49f2225c engine was lost in a cleanup and rebuilt bit-identical (e4116175…). Its .provenance.json receipt now carries the rebuild's built_at (5ba0dde9…, pinned 469d13fd…). The Axiom launch plans were re-pinned to the new receipt and the file is uchg-locked, so the old receipt cannot be restored.	Skip when any bound file is absent, a symlink, or not its pin. This is the same per-file judgement AxiomEngineBinding.verify() makes; the transport digest was an assert and is now a skip.
test_track_c_aime_agreement.py::test_actual_engine_agrees_with_the_oracle_on_invented_careers	Same receipt	Same helper

What stays a failure. Once every input is the pinned one, every existing assertion runs unchanged. Checks that do not depend on the revision run before the revision skip. That includes comparing the recaptured documents with their revision fields masked. The revision comes from git directly, not from the loaders under test, so a change to how they record it still fails. The manifest-binds-engine check in verify() is deliberately not a skip reason: with every file pinned it can fail only on the repository's own pins, so it still fails the test.

Reproducing tests 1, 2 and 4. I pointed POPULACE_DYNAMICS_PE_US_DIR at a policyengine-us checkout at 2fac47ced9 whose pinned SSA files match:

• Before this PR, all three fail on the revision field. That is the same trio that failed in the earlier sequential runs, while test_every_amount_is_the_statutes passed.
• After it, all three skip, with a reason naming both revisions.

At the pinned a03e82e503 checkout, all three pass. tests/cola_track_a run sequentially passes (154). Which checkout, or which git failure, the earlier runs actually saw is undetermined; the tests now skip in either case.

Invariants

• Engine skip is exact (new differential test test_unpinned_inputs_are_exactly_the_files_verify_rejects). For each bound file, and for each of changed, absent and symlink, unpinned_engine_inputs reports exactly that role, and verify() raises BindingMismatch naming it. With no drift, the list is empty and verify() passes.
• The manifest check is not skipped (test_a_manifest_that_does_not_bind_the_engine_is_not_skipped). A pinned manifest that does not bind the engine gives an empty skip list, and verify() still raises "does not bind engine".
• tests/tier_counts.json: unit rises from 3650 to 3652, and the tier policy test passes on the full collection.

Local results (this Mac, 49ffdea + this commit)

• Touched modules: 109 passed.
• 7 skipped, each for the expected reason: the engine receipt (3) and pdftotext 26.09.0 (4). Tests 1, 2 and 4 pass on the pinned checkout.
• Ruff and black are clean.

Review

• Round 1 (Opus 5.5, Subfleet 20260927-043446-pr475-review): CHANGES_REQUESTED.
  • The medium finding: the oracle test skipped before its revision-independent checks ran. Fixed in 62684be, together with the lows: the independent git probe, the scripts' literal pin, and the regex escape.
  • Left as is: the nit about importing the helper from another test module. There is precedent for it: from tests.test_closed_cohort_history import step.
• Round 2 (Opus 5.5, 20260927-044008-pr475-review-r2, with execution): APPROVE on 62684be.
  • The reviewer ran the touched modules (109 passed, 7 skipped), the moved checkout and the tier test.
  • It also ran a mutation battery against the code under test.
  • Its non-blocking findings are applied in 8d69f02:
    • the captures are compared with the revision fields masked, before the revision skip, so a wage-base change for 2011–2030 is now caught on a moved checkout;
    • the git probe skips when git is absent;
    • the QC capture must record a single revision.
• Round 3 (delta review of 8d69f02): pending.

Follow-up (not in this PR)

Chip task_789c9ae5 covers this. Re-pinning REVIEWED_PINS to the rebuilt receipt (5ba0dde9…) and to the reviewed rebuild manifest (engine-rebuild-20260925/lazy-engine-binding-rebuild-20260925.json, b8c029d1…) would restore the actual-engine tests on this Mac. That is a change to reviewed pins, so it is left out.

axiom: n/a: test-only skip conditions

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
social-security-model	Ready	Preview	Sep 27, 2026 10:33am UTC