Run only the CI a pull request can affect - #296
Merged
Merged
Conversation
jserv
force-pushed
the
ci-refine-proof
branch
4 times, most recently
from
August 14, 2026 20:11
a05a4f7 to
a84293b
Compare
Every pull request paid for every workflow whatever it touched: seventeen macOS runners proving seventeen targets, three more analyzing a build, one building and testing, for a branch that edited a test script. The proofs are the bulk of that, and a target whose inputs a branch does not touch already has the verdict the base established. scripts/proof-scope.py decides which proof targets a set of changed files can reach. It takes each proved source's include closure from the compiler's own -MM, preprocessed with that target's VERIFY_<T>_CPP_DEFS and unioned with its VERIFY_<T>_SCAN list, so the scan sees what the prover sees. Inputs no closure can see widen the scope to everything: the makefiles, the checkers the recipes run, the Frama-C stub headers, the workflow itself. So does every "cannot tell" answer, whether an unresolvable base ref or a compiler scan that cannot be trusted. Narrowing on a guess turns a speed-up into a correctness problem; proving something twice does not. verify.yml asks that twice. Which targets to prove is one question; which mutation sets to re-run is the other, and it is narrower, because a file that only decides what runs cannot change whether a target rejects a broken source. SCHEDULING_FILES names those files, everything else counts as judging so an unclassified newcomer widens both, and the self-test refuses a scheduling workflow that carries a prover budget or a make invocation. An empty scope is a verdict rather than a failure. A push to main and a merge queue run prove and mutate everything, so the guarantee on the branch a pull request merges into is never the narrowed one. The jobs follow the same shape. A mutation leg proves an unmutated control through the very rule the proofs use, so the proving job takes only what the matrix will not cover, which over the last 400 commits is nothing at all: sharding the proofs cost 82 seconds of setup per leg to do 23 seconds of proving. Both halves report under the check name that predates the split, which covered proofs and mutations together and still does, and check-proof-targets.py now asserts every macOS job is reachable from it. The setup those jobs share, and the toolchain pins they have to agree on, live in .github/actions/framac; the analyzer moves to Frama-C 33.0, checked rather than assumed, at 803 of 803 obligations across all 17 targets with the mutation gate still biting. build.yml and static-analysis.yml take the cheaper mechanism their inputs allow, a paths-ignore list of what cannot reach them, on pull_request only so there is one list per file and nothing can drift. The analysis jobs analyze "make elfuse" and nothing else, which is why tests/ is inert for them. The failure directions differ from the proof scoping, and that is why one needs a self-test and the other does not: a stale entry there matches nothing and the workflow runs more, while a stale harness entry widens nothing and the proofs run less. lint.yml loses its path filter entirely, which is what makes it the place the self-test can live. It had been ignoring markdown, and check-skill-refs.py validates nothing but the markdown under .claude, which the other two workflows treat as inert, so a documentation-only pull request ran no CI at all. That checker was also wired into no workflow until now, and it failed on its first run, naming two files the CI split had already removed. lint.yml runs its package-free checks first and carries if !cancelled() on every step, the apt setup included, so one failure reports alongside the rest instead of leaving the tool-dependent checks to fail on a missing binary. verify-mk.py refuses a VERIFY_<T>_SRC naming two sources rather than keeping the first, since every consumer assumes one, and check-char-signedness.py, the last reader with its own regex over those lines, goes through the shared table now. Two harness entries answer a question about content rather than about the path, because the name rule was measurably too blunt: over the last 400 commits the top-level Makefile was the sole reason 36 of the 60 full-scope runs happened, every one of them a rule for a test binary or a source added to SRCS. It reaches a proof only through its include lines, since it defines none of the variables the verify recipe reads, and mk/config.mk qualifies the same way because it defines no rules at all. mk/common.mk does not: it owns the $(BUILD_DIR) rule every verify target carries as an order-only prerequisite, which a slicer reading values cannot see. That slicer drops what is provably inert rather than keeping what it recognizes, so an unrecognized construct widens. Reviewers found the difference the hard way: sorting the slice hid a reordered include, and override, define, a target-specific assignment, vpath and unexport each reached a proof while every line an earlier pattern read stayed byte-identical. All of them are self-test cases now, alongside the conditions the approach rests on, that neither file defines a verify rule or a target the verify rules depend on. Which names count as proof-relevant is a fixpoint, not a union: start from what mk/verify.mk expands, since that is the only file the recipe reads, then follow definitions. The union was self-defeating, because mk/config.mk references its own test lists, so NATIVE_TESTS and its neighbours counted and any edit to them re-proved everything. The seed still takes references outside an assignment's right-hand side in the other three makefiles, or mk/common.mk's "ifeq ($(V),1)", which picks the Q every recipe expands, would leave V inert. Over the same 400 commits: 231 run no proof at all, 16 run the full set, and the targets mutated fall from 1103 to 374. The data model comment was wrong where it mattered most. It claimed plain char is unsigned on arm64 macOS; it is signed there, and unsigned on aarch64-linux, which is the cross toolchain for the guest tests, both answers taken from the compilers rather than from memory. So gcc_x86_64 matches the platform the proved sources compile for on all four properties, not three. Frama-C 33 does ship a macos_arm machdep and it cannot be used here: it is not GCC-based, and the atomics stub pulls in Frama-C's __fc_gcc_builtins.h, which refuses __int128 outside a GCC-based machdep. Every target aborts at parse time, which is now the recorded reason for the pin. The mutation matrix is packed rather than one leg per target. A leg spends about 90 seconds on Homebrew and the opam switch before it proves anything, and GitHub bills wall time rounded up to the minute, so 17 legs cost 85 macOS-minutes where 5 buckets cost 61; five also fits one concurrency wave, so the wall clock improves too. check-mutants.py packs because it owns the mutation counts it packs by, and each leg still runs one target at a time so a failure names the target rather than the group. Three defects in this machinery came out of review rather than out of the self-test, and all three are now cases in it. A "printf | grep -q" under pipefail reports "unchanged" once the diff fills the pipe buffer, which would have skipped clang-format and cppcheck on exactly the largest pull requests. check-mutants.py was taking its source table through proof-scope.py, putting a file classified as unable to affect a mutation verdict on the path that decides which file gets mutated. And check-proof-targets.py's job splitter read workflow sub-keys as jobs while missing any job id not starting lowercase, so the check meant to notice a prover job leaving the required check could not see one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every pull request paid for every workflow whatever it touched: seventeen macOS runners proving seventeen targets, three more analyzing a build, one building and testing, for a branch that edited a test script. The proofs are the bulk of that, and a target whose inputs a branch does not touch already has the verdict the base established.
scripts/proof-scope.py decides which proof targets a set of changed files can reach. It takes each proved source's include closure from the compiler's own -MM, preprocessed with that target's VERIFY_CPP_DEFS and unioned with its VERIFY_SCAN list, so the scan sees what the prover sees. Inputs no closure can see widen the scope to everything: the makefiles, the checkers the recipes run, the Frama-C stub headers, the workflow itself. So does every "cannot tell" answer, whether an unresolvable base ref or a compiler scan that cannot be trusted. Narrowing on a guess turns a speed-up into a correctness problem; proving something twice does not.
verify.yml builds its matrix from that answer on a pull request, so an untouched target gets no runner rather than a runner that skips its work. An empty scope is a verdict rather than a failure, and only the mutation gate decides what counts as a pass; the verify job mirrors it, so the rule lives in one place instead of two shell blocks that have to be edited together. A push to main and a merge queue run prove everything, so the guarantee on the branch a pull request merges into is never the narrowed one.
build.yml and static-analysis.yml take the cheaper mechanism their inputs allow, a paths-ignore list of what cannot reach them, on pull_request only so there is one list per file and nothing can drift. The analysis jobs analyze "make elfuse" and nothing else, which is why tests/ is inert for them. The failure directions differ from the proof scoping, and that is why one needs a self-test and the other does not: a stale entry there matches nothing and the workflow runs more, while a stale harness entry widens nothing and the proofs run less.
lint.yml loses its path filter entirely, which is what makes it the place the self-test can live. It had been ignoring markdown, and check-skill-refs.py validates nothing but the markdown under .claude, which the other two workflows treat as inert, so a documentation-only pull request ran no CI at all. That checker was also wired into no workflow until now, and it failed on its first run, naming two files the CI split had already removed.
Summary by cubic
Runs only the CI jobs a pull request can affect, widening to the full set on any uncertainty. Previously every PR ran all proofs, mutations, builds, and analysis; now proofs/mutations narrow to targets reachable from the diff, builds/analysis use path filters, and lint always runs to cover self-tests and docs checks.
Proofs and mutations
scripts/proof-scope.pyto derive per-target reach from each source’s compiler include closure withVERIFY_<T>_CPP_DEFSunioned withVERIFY_<T>_SCAN; any “cannot tell” widens to all. Separates “scheduling-only” files and self-tests to forbid budgets ormakein scheduling workflows.verify.ymlreads targets frommk/verify.mk, narrows matrices on PRs, and proves/mutates the full set onpushandmerge_group. Introducesverify-proofsto backfill anything mutation legs don’t prove;verify-mutants-gatestays the required aggregate andscripts/check-proof-targets.pyasserts the matrix is derived and that macOS jobs are reachable from it../.github/actions/framacfor shared setup and moves toFrama-C 33.0.scripts/check-mutants.pydelegates changed-only selection toproof-scope.py;scripts/verify-mk.pyexposes targets/SCAN/CPP_DEFS, discovers proof workflows, and rejects multi-sourceVERIFY_<T>_SRC.scripts/check-char-signedness.pyreads the shared table.mk/verify.mkcorrects data-model notes (plain char is signed on arm64 macOS) and records why themacos_armmachdep cannot be used.CI and repo
build.ymlandstatic-analysis.ymlusepaths-ignoreonpull_requestonly (tests are inert for analysis).lint.ymldrops its path filter, runs package-free checks first withif: !cancelled(), runsproof-scope.py --self-test, and wiresscripts/check-skill-refs.pyso docs-only PRs still run CI..ci/check-security.shnow flags “agent scratch markers”; removes one fromsrc/syscall/asyncio.c. UpdatesSKILL.mdto referencemk/verify.mk.Frama-C 33.0. To reproduce CI scoping locally:make verify-mutants MUTANT_SINCE=<base>.Written for commit 8993631. Summary will update on new commits.