tools: draft RELEASE-NOTES.md from the release range#38
Merged
Conversation
Automates the half of release notes that is a computer's job, and deliberately refuses the other half. WHAT IT FILLS IN (previously looked up by hand, every release): - the previous tag, the commit range, and the compare URL - commits grouped by conventional-commit type, with the PRs they came from - test coverage MEASURED by running the suites and reading each one's own count, rather than recalled — a stale count is a small lie nobody catches WHAT IT WILL NOT DO: write the prose. Choosing which 2 of 15 commits a user actually cares about, and how to frame them, is judgement. v0.9.2 needed the sentence "credits are a change of unit, not of price" — no commit subject contains that, and inventing it from subjects is how release notes become unreadable. The draft marks those spots TODO instead of guessing. IT SHOWS ITS WORKING. Every commit classified as internal is listed under "excluded" in the scaffolding block. A tool that silently drops commits is worse than no tool: you cannot review an omission you never see. Verified against the real v0.9.1..HEAD range — it excluded exactly what I had excluded by hand (the CI Playwright fix and the notes commits) and showed all three. Guardrails: rejects a missing/non-semver version, refuses a version whose tag already exists (notes are written BEFORE tagging so the tag contains them), warns on a dirty tree, and EXITS NON-ZERO if any suite fails — a red suite is a release blocker, not a footnote. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new scripts/draft-release-notes.mjs helper to generate a factual draft of RELEASE-NOTES.md for a given upcoming semver release, including the previous tag/range, commit grouping by conventional-commit type, and measured test-suite counts by executing the repo’s Node-based unit tests.
Changes:
- Introduces a release-notes drafting script with semver argument validation and optional
--writeoutput. - Computes the release range from the latest
v*tag and renders grouped commit candidates plus an “excluded/internal” scaffolding block for manual review. - Measures test coverage by running extension unit-test files and parsing their reported “N tests passed” counts, failing hard if any suite fails.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
All five points were real. Two of them made the tool quietly report LESS than the truth, which is the failure mode that matters most for something whose whole job is producing accurate facts. 1. prevTag was interpolated into a shell string. A tag name is repo-controlled but still runtime-discovered input, so this is now defended twice: all git calls go through spawnSync with an argv array (a shell cannot be quoted out of one), and the base tag must match ^vX.Y.Z$ before use. Shape matters independently of injection — a stray v0.9.2-rc1 sorts into the 'v*' glob and would silently produce the wrong range, wrong commit list and wrong compare link. Verified: v0.9.2-rc1, v-wip and "v0.9.2; rm -rf /" are all rejected. 2. PRs were only detected from merge commits, so on a squash-merging repo the list would read "(none detected)" while every subject carried "(#123)". Both shapes are now collected. 3. Suites were keyed by basename, so two extensions with the same test filename would produce an ambiguous "biggest suites" list. Keyed by relative path now. 4. The coverage line printed "N cases in total" while suites whose summary could not be parsed silently contributed 0 — authoritative-sounding and UNDER- reporting. This one bit immediately: 5 of 24 suites do not print a count, so "273 cases in total" was wrong. It now says how many suites were counted and names the ones that were not. 5. The fix/perf heading also carried reverts; renamed to say so. Also fixes a bug I introduced while making #2: prNumbers became a Set, but the renderer still read .length, which is undefined on a Set — so the PR list emptied itself silently. Caught by re-running against the real range (expected #33-36, got "(none detected)"). Materialised to a sorted array. Verified end to end against v0.9.1..HEAD: PRs #33-36 detected, coverage line now honest about the 5 unparsed suites, and all guardrails still fire. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
scripts/draft-release-notes.mjs:124
- If
extensions/is missing,measureSuites()currently returns an empty suite list, which then renders as “0 suites … all green” and “0 cases counted”. Since the output is meant to be factual, this should be treated as a hard error (likely means the script is running in the wrong checkout / directory).
function measureSuites() {
const extRoot = join(REPO, 'extensions');
if (!existsSync(extRoot)) { return { suites: [], failed: [] }; }
const suites = [];
…measured tree green PR #38 review. Reproduced before fixing — run from scripts/ and the tool did not fail, it produced: - **0 suites** across the bundled extensions — all green. It measured nothing and certified it. --write would also have dropped RELEASE-NOTES.md into whatever directory you happened to be standing in. Fixed at the root: everything is anchored to `git rev-parse --show-toplevel` instead of process.cwd(), so the invocation directory stops mattering. Verified from scripts/ — now reports the real 24 suites / 273 cases. Also fixed the CLASS, not just the instance. Zero suites is "nothing was measured", not "all green", and the two must never render the same. A backstop now refuses to draft at all in that case. Anchoring should make it unreachable; it stays because the cost of being wrong is a release note certifying coverage nobody ran. Verified in a scratch repo with no extensions/: exits 1 with a plain explanation. Running outside a git repo exits 1 too, rather than silently treating some parent directory as the root. This is the fourth false-green in this codebase this week (tsc --noEmit checking zero files, pgrep matching the verifier itself, a status assertion that passed on the SPA fallback). The shape is always the same: a check that cannot fail reads exactly like a check that passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automates the half of release notes that is a computer's job, and deliberately refuses the other half.
What it fills in
Everything that was previously looked up by hand each release — and therefore misremembered:
What it will not do
Write the prose. Choosing which 2 of 15 commits a user actually cares about, and how to frame them, is judgement. v0.9.2 needed the sentence "credits are a change of unit, not of price" — no commit subject contains that, and manufacturing it from subjects is exactly how release notes become the thing nobody reads. The draft marks those spots
TODOinstead of guessing.It shows its working
Every commit classified as internal is listed under excluded in a scaffolding block you delete before committing. A tool that silently drops commits is worse than no tool — you cannot review an omission you never see.
Verified against the real
v0.9.1..HEADrange: it excluded exactly what I had excluded by hand (the CI Playwright fix, the notes commits) and displayed all three for review. It also detected PRs #33–#36 and warned that the working tree was dirty.Guardrails
v0.9.2/ not semverNot included
A line in
docs/RELEASING.mdpointing at this. That file currently carries ~38 lines of uncommitted documentation (the version-stamp check, which it notes "fails silently"), and I did not want to sweep someone else's unfinished work into this commit. Worth rescuing separately — it has survived several releases only by luck.🤖 Generated with Claude Code