A three-step agentic pipeline that stops ctaio.dev publishing a factual claim its own cited sources cannot support.
Built on 22 August 2026 for the We The Flywheel Sr Agentic Engineer task. It runs against real claims read off the live site that morning.
ctaio.dev is 659 URLs across six locales, and it is heavily programmatic: about sixty salary pages, a certifications cluster, tool comparisons, a who-is-hiring archive. Localised pages mirror English ones. The brand sells "verifiable workflow details" to senior engineers, which is an audience that checks numbers. At that page count no human review scales, and the failure that costs the most is not an outage, it is a page that is quietly wrong and stays up.
It already fails. /de/gehalt/cto-gehalt/ states German CTO salaries in euros and its sources
section cites Kruze Consulting, Riviera Partners, Glassdoor, Indeed and the US Bureau of
Labor Statistics. BLS publishes no German wage data. The citations were carried across from the
English page onto a localised one where they cannot support the claims. The two pages are also two
months out of sync, English updated 2 June 2026, German 1 April 2026.
That is the defect class this catches: a claim whose cited source cannot possibly support it.
| Step | What it is | What it does |
|---|---|---|
| 1. extract | agent, Claude with a forced tool schema | page in, typed Claim objects out. A malformed answer is a schema error, not something to parse |
| 2. verify | agent plus a deterministic coverage table | each claim becomes SUPPORTED, CONTRADICTED or UNRESOLVED, with the evidence |
| 3. gate | ordinary code, no model | fails closed. Anything not SUPPORTED is quarantined and journalled |
The gate is deliberately not an agent. A model asked to grade its own output is not a gate. The model may propose claims and may classify them; the thing that decides whether to publish is code you can read, diff and test.
UNRESOLVED is a real return value, never a default. A claim that could not be checked is not a claim that passed. The gate treats it exactly like a contradiction. That single decision is what separates a gate from a formality.
Whether the US Bureau of Labor Statistics publishes German wages is a fact, not a judgement, so it
lives in sources.py as a table rather than in a prompt.
pip install -r requirements.txt
python -m pytest tests -q
python -m claimguard.cli fixtures/de-cto-gehalt.json fixtures/en-cto-salary.json
Neither the tests nor that run needs an API key: the two agent steps are exercised through fixtures
captured from the live site, so the logic is reviewable on its own. extract.extract_agent is the
real step 1 and needs ANTHROPIC_API_KEY.
https://ctaio.dev/de/gehalt/cto-gehalt/
2 passed, 2 blocked
[CONTRADICTED] DAX-Konzern: 250.000 EUR und mehr
no cited source can support this claim. Kruze Consulting: covers ['US'], claim is DE;
reports ['USD'], claim is in EUR | Riviera Partners: covers ['US'] ... |
Bureau of Labor Statistics: covers ['US'], claim is DE; reports ['USD'], claim is in EUR
[UNRESOLVED] Muenchen-Premium 15 bis 20 Prozent gegenueber dem Bundesdurchschnitt
the page states a figure and cites nothing
PUBLISH BLOCKED
https://ctaio.dev/en/salary/cto-salary/
3 passed, 0 blocked
The English page passes clean. That matters as much as the German page failing: a checker that blocks everything is indistinguishable from a broken one, so it has to be able to say yes.
Every claim, passed or blocked, is appended to out/evidence.jsonl with the check that ran and the
inputs to it. Not a log line, a record you can audit later.
Cross-locale agreement (verify.check_locale_agreement) is implemented and tested but not wired
into the CLI, because it needs a locale-sibling map and an exchange rate source, and inventing
either would have meant shipping a number I could not defend. It returns UNRESOLVED when it has no
sibling, which is the correct behaviour and not a stub.