AI Output Guardrail & Risk Mitigation System

ground-truth-lock

Stop LLM rewrites from inventing facts about your data.

Pass any LLM-rewritten dict + the canonical source-of-truth + a list of paths that must not change. Get back the rewrite with those paths locked back to truth, plus a list of any invented numbers the LLM slipped in. Use it as the boundary check between any LLM call and the downstream consumer.

from ground_truth import lock_facts

locked, problems = lock_facts(
    output=llm_response_dict,
    source=master_truth_dict,
    immutable_paths=["experience.*.title", "personal.email"],
    reject_invented_numbers=True,
)

If the LLM rewrote experience[0].title from "Senior Consultant" to "Principal Product Manager", locked will have it back as "Senior Consultant". If the LLM wrote "Cut latency 47%" when the source said 26%, problems will contain invented numbers: ['47%'].

Single file. No dependencies. MIT.

---

Why this exists

Most LLM "rewrite this dict for X" calls work fine on a happy day. The failure mode that bites you is silent: the model occasionally upgrades labels, invents metrics, or rewords identity fields to make the rewrite read better. By the time the output crosses a boundary (rendered to a document, sent to a downstream system, signed by a stakeholder, persisted to a DB), the fabrication is already shipped — and you find out weeks later when an audit, a reference check, or an analytics query surfaces the contradiction.

The naïve fixes don't hold:

• "Just prompt the model harder" — fails ~5% of the time, which is still fatal at production volume.
• "Re-parse the output and compare to source" — works, but every consumer reinvents the same regex/walk code badly.
• "Use a JSON schema" — catches structure drift, misses value drift.

This is the small library you needed: declare what's locked, get it back from source, and surface any numeric drift in the same call.

What it does

• Path-based locking. Pass dotted paths ("experience.*.title", "personal.email") and the matching values in output are overwritten from source after the LLM returns. The * wildcard works for list indices and dict keys.
• Invented-number detection. Pulls every numeric token ("47%", "3.5 weeks", "$1M", "1,500 users") from the output and flags any that don't appear in the source. Normalises whitespace, decimals, and trailing punctuation so equivalent tokens compare equal.
• Whitelist for unavoidable trivia. Default whitelist includes 0–10 and 100 so common phrasing ("3 pillars", "first 100 days") doesn't false-flag. Pass your own to extend or replace.
• Strict mode. strict=True raises FactDriftError instead of returning problems — useful when you want to fail loudly.

What it doesn't do

• It doesn't call the LLM. It's the post-LLM check.
• It doesn't enforce schema. Use Pydantic / JSON Schema for that.
• It doesn't catch text-level paraphrasing drift (renaming "Apple" to "the Cupertino giant"). For free-text fact consistency you need a heavier checker; this library handles structured data + numeric integrity, which covers the cases that actually bleed into production.

What I learned operating this in production

This library was extracted from the LLM-rewrite layer of an autonomous content pipeline that processed 600+ generated documents against a fixed source-of-truth. Three failure modes drove the design:

• The model rewrites identifier fields to "improve fit," not to lie. Asked to rewrite a structured record to match a target context, the model would upgrade a position-style label from its truthful value to a higher-prestige one because the new label scored better on the model's internal alignment heuristic. The model isn't being adversarial — it's optimising the wrong objective. The fix can't be prompt-only; the output structure must declare which fields are facts about the world and which are framing the model may choose.
• Metric drift is the most expensive hallucination. When 26% reduction becomes 47% reduction across two LLM rewrites of the same source record, the contradiction surfaces the moment any downstream reader compares two variants side-by-side. Numbers are the trust currency in a generated document; one drift kills the rest.
• Regex over the whole output blob beats per-field validators. Trying to walk the dict and validate each numeric leaf separately produces a maze of special cases. Extracting every numeric token from output and source, normalising both, and computing the set difference catches more drift in 5 lines than 50 lines of per-field code.

The pattern across all three: the output is the contract. Validation that lives in the model's head doesn't survive production; validation that lives in code does.

Run it

python ground_truth.py

The module's __main__ block runs a demo that shows title fabrication being locked back to truth and an invented 47% being flagged. No install needed — standard library only.

pip install ground-truth-lock  # (when packaged)

License

MIT — see LICENSE.

---

Companion library to resilient-llm-router. Extracted from the production layer of an autonomous agent pipeline; the application-specific logic is not included.