golden-dataset

Turn real traffic/traces into a versioned, deduped, coverage-tracked LLM regression suite — a curation accelerator with a human-confirmed golden gate.

A retrieval-augmented generation pipeline in Python with a rigorous offline evaluation harness. Chunks and embeds documents, retrieves by vector similarity, and generates grounded answers — with pluggable LLM providers (including a deterministic local fake for tests) and metrics for retrieval quality and answer faithfulness. No API key required.

A Claude skill for building golden sets to test AI systems — matching, RAG, LLM-as-judge — without false greens.

A markdown-first governance framework for building trustworthy golden datasets with clear ownership, labeling, adjudication, leakage controls, versioning, retention, and release evidence.

Why a passing benchmark isn't safe to ship: a free 2-stage (benchmark + replay) validation recipe for LLM model swaps & prompt changes, run on flat-rate coding-agent subagents — no eval API bill.

Veris: agent-agnostic local LLM evaluation and regression testing gate. Runs golden datasets against any callable, scores with DeepEval, compares rolling SQLite baselines, and returns CI-ready exit codes.

Catch LLM quality regressions in CI: a golden-set regression gate with calibrated graders (grounding, hallucination, tool-call, LLM-judge) that fails the PR when answer quality drops.

SAGE — Skills for Agent-Grade Evaluation. Two Claude Code skills: assess data before you build, measure quality after. From 3 real projects. 智能体评估方法论：前置数据评估+后置质量评估。

Evaluation harness for LLM apps: YAML suites over golden datasets, deterministic scorers and LLM-as-judge, and case-by-case prompt-regression diffing that gates CI.

Agentic diagnostic assistant for distributed-system incidents: multi-turn RAG, hypothesis updates, evidence packing, golden evals, and failure-attributed run reports.

Catch LLM quality regressions before they reach production — eval-driven CI/CD with LLM-as-Judge scoring, Wilson 95% CI diffing, and automatic PR alerts.

Evaluation-driven LLM harness for support-ticket intent classification: golden dataset, per-class & per-difficulty accuracy, latency/cost, and a pytest regression gate.

Portfolio-grade AI quality evaluation lab with golden datasets, prompt regression, groundedness checks, hallucination tests and CI thresholds.

Turns a brand's published policy pages into a machine-checkable agent evaluation, with failures tiered by what they actually cost.

Evaluation-driven architecture for a source-grounded AI business advisor

Deterministic evaluation harness for AP document-matching agents. Scores 3-way findings against a hand-audited, held-out golden dataset: per-category precision and recall, over-flagging measured on a zero-defect control, byte-reproducible scorecards, answer key structurally out of reach.

A hybrid RAG pipeline whose point is the evaluation gate: BM25 plus dense cosine fused by Reciprocal Rank Fusion over a deliberately tiny fixture corpus, scored against a 40-question golden set by a harness that exits non-zero below threshold. Pure Python stdlib, 0 pip dependencies, no GPU, no network. A discipline demo.

Evaluation-driven CI for AI features: tiered golden sets, paired bootstrap significance on per-item score deltas, validated judges, and a PASS/BLOCK/INCONCLUSIVE merge gate that escalates tiers only when it cannot decide. 300 real gemma4 responses, reproducible offline.

Golden Datasets Templates for AI Evaluation

CI/CD regression detection for LLM features: golden-dataset evals, LLM-as-judge scoring, run diffing, Slack alerts, merge gating