llm-eval-ci

An eval harness for LLM-powered applications. Define graded test suites in YAML, run them concurrently against a model, and get pass rate, cost, and latency per configuration -- with an exit code you can gate CI on.

The premise: prompt changes are code changes, and code changes need regression tests. Most LLM projects have no way to answer "did that prompt edit make things worse?" other than trying a few examples by hand. llm-eval-ci makes that answer a number, and makes a regression fail the build.

Quick start

python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"

# Runs offline against the mock provider -- no API key needed.
.venv/bin/llm-eval run suites/capitals.yaml --provider mock

# Against the real model:
export ANTHROPIC_API_KEY=sk-ant-...
.venv/bin/llm-eval run suites/capitals.yaml

capitals  (claude-opus-5)
----------------------------------------------------------------
  PASS   australia                       1ms  $0.00005
  FAIL   brazil                          1ms  $0.00003
         exact_match: expected 'Brasilia', got 'I am not able to determine that.'
  PASS   canada                          1ms  $0.00005
  ...
----------------------------------------------------------------
  4/6 passed (67%)   $0.0003   p50 1ms   0.002s

  by tag:
    americas                 1/2
    europe                   2/2
    tricky                   2/3

Writing a suite

name: capitals
model: claude-opus-5
effort: low              # low | medium | high | xhigh | max
max_tokens: 256
system: Answer with the city name only.
scorers:
  - name: exact_match    # applied to every case unless a case overrides it
cases:
  - id: france
    prompt: What is the capital of France?
    expected: Paris
    tags: [europe]       # tags roll up into the per-tag breakdown

Scorers: exact_match, contains, not_contains, regex. Any case can override the suite's scorers with its own list -- see suites/extraction.yaml, which mixes a regex format check with a prompt-injection check.

Adding a scorer is a decorated function in scorers.py:

@scorer("starts_with")
def starts_with(output: str, case: Case, *, prefix: str) -> Score:
    passed = output.strip().startswith(prefix)
    return Score(scorer="starts_with", passed=passed, value=float(passed))

Comparing models

The same suite against a different model is one flag, and the report prints what the accuracy difference cost:

.venv/bin/llm-eval run suites/capitals.yaml --model claude-haiku-4-5
.venv/bin/llm-eval run suites/capitals.yaml --model claude-opus-5

Gating CI

--min-pass-rate makes the process exit non-zero below a threshold, which is all a CI job needs:

- run: llm-eval run suites/capitals.yaml --min-pass-rate 0.95

.github/workflows/ci.yml runs three jobs on that idea:

Job	Trigger	What it proves
test	every push and PR	unit tests pass on Python 3.11-3.13
smoke	every push and PR	the CLI runs every suite end to end, offline, no API key
eval	manual dispatch only	the live model still clears the pass-rate gate

The split matters. The offline smoke job proves the harness works and is safe to run on every commit and every fork's pull request, because it spends nothing and needs no secrets. The live eval job is the actual regression gate, and it costs real money per run -- so it is workflow_dispatch only, never automatic. Its run records upload as a build artifact.

To use the live gate, add an ANTHROPIC_API_KEY secret to the evals environment, then run the workflow from the Actions tab.

Every run also writes a full JSON record to runs/, including per-case output, scores, tokens, and cost.

Design notes

• Bounded concurrency. Cases fan out through a semaphore. Unbounded fan-out at a rate-limited API just converts into 429s and retries -- no faster, same cost.
• Refusals are a status, not a failure. A model declining a request is a distinct outcome from answering incorrectly, and collapsing the two hides a real signal. Refused cases are reported separately and are not scored.
• Errors are per-case. One case failing on a 500 does not abort the run.
• The provider is a Protocol. The runner is tested against a stub that records concurrency and raises SDK errors on demand -- no network, no spend.
• No retry loop of our own. The Anthropic SDK already retries connection errors, 429s, and 5xx with backoff; wrapping that in a second retry loop multiplies the wall-clock without improving the odds.

Roadmap

• LLM-as-judge scorer for open-ended outputs
• llm_eval_ci compare <run-a.json> <run-b.json> -- per-case regression diff
• pandas aggregation and a trend report across runs
• Local models via Ollama, for the cost/capability comparison