CodePrism benchmark reports are local, reproducible estimates. They are useful for comparing source, context-pack, graph, and focused-slice sizes, but they are not billing-grade token measurements and should not be read as guarantees for every repository.
The public benchmark story has two parts:
- Fixture suite: checked-in, reproducible, suitable for CI and release comparisons.
- Field notes: local alpha runs on larger real repositories, useful for product intuition but not treated as release guarantees.
Run the checked-in suite:
codeprism benchmark-suite examples/benchmarks --out .codeprism/benchmarks/suite.json
python scripts/render_benchmark_chart.py .codeprism/benchmarks/suite.json --out docs/assets/benchmark-snapshot.svgThe benchmark command writes per-fixture JSON reports, a combined suite.json, and a Markdown summary at .codeprism/benchmarks/suite.md. The chart renderer turns the suite JSON into docs/assets/benchmark-snapshot.svg for README and docs display.
To verify the checked-in chart is current without rewriting it:
python scripts/render_benchmark_chart.py .codeprism/benchmarks/suite.json --out docs/assets/benchmark-snapshot.svg --checkCompare two suite runs:
codeprism benchmark-compare previous-suite.json .codeprism/benchmarks/suite.json --out .codeprism/benchmarks/comparison.mdAdd --fail-on-regression --regression-threshold 5 when you want CI or a release script to fail if any matched fixture loses more than five percentage points of estimated source-to-slice savings.
For release review, use the maintainer helper:
python scripts/benchmark_trend.py --baseline-suite previous-suite.jsonThat local mode never calls GitHub. It runs the current suite, compares it to the supplied baseline, and writes .codeprism/benchmark-trends/comparison.md.
To compare against the latest successful public workflow artifact, install and authenticate the GitHub CLI, then omit --baseline-suite:
python scripts/benchmark_trend.py --repo kunolabs/codeprism --python-version 3.12The artifact mode downloads codeprism-benchmarks-py3.12, finds its suite.json, runs the current local suite, and writes a Markdown trend report. This is an explicit maintainer workflow; normal CodePrism commands do not make network calls.
To collect a fuller local pre-release proof packet:
python scripts/pre_release_proof.py --baseline-suite previous-suite.jsonThis writes .codeprism/pre-release/README.md, a machine-readable .codeprism/pre-release/manifest.json, benchmark trend output, sample session audit output, test and lint logs, and a public hygiene scan. If no baseline suite is supplied, the benchmark trend compares the current suite against itself so the command remains fully offline by default.
The current suite covers 8 fixtures across Python, TypeScript, Java, and Kotlin. Across all fixtures it maps 40 files, compares 15,291 estimated source tokens against 4,801 estimated focused-slice tokens, and reports a 68.75% average per-fixture source-to-slice reduction.
| Fixture | Files | Source tokens | Slice tokens | Source -> slice | Source -> context pack |
|---|---|---|---|---|---|
| Basic Java | 5 | 1,126 | 524 | 53.46% | 0.00% |
| Basic Kotlin | 4 | 1,139 | 442 | 61.19% | 13.26% |
| Basic Python | 5 | 1,263 | 172 | 86.38% | 70.39% |
| Basic TypeScript | 5 | 1,073 | 264 | 75.40% | 21.06% |
| Medium Java Server | 6 | 2,538 | 956 | 62.33% | 14.81% |
| Medium Kotlin Android | 5 | 2,804 | 1,171 | 58.24% | 48.25% |
| Medium Python Service | 5 | 2,897 | 783 | 72.97% | 59.54% |
| Medium TypeScript App | 5 | 2,451 | 489 | 80.05% | 64.18% |
Summary:
- Fixtures: 8
- Average source-to-slice saving: 68.75%
- Minimum source-to-slice saving: 53.46%
- Maximum source-to-slice saving: 86.38%
The checked-in fixtures are intentionally small so they can run quickly in CI. On tiny projects, graph metadata and slice headers can be a meaningful part of the output, so a context pack or slice can show little or no reduction even when the workflow is behaving correctly.
The real product loop is larger than this table: run codeprism prime "<task>", read the compact brief, then use query, get, references, and read --mode signatures/diff before opening raw files. Large repositories and narrow tasks usually show stronger reductions because CodePrism can avoid generated files, repeated shell output, dependency folders, and unrelated source.
When README or release notes mention larger real-world reductions, keep those examples clearly labeled as field notes. Do not mix them into the reproducible fixture average unless the repository, fixture data, and commands are checked in.
For public repo comparison work, use the maintainer runner instead of hand-copying command output:
python scripts/run_field_notes.py --config examples/field-notes/public-repos.json --repos-root externalThe runner expects local checkouts under external/ and does not clone or call the network. Missing repos are reported with clone hints by default; add --fail-on-missing only when every checkout is intentionally prepared. Results are written under .codeprism/field-notes/ as per-target result.json files plus summary.json and summary.md.
Field notes should stay separate from the checked-in fixture suite. They can support product intuition, roadmap choices, and dated public examples, but they should not replace the reproducible fixture average.
Run: 2026-05-06T02:54:46Z
Command:
python scripts/run_field_notes.py --config examples/field-notes/public-repos.json --repos-root external --fail-on-missingOutcome:
- 7 passed
- 0 missing
- 0 failed
- All runs capped to default
--max-tokens 8000unless source repo language mix forced earlier truncation.
| Target | Source tokens | Slice tokens | Saving |
|---|---|---|---|
| Graphify | 526,883 | 7,991 | 98.48% |
| Serena | 1,525,349 | 8,000 | 99.48% |
| GSD | 2,846,370 | 7,995 | 99.72% |
| jCodeMunch MCP | 1,477,903 | 8,000 | 99.46% |
| LeanCTX | 1,291,656 | 7,991 | 99.38% |
| CocoIndex Code | 98,806 | 7,999 | 91.90% |
| codesight | 237,797 | 7,995 | 96.64% |
Average source-to-slice saving across this snapshot: 96.81%.
The basic-* fixtures are tiny language smoke tests. They keep Python, TypeScript, Java, and Kotlin extraction behavior visible.
The medium-* fixtures add a small amount of generated or noisy code around a focused task. They are still CI-sized, but they are closer to the cases CodePrism is meant to handle: identify the relevant slice and avoid dragging every nearby file into the agent prompt.
The GitHub Actions smoke job uploads benchmark JSON and Markdown reports for each Python version as codeprism-benchmarks-py<version>. Download two artifacts and run codeprism benchmark-compare, or use scripts/benchmark_trend.py, when reviewing a performance-sensitive change or preparing release notes.