Skip to content

[bench][retention] Re-record the Sonnet curve - #26

Merged
bherrera merged 1 commit into
mainfrom
sonnet-curve
Aug 26, 2026
Merged

[bench][retention] Re-record the Sonnet curve#26
bherrera merged 1 commit into
mainfrom
sonnet-curve

Conversation

@bherrera

Copy link
Copy Markdown
Contributor

Summary

The committed Sonnet curve predated the marker change, so the two curves in the
tree were not a pair — one described a filter that no longer exists. This is the
same agent against the current tree: five tasks, four variants, three runs each,
60 cells, all passed, no knee.

task raw level 2 level 1 level 0
cargo-cascade 1.00 1.00 1.00 2.32
cascading-errors 1.00 0.99 1.17 2.96
fix-failing-test 1.00 0.97 0.78 1.32
last-line-trap 1.00 0.99 1.00 1.44
mid-stream-trap 1.00 0.79 1.01 1.92

What it says

Level 2 is free and mostly does nothing. Four of five tasks land between
0.97 and 1.00 while reading a fraction of the bytes — 1,453 of 49,556 on
cargo-cascade — and the session costs the same. The tool calls say why: 4
against 4, 6 against 6. This agent does the same work either way, so what the
filter removes on the way in is not what the session is made of.

Level 0 is the worst view in the tree, by a distance. 2.32x with 9 tool
calls against 4; 2.96x with 15 against 6. A view that reports counts instead of
content sends the agent to do the reading itself, and it does two to three times
the work. It has now cost more than raw in every curve recorded here, on both
agents.

The clear win is mid-stream-trap at 0.79, where the answer is one line in the
middle of 4,800 and the filter hands it over directly.

Test plan

bench/runner.py --run --driver claude

60/60 attempted and passed, across three invocations. Assembled from one sweep
plus a re-run of the cells it did not attempt — the same convention the previous
baseline used. Before saving, the merge asserts that every (task, variant) has
three cells and that one binary fingerprint produced all sixty; it refuses
otherwise.

Note for reviewers

The first sweep was blocked partway and left 39 cells unattempted. Those are not
in this file: the harness records them as not attempted rather than as
failures, keeps them out of the rate, and --save-baseline refuses a curve with
holes. The re-run covers exactly the cells that were missing.

Both curves now describe the same filter, which they have not since the
marker change. They still disagree in useful ways — cursor showed 1.22x on
cargo-cascade level 2 where this shows 1.00 — and that disagreement is the
reason both are committed.

The committed Sonnet curve predated the marker change, so the two
curves in the tree were not a pair: one described a filter that no
longer exists. This is the same agent against the current tree — five
tasks, four variants, three runs each, 60 cells, all passed, no knee.

Level 2 is free and mostly does nothing. Four of five tasks land between
0.97 and 1.00 while reading a fraction of the bytes: 1,453 of 49,556 on
cargo-cascade, and the session costs the same. Four tool calls against
four, six against six. This agent does the same work either way, so what
the filter removes on the way in is not what the session is made of.

Level 0 is the worst view in the tree. 2.32x with nine tool calls against
four, and 2.96x with fifteen against six. A view reporting counts instead
of content sends the agent to do the reading itself, and it does two to
three times the work. It has now cost more than raw in every curve
recorded here, on both agents.

The win is mid-stream-trap at 0.79, where the answer is one line in the
middle of 4,800 and the filter hands it over.

Assembled from one sweep plus a re-run of the cells it did not attempt.
Every cell is one the agent tried and one binary produced all of them,
both checked before saving.

Test: bench/runner.py --run --driver claude — 60/60 attempted and
      passed across three invocations, one binary fingerprint throughout

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bherrera
bherrera merged commit 708c948 into main Aug 26, 2026
2 checks passed
@bherrera
bherrera deleted the sonnet-curve branch August 26, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant