[bench][retention] Re-record the Sonnet curve - #26
Merged
Merged
Conversation
The committed Sonnet curve predated the marker change, so the two
curves in the tree were not a pair: one described a filter that no
longer exists. This is the same agent against the current tree — five
tasks, four variants, three runs each, 60 cells, all passed, no knee.
Level 2 is free and mostly does nothing. Four of five tasks land between
0.97 and 1.00 while reading a fraction of the bytes: 1,453 of 49,556 on
cargo-cascade, and the session costs the same. Four tool calls against
four, six against six. This agent does the same work either way, so what
the filter removes on the way in is not what the session is made of.
Level 0 is the worst view in the tree. 2.32x with nine tool calls against
four, and 2.96x with fifteen against six. A view reporting counts instead
of content sends the agent to do the reading itself, and it does two to
three times the work. It has now cost more than raw in every curve
recorded here, on both agents.
The win is mid-stream-trap at 0.79, where the answer is one line in the
middle of 4,800 and the filter hands it over.
Assembled from one sweep plus a re-run of the cells it did not attempt.
Every cell is one the agent tried and one binary produced all of them,
both checked before saving.
Test: bench/runner.py --run --driver claude — 60/60 attempted and
passed across three invocations, one binary fingerprint throughout
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The committed Sonnet curve predated the marker change, so the two curves in the
tree were not a pair — one described a filter that no longer exists. This is the
same agent against the current tree: five tasks, four variants, three runs each,
60 cells, all passed, no knee.
What it says
Level 2 is free and mostly does nothing. Four of five tasks land between
0.97 and 1.00 while reading a fraction of the bytes — 1,453 of 49,556 on
cargo-cascade— and the session costs the same. The tool calls say why: 4against 4, 6 against 6. This agent does the same work either way, so what the
filter removes on the way in is not what the session is made of.
Level 0 is the worst view in the tree, by a distance. 2.32x with 9 tool
calls against 4; 2.96x with 15 against 6. A view that reports counts instead of
content sends the agent to do the reading itself, and it does two to three times
the work. It has now cost more than raw in every curve recorded here, on both
agents.
The clear win is
mid-stream-trapat 0.79, where the answer is one line in themiddle of 4,800 and the filter hands it over directly.
Test plan
60/60 attempted and passed, across three invocations. Assembled from one sweep
plus a re-run of the cells it did not attempt — the same convention the previous
baseline used. Before saving, the merge asserts that every (task, variant) has
three cells and that one binary fingerprint produced all sixty; it refuses
otherwise.
Note for reviewers
The first sweep was blocked partway and left 39 cells unattempted. Those are not
in this file: the harness records them as not attempted rather than as
failures, keeps them out of the rate, and
--save-baselinerefuses a curve withholes. The re-run covers exactly the cells that were missing.
Both curves now describe the same filter, which they have not since the
marker change. They still disagree in useful ways — cursor showed 1.22x on
cargo-cascadelevel 2 where this shows 1.00 — and that disagreement is thereason both are committed.