[bench][retention] Exclude a cell the agent never finished - #28
Merged
Conversation
A cell whose agent exited non-zero was scored as a task failure. The
code already said it should not be — "silently scoring it as a task
failure would blame the filter for the harness" — and then set a subtype
nothing read. attempted() asked only whether tokens had been spent, and
a run cut off partway has spent some.
Measured: a sweep blocked partway recorded a cell that had spent 13,493
tokens and made two tool calls as a level 1 failure, and the curve came
back with a knee at level 1. There was no knee. Rescoring the same cells
with the guard wired puts it back to none.
A non-zero exit now marks the result, attempted() reads that mark, and
the cell is retried once and then recorded as one the sweep did not get.
Excluded from the rate, named in the report, and refused by
--save-baseline, which is what the other unattempted cells already do.
Test: the recorded cells from the blocked sweep, rescored — knee level1
with the cut-off cell counted, none with it excluded; a finished
cell still counts, one that never started still does not
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A cell whose agent exited non-zero was scored as a task failure. The code
already said it should not be —
— and then set a subtype that nothing read.
attempted()asked only whethertokens had been spent, and a run cut off partway has spent some.
What it cost
A Sonnet sweep was blocked partway. It recorded this:
and reported
knee: success falls below 100% at level1. There was no knee.The agent was cut off by the same block that left the other 51 cells
unattempted; it had simply spent tokens first.
Rescoring the same recorded cells with the guard wired:
level1NoneTest plan
Verified against the recorded run above, plus the three cases directly: a
finished cell still counts, one that never started still does not, and one cut
off mid-run is now excluded and named in the report.
Note for reviewers
This is the second guard in this file written and left unwired —
cell_note()was the first. The pattern is worth watching for: the comment states the
intention, the code sets a field, and nothing consumes it. A benchmark's guards
are exactly the code that never runs on a good day.
The consequence was not cosmetic.
--save-baselinerefuses a curve withunattempted cells, but this cell counted as attempted, so a blocked sweep could
have been saved as a baseline asserting a knee that does not exist.
No curve is committed here. The sweep that exposed this got 9 of 60 cells, which
is not a curve, and the Sonnet baseline still predates the level 0 change.