Skip to content

[bench][retention] Exclude a cell the agent never finished - #28

Merged
bherrera merged 1 commit into
mainfrom
harness-cutoff-guard
Aug 27, 2026
Merged

[bench][retention] Exclude a cell the agent never finished#28
bherrera merged 1 commit into
mainfrom
harness-cutoff-guard

Conversation

@bherrera

Copy link
Copy Markdown
Contributor

Summary

A cell whose agent exited non-zero was scored as a task failure. The code
already said it should not be —

# A cell that failed for a reason outside the thing being measured has
# to say so. Silently scoring it as a task failure would blame the
# filter for the harness.

— and then set a subtype that nothing read. attempted() asked only whether
tokens had been spent, and a run cut off partway has spent some.

What it cost

A Sonnet sweep was blocked partway. It recorded this:

cargo-cascade level1 #3   13,493 tokens   2 tools   pass=False
   note: agent exit 1:

and reported knee: success falls below 100% at level1. There was no knee.
The agent was cut off by the same block that left the other 51 cells
unattempted; it had simply spent tokens first.

Rescoring the same recorded cells with the guard wired:

knee
cut-off cell counted level1
cut-off cell excluded None

Test plan

ruff check bench/runner.py

Verified against the recorded run above, plus the three cases directly: a
finished cell still counts, one that never started still does not, and one cut
off mid-run is now excluded and named in the report.

Note for reviewers

This is the second guard in this file written and left unwired — cell_note()
was the first. The pattern is worth watching for: the comment states the
intention, the code sets a field, and nothing consumes it. A benchmark's guards
are exactly the code that never runs on a good day.

The consequence was not cosmetic. --save-baseline refuses a curve with
unattempted cells, but this cell counted as attempted, so a blocked sweep could
have been saved as a baseline asserting a knee that does not exist.

No curve is committed here. The sweep that exposed this got 9 of 60 cells, which
is not a curve, and the Sonnet baseline still predates the level 0 change.

A cell whose agent exited non-zero was scored as a task failure. The
code already said it should not be — "silently scoring it as a task
failure would blame the filter for the harness" — and then set a subtype
nothing read. attempted() asked only whether tokens had been spent, and
a run cut off partway has spent some.

Measured: a sweep blocked partway recorded a cell that had spent 13,493
tokens and made two tool calls as a level 1 failure, and the curve came
back with a knee at level 1. There was no knee. Rescoring the same cells
with the guard wired puts it back to none.

A non-zero exit now marks the result, attempted() reads that mark, and
the cell is retried once and then recorded as one the sweep did not get.
Excluded from the rate, named in the report, and refused by
--save-baseline, which is what the other unattempted cells already do.

Test: the recorded cells from the blocked sweep, rescored — knee level1
      with the cut-off cell counted, none with it excluded; a finished
      cell still counts, one that never started still does not

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bherrera
bherrera merged commit 7e53404 into main Aug 27, 2026
2 checks passed
@bherrera
bherrera deleted the harness-cutoff-guard branch August 27, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant