Two pages under docs/evaluations/ now describe the opposite of how
classifier (Jev) evaluations behave. Classifier evaluations can no longer be
tested before deploying — they behave like judge evaluations in this respect.
What is wrong
docs/evaluations/jev.mdx, under Testing and backfill:
Unlike a judge, a classifier evaluation can be tested before you deploy
it — test it against real sessions the same way you
would a code evaluation, and read the scores before anything goes live.
This is no longer true. Someone following it gets a refusal from the test
panel, having been told in the docs that it would work.
docs/evaluations/test.mdx does not mention judge or classifier
evaluations at all, so the refusal has no explanation anywhere in the docs.
Suggested wording
jev.mdx — replace the paragraph above with:
Like a judge, a classifier evaluation cannot be tested before you deploy
it. Both spend your organization's model budget, and that spend is only
authorized once the evaluation is deployed. Deploy it to a narrow scope
first and read the early scores instead.
The backfill paragraph that follows it is unaffected and should stay.
test.mdx — add after the intro paragraph:
Code evaluations only. Judge and classifier evaluations cannot be tested
here — both spend your organization's model budget, which is only authorized
once the evaluation is deployed. The panel tells you so if you try.
Worth checking while in there
- The Testing and backfill heading on
jev.mdx may read oddly once testing
is gone from it; Backfill alone may be the better heading.
- Anywhere else that contrasts classifier evaluations with judges on this
point — the distinction no longer exists.
- The judge page already states the rule correctly and can be the model for
the phrasing.
Two pages under
docs/evaluations/now describe the opposite of howclassifier (Jev) evaluations behave. Classifier evaluations can no longer be
tested before deploying — they behave like judge evaluations in this respect.
What is wrong
docs/evaluations/jev.mdx, under Testing and backfill:This is no longer true. Someone following it gets a refusal from the test
panel, having been told in the docs that it would work.
docs/evaluations/test.mdxdoes not mention judge or classifierevaluations at all, so the refusal has no explanation anywhere in the docs.
Suggested wording
jev.mdx— replace the paragraph above with:The backfill paragraph that follows it is unaffected and should stay.
test.mdx— add after the intro paragraph:Worth checking while in there
jev.mdxmay read oddly once testingis gone from it; Backfill alone may be the better heading.
point — the distinction no longer exists.
the phrasing.