Skip to content

docs: classifier (Jev) evaluations can no longer be tested before deploy #834

Description

@NiveditJain

Two pages under docs/evaluations/ now describe the opposite of how
classifier (Jev) evaluations behave. Classifier evaluations can no longer be
tested before deploying — they behave like judge evaluations in this respect.

What is wrong

docs/evaluations/jev.mdx, under Testing and backfill:

Unlike a judge, a classifier evaluation can be tested before you deploy
it — test it against real sessions the same way you
would a code evaluation, and read the scores before anything goes live.

This is no longer true. Someone following it gets a refusal from the test
panel, having been told in the docs that it would work.

docs/evaluations/test.mdx does not mention judge or classifier
evaluations at all, so the refusal has no explanation anywhere in the docs.

Suggested wording

jev.mdx — replace the paragraph above with:

Like a judge, a classifier evaluation cannot be tested before you deploy
it. Both spend your organization's model budget, and that spend is only
authorized once the evaluation is deployed. Deploy it to a narrow scope
first and read the early scores instead.

The backfill paragraph that follows it is unaffected and should stay.

test.mdx — add after the intro paragraph:

Code evaluations only. Judge and classifier evaluations cannot be tested
here — both spend your organization's model budget, which is only authorized
once the evaluation is deployed. The panel tells you so if you try.

Worth checking while in there

  • The Testing and backfill heading on jev.mdx may read oddly once testing
    is gone from it; Backfill alone may be the better heading.
  • Anywhere else that contrasts classifier evaluations with judges on this
    point — the distinction no longer exists.
  • The judge page already states the rule correctly and can be the model for
    the phrasing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions