Skip to content

Commit 8ba2519

Browse files
committed
docs: remove batch cli eval pattern
1 parent 458d34d commit 8ba2519

19 files changed

Lines changed: 21 additions & 1483 deletions

File tree

apps/web/src/content/docs/docs/next/evaluation/batch-cli.mdx

Lines changed: 0 additions & 277 deletions
This file was deleted.

apps/web/src/content/docs/docs/next/evaluation/examples.mdx

Lines changed: 0 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -302,92 +302,6 @@ tests:
302302
value: "Matches the reference answer: {{ expected_output }}"
303303
````
304304

305-
## Batch CLI
306-
307-
Evaluate external batch runners that process all tests in one invocation:
308-
309-
```yaml
310-
description: Batch CLI demo (AML screening)
311-
providers:
312-
- batch_cli
313-
314-
prompts:
315-
- "{{ input }}"
316-
317-
tests:
318-
- id: aml-001
319-
criteria: |-
320-
Batch runner returns JSON with decision=CLEAR.
321-
vars:
322-
input:
323-
- role: system
324-
content: You are a deterministic AML screening batch checker.
325-
- role: user
326-
content:
327-
request:
328-
type: aml_screening_check
329-
jurisdiction: AU
330-
effective_date: 2025-01-01
331-
row:
332-
id: aml-001
333-
customer_name: Example Customer A
334-
origin_country: NZ
335-
destination_country: AU
336-
transaction_type: INTERNATIONAL_TRANSFER
337-
amount: 5000
338-
currency: USD
339-
expected_output:
340-
- role: assistant
341-
content:
342-
decision: CLEAR
343-
344-
assert:
345-
- name: decision-check
346-
type: script
347-
command: [bun, run, ./scripts/check-batch-cli-output.ts]
348-
cwd: .
349-
350-
- id: aml-002
351-
criteria: |-
352-
Batch runner returns JSON with decision=REVIEW.
353-
vars:
354-
input:
355-
- role: system
356-
content: You are a deterministic AML screening batch checker.
357-
- role: user
358-
content:
359-
request:
360-
type: aml_screening_check
361-
jurisdiction: AU
362-
effective_date: 2025-01-01
363-
row:
364-
id: aml-002
365-
customer_name: Example Customer B
366-
origin_country: IR
367-
destination_country: AU
368-
transaction_type: INTERNATIONAL_TRANSFER
369-
amount: 2000
370-
currency: USD
371-
expected_output:
372-
- role: assistant
373-
content:
374-
decision: REVIEW
375-
376-
assert:
377-
- name: decision-check
378-
type: script
379-
command: [bun, run, ./scripts/check-batch-cli-output.ts]
380-
cwd: .
381-
```
382-
383-
### Batch CLI Pattern Notes
384-
385-
- `target: batch_cli` -- configure the CLI provider with `batch_requests: true`
386-
- The batch runner reads the eval YAML via `--eval` flag and outputs JSONL keyed by `id`
387-
- Put structured data in `user.content` as objects for the runner to extract
388-
- Use `vars.expected_output` with object fields for structured expected output
389-
- Each test has its own grader to validate its portion of the output
390-
391305
## Shared Prompt Context
392306

393307
Share a common prompt or system instruction across all tests with top-level

0 commit comments

Comments
 (0)