@@ -302,92 +302,6 @@ tests:
302302 value : " Matches the reference answer: {{ expected_output }}"
303303` ` ` `
304304
305- # # Batch CLI
306-
307- Evaluate external batch runners that process all tests in one invocation :
308-
309- ` ` ` yaml
310- description: Batch CLI demo (AML screening)
311- providers:
312- - batch_cli
313-
314- prompts:
315- - "{{ input }}"
316-
317- tests:
318- - id: aml-001
319- criteria: |-
320- Batch runner returns JSON with decision=CLEAR.
321- vars:
322- input:
323- - role: system
324- content: You are a deterministic AML screening batch checker.
325- - role: user
326- content:
327- request:
328- type: aml_screening_check
329- jurisdiction: AU
330- effective_date: 2025-01-01
331- row:
332- id: aml-001
333- customer_name: Example Customer A
334- origin_country: NZ
335- destination_country: AU
336- transaction_type: INTERNATIONAL_TRANSFER
337- amount: 5000
338- currency: USD
339- expected_output:
340- - role: assistant
341- content:
342- decision: CLEAR
343-
344- assert:
345- - name: decision-check
346- type: script
347- command: [bun, run, ./scripts/check-batch-cli-output.ts]
348- cwd: .
349-
350- - id: aml-002
351- criteria: |-
352- Batch runner returns JSON with decision=REVIEW.
353- vars:
354- input:
355- - role: system
356- content: You are a deterministic AML screening batch checker.
357- - role: user
358- content:
359- request:
360- type: aml_screening_check
361- jurisdiction: AU
362- effective_date: 2025-01-01
363- row:
364- id: aml-002
365- customer_name: Example Customer B
366- origin_country: IR
367- destination_country: AU
368- transaction_type: INTERNATIONAL_TRANSFER
369- amount: 2000
370- currency: USD
371- expected_output:
372- - role: assistant
373- content:
374- decision: REVIEW
375-
376- assert:
377- - name: decision-check
378- type: script
379- command: [bun, run, ./scripts/check-batch-cli-output.ts]
380- cwd: .
381- ` ` `
382-
383- # ## Batch CLI Pattern Notes
384-
385- - `target : batch_cli` -- configure the CLI provider with `batch_requests: true`
386- - The batch runner reads the eval YAML via `--eval` flag and outputs JSONL keyed by `id`
387- - Put structured data in `user.content` as objects for the runner to extract
388- - Use `vars.expected_output` with object fields for structured expected output
389- - Each test has its own grader to validate its portion of the output
390-
391305# # Shared Prompt Context
392306
393307Share a common prompt or system instruction across all tests with top-level
0 commit comments