|
| 1 | +# gooddata-eval |
| 2 | + |
| 3 | +CLI to evaluate the GoodData AI agent against a dataset of natural-language |
| 4 | +questions on a chosen workspace and LLM model. |
| 5 | + |
| 6 | +## Install |
| 7 | + |
| 8 | + uv add gooddata-eval |
| 9 | + |
| 10 | +Or install `gd-eval` as a standalone tool: |
| 11 | + |
| 12 | + uv tool install gooddata-eval |
| 13 | + |
| 14 | +## Quick start |
| 15 | + |
| 16 | +```bash |
| 17 | +export GOODDATA_TOKEN='your-api-token' |
| 18 | + |
| 19 | +gd-eval run \ |
| 20 | + --host https://your.gooddata.cloud \ |
| 21 | + --workspace demo \ |
| 22 | + --dataset ./my-dataset \ |
| 23 | + --model gpt-5.2 \ |
| 24 | + --runs 2 \ |
| 25 | + --json results.json |
| 26 | +``` |
| 27 | + |
| 28 | +## All flags |
| 29 | + |
| 30 | +### Connection |
| 31 | + |
| 32 | +| Flag | Env var | Description | |
| 33 | +|---|---|---| |
| 34 | +| `--host HOST` | — | GoodData host URL (e.g. `https://your.gooddata.cloud`). | |
| 35 | +| `--token TOKEN` | `GOODDATA_TOKEN` | API token. Pass via flag or env var. | |
| 36 | +| `--profile NAME` | — | Profile name in `~/.gooddata/profiles.yaml` (same file as the `gdc` CLI). Provides host + token when both flags are omitted. | |
| 37 | +| `--workspace ID` | — | **Required.** Workspace id to evaluate against. | |
| 38 | + |
| 39 | +### Dataset source (pick one) |
| 40 | + |
| 41 | +| Flag | Description | |
| 42 | +|---|---| |
| 43 | +| `--dataset PATH` | Path to a flat folder of JSON files — one question per file. | |
| 44 | +| `--langfuse-dataset NAME` | Pull dataset items by name from Langfuse. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST` env vars. | |
| 45 | + |
| 46 | +### Model selection |
| 47 | + |
| 48 | +| Flag | Description | |
| 49 | +|---|---| |
| 50 | +| `--model ID` | LLM model id to evaluate (e.g. `gpt-5.2`). Defaults to the workspace's currently active model. If the model is offered by a different provider than the active one, the workspace's active provider is switched automatically. | |
| 51 | +| `--provider NAME_OR_ID` | LLM provider name or id. Use when `--model` is offered by multiple providers and you need to pick one. Accepts either the human-readable provider name or its UUID id. | |
| 52 | + |
| 53 | +### Evaluation |
| 54 | + |
| 55 | +| Flag | Default | Description | |
| 56 | +|---|---|---| |
| 57 | +| `--runs K` | `2` | Number of independent conversation runs per item (pass@K). An item passes if any run passes. | |
| 58 | + |
| 59 | +### Output |
| 60 | + |
| 61 | +| Flag | Description | |
| 62 | +|---|---| |
| 63 | +| `--json PATH` | Write a machine-readable JSON report (keyed by item id, with per-item scores) to this path. Console summary is always printed. | |
| 64 | +| `--quiet` | Suppress per-item progress output. Only the final table and summary are printed. | |
| 65 | + |
| 66 | +### Langfuse sink |
| 67 | + |
| 68 | +| Flag | Description | |
| 69 | +|---|---| |
| 70 | +| `--langfuse` | Log evaluation results to Langfuse after each item. Requires `--langfuse-dataset` (so item ids can be linked to Langfuse dataset items). Creates a named experiment run (`gd-eval-{timestamp}-{model}`) in the Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. | |
| 71 | + |
| 72 | +## Dataset format |
| 73 | + |
| 74 | +A dataset is a folder of `.json` files, one per question. Each file must |
| 75 | +contain a common envelope: |
| 76 | + |
| 77 | +```json |
| 78 | +{ |
| 79 | + "id": "stable-unique-id", |
| 80 | + "dataset_name": "my_dataset", |
| 81 | + "test_kind": "visualization", |
| 82 | + "question": "Show revenue by quarter", |
| 83 | + "expected_output": { } |
| 84 | +} |
| 85 | +``` |
| 86 | + |
| 87 | +Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`, |
| 88 | +`search_tool`, `general_question`, `guardrail`. |
| 89 | + |
| 90 | +See the full dataset specification for `expected_output` shapes per test kind. |
| 91 | + |
| 92 | +## Supported test kinds |
| 93 | + |
| 94 | +| test_kind | What the agent must produce | Extra required | |
| 95 | +|---|---|---| |
| 96 | +| `visualization` | Correct AAC visualization (metrics, dimensions, filters, type) | — | |
| 97 | +| `metric_skill` | `create_metric` tool call with correct MAQL and format | — | |
| 98 | +| `alert_skill` | `create_metric_alert` tool call with correct operator, threshold, trigger, filters, metric, recipients | — | |
| 99 | +| `search_tool` | `search_objects` tool call (correct function called = pass; correct arguments = quality score) | — | |
| 100 | +| `general_question` | Text answer judged by LLM | `[llm-judge]` | |
| 101 | +| `guardrail` | Refusal/redirect (visualization response auto-fails) | `[llm-judge]` | |
| 102 | + |
| 103 | +## Optional extras |
| 104 | + |
| 105 | +### `[llm-judge]` — LLM-as-judge evaluators |
| 106 | + |
| 107 | +`general_question` and `guardrail` items are scored by an LLM judge (GPT-4o) |
| 108 | +that compares the agent's text response against your expected-output description. |
| 109 | +This requires the OpenAI Python package and an API key: |
| 110 | + |
| 111 | +```bash |
| 112 | +uv add 'gooddata-eval[llm-judge]' # project dependency |
| 113 | +# or, for the standalone gd-eval tool: |
| 114 | +uv tool install 'gooddata-eval[llm-judge]' |
| 115 | +``` |
| 116 | + |
| 117 | +Set your OpenAI key before running: |
| 118 | + |
| 119 | +```bash |
| 120 | +export OPENAI_API_KEY='sk-...' |
| 121 | +``` |
| 122 | + |
| 123 | +Without `[llm-judge]`, items with `test_kind: general_question` or `guardrail` |
| 124 | +are reported as **skipped**. |
| 125 | + |
| 126 | + |
| 127 | +## Exit codes |
| 128 | + |
| 129 | +| Code | Meaning | |
| 130 | +|---|---| |
| 131 | +| `0` | Run completed. Evaluation failures do **not** cause a non-zero exit — check the report. | |
| 132 | +| `2` | Operational error: bad connection, missing model, unreadable dataset, missing credentials. | |
| 133 | + |
| 134 | +## Scores (in JSON report and Langfuse) |
| 135 | + |
| 136 | +| Score | Description | |
| 137 | +|---|---| |
| 138 | +| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. | |
| 139 | +| `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. | |
| 140 | +| `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (where speed = max(0, 1 − latency/60s)). | |
| 141 | +| `latency_s` | Average per-run latency in seconds. | |
0 commit comments