[rishsriv]

This PR adds a behaviour test suite for the plugin: eight cases that run a real model against the skill with the FactIQ data server mocked, and a runner that scores every case twice, once with the full plugin and once with the same mocked tools but no skill and no scripts. The difference between the two scores is what the skill and the bundled scripts add. In the run reported below, the plugin arm passes every case at the chosen bar, and the suite has already found one defect in the plugin's own output: the chart spec it writes omits the required id key in two of three runs.

The problem

The unit tests in tests/ check the wording of the documentation and the output of the bundled scripts. Nothing checked what Claude does with the skill: whether it computes year-over-year by calendar date across a missing month, whether it adds a filing line to its own duplicate, whether it writes a chart spec the way the reference describes, or whether it stays out of the way on a question that has nothing to do with data.

What the suite is

Each case is a directory under evals/ with a prompt.md (the user's message and the run settings) and a graders/ directory (one check per file). Checks are either exact pattern matches, tool-call counts, a file-exists check, or a judgment by a second model that votes three times. The FactIQ server is never called: the files in evals/mocks/factiq/ answer each tool with real payloads captured from the server. Two of the mocks answer according to their arguments, so a request for series LNS14000000 gets the unemployment fixture and any other series id gets an error, and a filings search for PKG gets the Packaging Corporation of America fixture.

The eight cases:

Case	What it checks
latest-value-direct-answer	A one-number question gets a short prose answer with the period and the source, and no chart and no report file.
quick-chart-of-a-trend	A single-trend question writes one valid ChartSpec whose title states a finding, renders it with the bundled terminal renderer, pastes the preview, leaves the missing month empty, and says so.
yoy-across-a-missing-month	Year-over-year is computed by calendar date, and the missing month is stated, not filled in. The server's own note about the gap is present.
yoy-without-a-server-hint	The same request with the server note removed from the data, so it measures the skill's rule alone.
bilateral-trade-sql	A bilateral-trade query is built with the bundled generator, filters the partner by numeric code, pins one HS level, and keeps quantity series out of the value total.
filings-duplicate-line	A figure that appears twice in one filing is reported once, not doubled, with the filing's own link.
quote-and-source-discipline	Quotes are verbatim, each carries the exact supplied source link, a missing link is declared, and an analyst's words are not reported as the company's.
unrelated-coding-request	A plain Python question is answered directly, with no FactIQ tool call and no skill invocation.

Why the comparison uses a tools-only baseline

The built-in with-and-without comparison in claude plugin eval removes the whole plugin from the second arm, including its MCP server. That arm has no data tools at all, so every data case fails there for a reason that has nothing to do with the skill. The runner in evals/run_ablation.py instead builds a temporary copy of the plugin that keeps only the manifest and the MCP configuration, and runs both copies with the same mocks, the same model, the same judge, and the same tool grants. What the baseline gets wrong and the plugin gets right is the skill's contribution.

Evidence

One run of the whole suite, agent model Opus, judge model Sonnet, Claude Code 2.1.269, three runs per case, started 2026-09-12 UTC. No run errored or aborted.

Case	With plugin	Baseline	Delta	Runs	Aborted or errored
bilateral-trade-sql	1.00	0.17	0.83	3	0 / 0
filings-duplicate-line	1.00	1.00	0.00	3	0 / 0
latest-value-direct-answer	0.80	0.40	0.40	3	0 / 0
quick-chart-of-a-trend	0.83	0.47	0.36	3	0 / 0
quote-and-source-discipline	0.97	0.75	0.22	3	0 / 0
unrelated-coding-request	1.00	1.00	0.00	3	0 / 0
yoy-across-a-missing-month	1.00	1.00	0.00	3	0 / 0
yoy-without-a-server-hint	1.00	1.00	0.00	3	0 / 0

Cost: 14.93 US dollars for the plugin arm and 9.49 for the baseline arm.

Three cases score the same in both arms: the two year-over-year cases, the filings case, and the coding case. Opus gets those right from the tool output alone, so today they guard against regressions rather than measure the skill. The trade query, the chart, the short answer, and the source links are where the skill changes the outcome.

The three misses in the plugin arm, in detail:

• quick-chart-of-a-trend, two runs of three. The written chart.json had title, type, xField, series and data in the documented wide-row form, a title that stated the change, the October 2025 gap left empty, and a BLS source, but no id key. The chart-spec reference lists id as required. That is a defect in the plugin's output, and a follow-up should either make the skill emit it or drop it from the required list.
• latest-value-direct-answer, one run of three. The answer was two sentences: 4.1 percent in August 2026, per the BLS Current Population Survey, down from 4.5 percent in November 2025. Two of three judge votes failed it on the "answers in prose" check. I read this as judge disagreement, not a defect, and left the check as it is.
• quote-and-source-discipline, one run of three. The answer set the analyst's "71% to 72%" remark apart from the management quotes, as the case requires, but quoted a fragment of it in a caveat without the supplied link. The link check is weight 1 of 12 for that case.

Before this change, the cases as first written could not have produced this table. The "must not call a FactIQ tool" checks named the tools mcp__factiq__*, but inside the sandbox they are mcp__plugin_factiq_factiq__*, so those checks passed no matter what the model did. The trade-query check accepted any command that mentioned the generator's file name, including reading it. The "without plugin" arm had no tools, so its scores said nothing about the skill. All three are fixed here.

How it runs

• The unit tests in tests/ run on every push to main and on every pull request (.github/workflows/tests.yml).
• The behaviour suite costs money and takes about half an hour, so it does not run on push. .github/workflows/behaviour-evals.yml runs it by hand from the Actions tab, with the model, the number of runs per case, and the pass bar chosen there. The job needs an ANTHROPIC_API_KEY repository secret, which has not been set up yet, so the workflow is not yet verified in CI. Locally the same command is documented in evals/README.md.
• The pass bar defaults to 0.8 per case rather than 1.0, because one dissenting judge vote in a three-run case already lands near 0.9.

Details

• evals/run_ablation.py: builds the tools-only baseline, runs both arms, writes evals/results/<timestamp>/comparison.md with every case and every check in both arms, and exits 1 when a plugin-arm case scores below the bar.
• evals/mocks/factiq/_tools.json: the real tool list from the server, 14 tools with their real descriptions and input schemas, so the model sees the same tool surface it would see in use.
• evals/mocks/factiq/get_series.md and search_company_filings.md: answer from fixtures/<series_id>.json and fixtures/<company>.json. The transform argument is ignored on purpose: a request for a year-over-year transform receives plain values, which is what the year-over-year cases need.
• Fixtures: the US unemployment rate (BLS LNS14000000, 31 monthly values, no October 2025 observation) and Packaging Corporation of America segment revenue for the quarter ended June 2026 (eight facts, the segment total present twice, with the server's warning note stripped so the case measures the skill's rule). The NVIDIA gross-margin transcript rows from the first version are kept, with one source link blanked on purpose.
• evals/README.md explains the sandbox: tool names, that the plugin root variable is empty inside a run, and the rules for writing a check that can actually fail.

🤖 Generated with Claude Code