Skip to content

feat(compare): add offline result comparison - #183

Open
googs1025 wants to merge 1 commit into
alibaba:mainfrom
googs1025:feat/add-offline-result-compare
Open

feat(compare): add offline result comparison#183
googs1025 wants to merge 1 commit into
alibaba:mainfrom
googs1025:feat/add-offline-result-compare

Conversation

@googs1025

Copy link
Copy Markdown
Contributor

Summary

  • Add offline skill-up compare <old-result.json> <new-result.json> for text and JSON result diffs.
  • Compare primary case metrics, metadata, case transitions, and CI gates without rerunning evals.
  • Document the new command and update changelog.

Test Plan

  • GOCACHE=/private/tmp/skill-up-go-build-cache GOTOOLCHAIN=go1.26.4 make verify
  • GOCACHE=/private/tmp/skill-up-go-build-cache GOTOOLCHAIN=go1.26.4 make test

Closes #175

@googs1025
googs1025 requested a review from hittyt as a code owner August 12, 2026 08:05

zpzjzj commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the proposal and implementation. I understand that this command provides domain-aware comparison, such as primary-case handling, case transition classification, and CI exit codes.

However, if the scope is only comparing two local result.json files, the value of adding a dedicated built-in command is not yet very clear to me. The current pass-rate, status, token, and duration comparison can already be implemented fairly cheaply with jq or a small script, while this PR introduces a relatively large amount of code and a new CLI contract that we will need to maintain.

I think the feature would be more compelling if it formed part of a complete baseline workflow—for example:

  • locating or selecting the baseline result automatically;
  • comparing against the latest release or a named baseline;
  • handling result schema compatibility across versions;
  • providing richer regression evidence beyond case status;
  • offering a reusable CI integration that individual repositories would otherwise need to maintain themselves.

Could you clarify the concrete user workflow this is intended to unlock, and what advantage the built-in command provides over a documented jq/script example? If the intention is only raw offline file comparison, I’m not yet convinced that the maintenance cost is justified.

@JHWang-1997 JHWang-1997 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compare command is well structured, but the token gate mixes primary-case metrics with full benchmark-run totals. This can make CI report the opposite of the tested skill's actual token trend. Please see the inline example and suggested fix.

primary := input.PrimaryCaseResults()
metrics := RunMetrics{
CaseCount: len(primary),
TotalTokens: input.TotalTokens,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use primary-case tokens for the token growth gate

This field uses the report-level TotalTokens, while InputTokens and OutputTokens below are calculated from PrimaryCaseResults(). In benchmark mode, report-level TotalTokens includes both with_skill and without_skill executions, so changes in the baseline can mask a token regression in the tested skill.

For example:

Run with_skill tokens without_skill tokens report TotalTokens
Old 100 100 200
New 120 50 170

The tested skill's primary token usage increased by 20%, but --max-token-increase-percent 10 sees 200 -> 170 and incorrectly passes because the full benchmark total decreased by 15%. The text output is also internally inconsistent: primary input/output tokens increase while total tokens decreases.

Since this command otherwise applies primary-case semantics to run metrics and transitions, please derive the comparable total from the primary results as well, for example after accumulation:

metrics.TotalTokens = metrics.InputTokens + metrics.OutputTokens

If full-run benchmark cost is also useful, it should be exposed under a separate, clearly named metric and should not drive the primary token regression gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): add offline result compare command

3 participants