Skip to content

test(metrics): assert the two usage aggregates agree - #167

Open
lizhuojunx86 wants to merge 1 commit into
sandbaseai:mainfrom
lizhuojunx86:test/usage-accounting-identity
Open

test(metrics): assert the two usage aggregates agree#167
lizhuojunx86 wants to merge 1 commit into
sandbaseai:mainfrom
lizhuojunx86:test/usage-accounting-identity

Conversation

@lizhuojunx86

Copy link
Copy Markdown

What changed

Two test files, no source changes.

/metrics/summary returns the same total twice, computed two ways (runtime.ts:137-149):

events.input_tokens    SUM(tokens_in) FROM events WHERE type = 'span.model_request_end'
sessions.input_tokens  SUM(usage_tokens_in) FROM sessions

Since bdd164a6d those are supposed to be equal. Nothing checked that they are, so drift on either path ships silently. These tests check it.

tests/unit/usage-accounting.test.ts builds a log by hand and asserts the identity through the real route, which is mounted on a database alone because the /metrics/summary handler only reaches for deps.db. Seven cases, 0.6s. Three of them assert the identity fails where it should, so the check has teeth rather than passing by construction.

tests/integration/usage-accounting.test.ts runs DefaultStrategy against a mock model reporting known usage, following the pattern in strategy-tool-schema.test.ts. Four cases, 1.9s. This one covers the write paths at default-strategy.ts:190 and :199 rather than simulating them.

No ground truth, no vendor account, no network.

Why the identity is worth pinning

It catches drift on either of the two paths the fix created, and it costs nothing to run. I checked that it catches both by breaking them:

Mutation Result
Drop WHERE type = 'span.model_request_end' from eventUsage 4 of 7 unit tests fail, expected 360 to be 120
Drop eventLog.recordUsage(...) at default-strategy.ts:199 all 4 integration tests fail, expected 120 to be +0

The first failure message names the double count directly, which is the behaviour the filter was added to prevent.

It also makes one existing silent failure visible. recordUsage is a bare UPDATE ... WHERE id = ? and the changes count from run() is not checked, so a request against a session row that isn't there increments nothing and raises nothing. The current test pins that behaviour as not-throwing (event-logger.test.ts:120-122); with the identity in place the divergence it produces is caught instead of absorbed. That case is in the unit file.

What these tests do not claim

The identity is necessary, not sufficient. Both aggregates can be equally wrong and it still holds. Two known cases, both of which you flagged as deferred in #77:

  • context-compactor.ts:77 destructures const { text } = await generateText(...), so usage is never bound. outcome-evaluator.ts:58 keeps the whole response and reads only response.text (:70). Neither path writes usage anywhere, so the two sides agree while both are short by the same amount.
  • InMemoryEventLog.recordUsage is a no-op (in-memory-event-log.ts:43-45) and the class writes nothing to the database, so delegated runs miss both columns equally. The comment there already says this is intended for ephemeral delegation, which answers the question I left open in Usage accounting: one model request lands on 2..N+2 events, and sessions.usage_tokens_* is never written #77. Treat that one as withdrawn.

Neither is in scope here. I mention them so the green check isn't read as covering more than it does.

Validation

  • npm run typecheck
  • npm test
  • npm run build
  • Documentation or tests were updated when behavior changed
  • No secrets, credentials, or personal data are included

Run against a634eb431 on Node 22.23.2 (82 files, 623 passed, 15 skipped) and on
Node 26.5.1 (82 files, 624 passed, 14 skipped). The skip counts differ because the
Docker and Kubernetes sandbox suites skip on backend availability, not because of
anything here.

Offered, not included

If you would rather the SQL live in one place than be re-stated in the tests, I can extract the two aggregates into a small helper that runtime.ts and both tests call. I left it out to keep this focused, and because it touches code you just wrote. Say the word and I'll push it here.

Refs #77

/metrics/summary reports the same total twice, computed two ways. Since
bdd164a they are meant to agree and nothing checked that they do.

Adds a unit test over a hand-built log and an integration test that drives
DefaultStrategy with a mock model, so both the read path and the two write
paths are covered.

Refs sandbaseai#77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant