Report real Grok token usage and list-price cost from CLI logs - #3135
Report real Grok token usage and list-price cost from CLI logs#3135olddonkey wants to merge 23 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09cf7edb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
|
Codex review: needs maintainer review before merge. Reviewed August 29, 2026, 9:32 PM ET / August 30, 2026, 01:32 UTC. ClawSweeper reviewWhat this changesThis PR replaces Grok context-window totals with bounded completed-turn CLI-log scanning, estimates native Grok usage at disclosed public xAI list prices, and keeps OpenCodex xAI history token-only without request-time credential provenance. Regression provenancePossible regression — probable (reviewed change; failure trace). No predecessor PR is attributed. Merge readinessKeep this PR open for the repository owner’s product decision on displaying a non-billed public xAI list-price estimate by default. The completed-turn scanner, bounded execution path, disclosures, regressions, and current-head real-session proof support the technical implementation. Priority: P2 Review scores
Verification
How this fits togetherCodexBar collects provider usage into snapshots consumed by menu cards and the Usage & Spend dashboard. This change reads local Grok CLI session logs, prices completed turns from the models.dev xAI catalog, and publishes token and estimate data to those views. flowchart LR
A[Grok CLI session logs] --> B[Bounded completed-turn scanner]
C[models.dev xAI catalog] --> D[List-price calculation]
B --> D
D --> E[Grok usage snapshot]
E --> F[Menu and cost history]
E --> G[Usage and Spend dashboard]
H[OpenCodex xAI history] --> I[Token-only route]
Decision needed
Why: The previous Grok surface exposed token totals without a dollar estimate, so choosing the meaning of the newly displayed default signal requires product judgment. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Retain the bounded completed-turn implementation and explicitly approve the disclosed estimate as the default, or direct a narrowly scoped token-only/opt-in alternative. Do we have a high-confidence way to reproduce the issue? Yes. The supplied exact-head native-session transcript exercises the bounded Grok scanner against real local CLI logs and reports tokens, list-price provenance, and the disclosed estimate. Is this the best way to solve the issue? Unclear. The technical path is well covered, but an owner must decide whether a non-billed list-price dollar value belongs in the default Grok experience. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against e0d2fd90be45. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (30 earlier review cycles; latest 8 shown)
|
08360b5 to
e3cd3b9
Compare
olddonkey
changed the title
clawsweeper
Bot
added
merge-risk: 🚨 auth-provider
|
Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against P1 — Preserve the Grok fallback on repeated probe failuresFixed in if provider == .grok {
if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
Task { @MainActor [weak self] in
await self?.scanAndPublishGrokLocalTokenSnapshot(...)
}
}
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
self.clearTokenSnapshot(for: provider)
}Regression coverage is in P2 — Refresh pricing before scanning Grok sessionsCorrect, and thank you — this was a genuine gap and not one the local tests would have surfaced. Fixed in Note the inline comment still points at Coverage: Real-session evidence
The same corpus on Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent. Merge risk / branch stateRebased onto current One thing deliberately left undone: no |
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
Both new findings addressed at P1 — Do not map every xAI log record to the Grok subscriptionAgreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own The usage log carries no per-record credential evidence; I checked every field emitted for So attribution now requires positive OAuth evidence:
Coverage:
P2 — Republish the Grok snapshot after a missing catalog refreshesI looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree: The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every The alternative — plumbing a completion signal back across the actor boundary into the Happy to build it if you would rather have it. EvidenceThe attribution itself only becomes visible in the app: The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is
Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config. StateFull suite on 
|
clawsweeper
Bot
added
proof: 📸 screenshot
olddonkey
changed the title
|
Addressed both current findings in
Validation on the exact pushed head:
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
merge-risk: 🚨 availability
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
the
proof: sufficient
|
Rebased PR #3135 onto current upstream
Exact-head validation:
The PR body now carries the rebased exact-head evidence. ClawSweeper is already reviewing |
|
Rebased onto current upstream Conflict resolution
Four issues the rebase surfaced that were not conflicts
Pi cache predecessor gate — please reviewFollowing the precedent in The compatibility basis: the only parser-hash-scoped change in this branch is The bound I could not close: Validation on exact head
|
|
Rebased onto 0.56.0 Why the description changedIt still cited exact head It also contained a contradiction worth calling out rather than quietly fixing: the body claimed the Parser-hash bookkeeping for
|
|
All three P2 findings from the latest ClawSweeper review are addressed on exact head Run refreshable Grok scans on the dedicated executor
The unused Regression: a blocker occupies the executor, and the Grok scan is asserted not to finish for 300 ms — an inline scan of that fixture completes in microseconds. Releasing the blocker yields the correct total. Use an inclusive local-day cutoff for Grok historyThe cutoff is now Invalidate Pi caches across the newly priceable xAI routeI took this one slightly differently from the literal suggestion, and want to flag that rather than bury it. Rather than dropping the reviewed-predecessor gate outright, adoption is now gated per cache by The reasoning: reviewing a parser transition establishes that the cache shape is unchanged, which is a different claim from output equivalence. Gating on content proves equivalence per cache instead of assuming it, and it keeps the existing If you would still rather have the blunt version (no adoption at all across this transition), say so and I will swap it; the content check is the only thing standing between the two. Regression: a cache holding Validation on exact head
|
…on logs two ways and expensive in a third. Wrong tokens: the scanner summed `contextTokensUsed` from `signals.json`, which is the session's ENDING context-window occupancy, not what it consumed. On a real machine that reported 653K where actual consumption was 48.0M. Read the sibling `updates.jsonl` instead, where every `turn_completed` event carries the turn's real usage, and bucket by the per-line timestamp so a session crossing local midnight lands in both days. No cost: `toCostUsageTokenSnapshot` hardcoded nil dollars, and nothing could have priced a Grok model anyway because `codexModelsDevProviderIDs` had no `xai`. Add it, and resolve `grok-<version>-build` onto its base catalog model — the `-build` suffix is an artifact of the responses-API surface, not a separate SKU. `grok-build-0.1` is a real model and is never rewritten. Cost is the public xAI card via models.dev, provenance `.listPriceEstimate`, so Grok stays comparable with Claude and Codex. grok's own `costUsdTicks` is deliberately not used for display. A turn's `usage` is the aggregate of `modelCalls` API calls, so tiering on the turn total would push nearly every multi-call turn into the >=200k bracket. Price on the per-call average instead, in closed form over the two synthetic call groups. This under-tiers slightly when context grows within a turn (measured ~4% below the vendor's own accounting on a 27-turn sample, against ~+10% for aggregate tiering); the trade is documented at the call site and pinned by a test. Main-actor cost: the scan ran synchronously inside `@MainActor UsageStore` on every menu-card build, refresh and dashboard load. It now reads the projection the async probe already produced, and the remaining fallback scans on a detached task with one scan in flight at a time. The probe projects the maximum window and consumers narrow it, so `costUsageHistoryDays` and the dashboard's 365-day request are both honoured. Hardening: `modelCalls` comes from a file, so it is validated before it can size any work; parsing is cached per (path, size, mtime) with entries evicted when a file is no longer visited; the cache lock is not held across file reads. Note for upgraders: adding `xai` to `codexModelsDevProviderIDs` changes the Codex pricing-cache key, so the first launch after this re-prices existing Codex history once. Same one-time cost as when kimi and deepseek were added.
OpenCodex sends inference straight to api.x.ai using the Grok account's OAuth credentials, so it burns the same SuperGrok subscription the Grok provider reports on. It only spawns the `grok` binary to refresh tokens, so those requests never reach ~/.grok/sessions and the local session scanner cannot see them — 1,435 requests on one real machine that CodexBar attributed to nothing. Route the `xai` provider prefix to the Grok subscription, the same way `openai` already routes to Codex. Like that mapping, this routes on the prefix and does not distinguish OAuth from API-key traffic. The `-build` suffix seen in the data is a responses-API protocol artifact, not a separate billing pool, so traffic is not split by it. Routing alone would have produced tokens with no dollars. The aggregator priced the bare `entry.model`, and a name without a route prefix is resolved against the `openai` provider — which is why `gpt-5.6-sol` prices today and `grok-4.6` resolved to `openai/grok-4.6` and missed. Qualify an unprefixed model with its provider before pricing. Codex rows are unaffected (the qualified name resolves to the same target), and providers outside the supported set keep returning nil.
Grok resolved list prices straight out of the cached models.dev catalog, but nothing in its path ever fetched that catalog. The only fetch trigger is CostUsageFetcher.refreshPricingIfAllowed, which is gated to Codex and Claude — and Grok never reaches it at all, because its snapshot comes from the provider probe rather than the shared token-cost pipeline. On a machine where Codex or Claude is also enabled the cache is already there, so this is invisible. Enable only Grok and the file never appears: every price lookup returns nil and the Cost row shows tokens with no money, permanently. Request ModelsDevPricingPipeline.refreshIfNeeded from the Grok scan paths. It is safe to call repeatedly — it returns immediately unless the cache is stale and serialises through its own coordinator — and it is detached rather than awaited, matching how the Codex and Claude paths already treat it: pricing availability must never delay or fail a local scan, and the next refresh fills in the value. `summarize` stays synchronous and side-effect free; the refresh lives in a wrapper so the parse-cache behaviour and existing tests are untouched. Reported as P2 by the automated review on the pull request.
… proof The opt-in live proof scanned real sessions but printed tokens only, which cannot evidence the half of this change that is about money. It now also reports today's and the window's list-price cost, the provenance, the window actually used, and how many days carried a price versus tokens — so an all-unpriced result is visible in the output instead of reading as zero. Still skipped unless CODEXBAR_LIVE_GROK_CATALOG_PROOF=1.
The regression guard drove a single failing refresh after a local publication existed. The defect it covers is specifically about the *second* failure: the first one publishes through the fallback scan, and only the next one arrives with a publication already in place — which is what used to hit the generic clear branch. Drive the failure twice and assert the row and the scan count both hold.
Routing every OpenCodex `xai` record to the Grok subscription is right for the case that motivated it — traffic authenticated with the user's Grok account, which is what makes it burn the SuperGrok quota. It is wrong for anyone using an xAI API key: their pay-as-you-go developer-platform spend gets folded into the subscription row, silently inflating it. CodexBar models that platform as its own xAI provider precisely to keep the two apart. The usage log carries no per-record credential evidence, so the decision has to come from the OpenCodex provider config, which records `authMode` per provider. Read it, and attribute to Grok only when that mode is OAuth; anything else is token-only spend that belongs to no tracked subscription. Fail closed: a missing or malformed config, no `xai` entry, or an absent `authMode` all count as no OAuth evidence and keep the records off the Grok row. The dispatcher stays a pure function — the set of OAuth-backed provider ids is threaded in from the caller rather than read at the routing site — and the gate applies only to `xai`, leaving the other routes exactly as they were. Also records why the Grok pricing refresh stays fire-and-forget: the parse cache holds parsed turns rather than prices, so the next scan reprices against the refreshed catalog, and plumbing completion back to republish was judged disproportionate to a delay Codex and Claude already share. Raised as P1 by the automated review; the owner chose verifiable attribution over prefix-only routing.
Rebase onto current main and reconcile the surfaces it moved: - Route the Grok local summary through the injectable `localSummary`/`cliVersion` seams steipete#3237 introduced, keeping the 365-day lookback and the models.dev pricing refresh in the injected defaults rather than at the call sites. - Regenerate the Codex parser hash and record main's `21f10143afe00c55` as a compatible predecessor; the Grok-only parser additions leave persisted Codex rows unchanged. Drop the stale branch-internal predecessor entry. - Re-anchor the provider-architecture gatekeeper suppressions and allowlists to the line numbers main's Spend dashboard and usage store now sit at. - Anchor the unstubbed Grok publish test to the real clock instead of a fixed calendar day, which had drifted outside its own seven-day window. - Move the xAI pricing-fingerprint test next to the other Grok pricing tests so `CostUsagePricingTests.swift` stays byte-identical to main and inside the file- and type-length limits.
0.56.0 shipped without this change, so the entry belongs in the open `0.56.1 — Unreleased` section rather than the released one.
- Queue the refreshable Grok summary through `CostUsageScanExecutor`. It called the synchronous corpus scanner inline, so the probe and descriptor callers ran a potentially multi-minute scan on the cooperative pool the executor exists to protect. A cancelled scan now reports unestablished coverage rather than an authoritative zero. - Derive the Grok lookback cutoff from the local start of day minus `historyDays - 1` so the scan window matches the inclusive local-day window `narrowed(toHistoryDays:)` renders, instead of collecting a partial extra day consumers discard. Each fix carries a regression that was confirmed to fail without it.
steipete#3259 moved the Codex parser hash to `d9a91f31d0addc15` and removed the Pi cache's reviewed-predecessor adoption entirely, so this branch no longer carries a Pi gate: `PiSessionCostScanner` and `PiSessionCostCompatibilityTests` are byte-identical to main again, which is the conservative option the maintainer offered on the previous head. - Regenerate `CodexParserHash.value` to `ac4862abcdfe21a8`. - Record main's `d9a91f31d0addc15` in `CostUsageStore.compatiblePredecessorParserHashes` and in its exact-set assertion.
|
Rebased onto current upstream The Pi cache question is closed — by #3259, not by me#3259 removed I dropped this branch's whole Pi contribution accordingly, including the Conflict resolution
Parser-hash bookkeepingTwo gates now instead of three:
I also rewrote two commit messages that still described the dropped Pi gate, so the history matches what the commits actually contain. Validation on exact head
|
2 participants
Problem
#3085 enabled Grok token cost, but the local session projection was not measuring actual consumption:
signals.jsonexposes ending context-window occupancy, not per-turn token usage. On a real corpus it reported 653K tokens where the completed turns contained 54.1M.nil, and the models.dev resolver did not include the xAI catalog.updates.jsonlfiles could not safely remain on@MainActor.What this changes
Read the completed-turn usage that the Grok CLI actually records
GrokLocalSessionScannerreadsturn_completedevents from each session'supdates.jsonl, matches bothsession/updateand_x.ai/session/update, and buckets each line by its own timestamp. It preserves raw aggregate token totals and uses the recordedmodelCallscount only to approximate per-call tiered list pricing in O(1).The parser streams through the shared chunked JSONL reader instead of loading whole files. Production bounds are explicit: a 64 MiB tail per file, 1 MiB per record, 20,000 retained turns per file, and global scan budgets of 256 recent sessions, 256 MiB, and 100,000 turns. The process-wide LRU cache retains at most 64 files or 50,000 turns. Cancellation is checked between chunks, I/O and cancelled results are not cached, and any truncation marks history incomplete rather than presenting a partial total as complete.
Price the Grok models and preserve provenance
The models.dev xAI catalog is now eligible for cost pricing. Responses-API names such as
grok-4.6-buildresolve to the basegrok-4.6SKU after exact lookup, while real independent names such asgrok-build-0.1remain untouched. Cost is published as.listPriceEstimate; Grok's internalcostUsdTicksis not presented as billed spend.On a stale catalog, the scan prices immediately and refreshes in the background. On a fresh install with no catalog artifact, the first scan awaits the initial best-effort refresh attempt before creating the snapshot, so a successful first refresh is visible in the first publication. A refresh failure still degrades to token-only data rather than failing the local scan.
Keep scanning and publication off the main actor
The provider projection consumes the async probe's snapshot. Remaining fallback paths scan on a detached utility task with a single scan in flight. A maximum-window snapshot is narrowed by each consumer through
CostUsageTokenSnapshot.narrowed(toHistoryDays:calendar:), including the spend dashboard's 365-day request.The 365-day lookback and the models.dev pricing refresh now live in the injected defaults of the
localSummary/cliVersionseams introduced by #3237, so the credential-binding seam and this branch's scan behavior both hold.Keep OpenCodex xAI history out of the Grok subscription row
OpenCodex
usage.jsonlrecords provider/model usage but does not retain the credential mode used for each request. Reading today'sconfig.jsoncannot distinguish older API-key traffic from older OAuth traffic after a configuration switch.For that reason, xAI entries remain
.tokenOnlyand are not merged into the Grok subscription row until the producer records request-time credential provenance. The current-config reader and its routing parameter were removed, with dispatcher and fan-out regressions and updated documentation. Other existing OpenCodex subscription routes are unchanged.Review findings addressed
All items below are present in the current head; the commit SHAs they were originally fixed on were rewritten by later rebases and are therefore not cited.
xaiModelsDevProviderIDs), so xAI-only catalog changes do not invalidate Codex caches. Route resolution still accepts xAI throughcodexCompatibleModelsDevProviderIDs.updates.jsonl, the requested window up to 365 days, list-price provenance, and every production bound.ClawSweeper P2 findings — addressed
summarizeRequestingPricingRefreshqueues its corpus scan throughCostUsageScanExecutorinstead of calling the scanner inline, so it no longer occupies the cooperative pool. A cancelled scan reports unestablished coverage rather than an authoritative zero.startOfDay(now)minushistoryDays - 1, matching the windownarrowed(toHistoryDays:)renders, so no partial extra day is collected for consumers to discard.PiSessionCostScannerat all; the per-cache content gate it previously carried is gone with the mechanism it gated.The first two carry regressions confirmed to fail without their fix.
Changelog placement — corrected
An earlier revision removed the
CHANGELOG.mdentry on the assumption that release notes were release-owned. That was wrong for a normal PR in this repository, and the entry had been silently absent since. It lives under the## 0.56.1 — Unreleasedsection main already carries, because 0.56.0 shipped without this change.Parser-hash bookkeeping
This branch touches
CostUsagePricing.swift, which is inside the Codex parser-hash scope, so both remaining gates were updated together on the current head:CodexParserHash.valueregenerated toac4862abcdfe21a8.CostUsageStore.compatiblePredecessorParserHashesrecords main'sd9a91f31d0addc15, with the exact-set assertion inCostUsageStoreTestskept in sync.There is no longer a third gate. #3259 removed
PiSessionCostScanner's reviewed-predecessor adoption, so a parser-hash change invalidates Pi caches normally and there is nothing to review or advance.PiSessionCostScanner.swiftandPiSessionCostCompatibilityTests.swiftare byte-identical to main on this head, which is exactly the conservative outcome offered as the alternative on the previous head.Deterministic behavior proof
The production bounded JSONL scanner, publication path, failure preservation, timestamp selection, menu consumers, and spend-dashboard capture are driven through temporary on-disk Grok session files:
The regression writes a completed 77-token turn and installs its projection as the retained remote snapshot. It appends a second completed 23-token turn, injects
URLError.timedOut, verifies the remote snapshot is preserved at 77, verifies the local publication advances to 100, and verifies the live consumer selects the newer 100-token snapshot. A second failed refresh proves local logs are rescanned again. The status-menu regression separately proves newer local tokens beat stale remote tokens in both the visible card and the cost-history submenu, while the override-card isolation regression remains green. The dashboard regression independently installs a one-minute-older 77-token remote snapshot and a current 100-token local publication, then verifies capture-only Usage & Spend selects the local 100-token, $1 list-price snapshot and its newer timestamp.Real-session evidence (current head)
The opt-in native-session proof below was re-run on this exact head against the real
~/.grok/sessions/**/updates.jsonlcorpus and the local models.dev catalog, so it is no longer earlier-head evidence. It reads local session logs only — no credentials, no network probe, no Keychain access.All 3 tests in that suite passed. The transcript prints aggregate fields only. The corpus had no completed-turn tokens that day and one priced day in the last 30, so these values change as local logs age. The populated-surface assertion in the same suite verifies the visible
Public xAI list-price estimate · not a bill.disclosure.No credential values, account identity, session contents, or interactive Keychain and browser-cookie reads were emitted by any of this work.
Testing
Verified on exact head
eef38cd29ca08d98abd730ac1c3dd154359ae2d9:make check: passed — parser hashac4862abcdfe21a8, provider/package/documentation gates, SwiftFormat clean, SwiftLint 0 violations in 2055 files.make test: passed — 965/965 selections, 81/81 groups successful on the first pass, 0 failed groups, 0 retries, 0 timeouts (666.0 seconds).ProviderArchitectureGatekeeperTests39/39 and 135 tests across the Grok scanner/pricing/menu/dashboard, OpenCodex routing/fan-out,CostUsageStoreTestsandPiSessionCostCompatibilityTestssuites.maine0d2fd90b(0.56.1 development head), 0 behind / 23 ahead,git diff --checkclean.Automated tests use temporary homes, injected catalogs and transports, and no live credentials or interactive Keychain reads.
Maintainer decision
Every code finding from the 2026-08-25 review and the ClawSweeper follow-up is addressed and still in place after this rebase. The branch is mergeable again and fully green locally.
One thing is still yours to call:
The Pi cache question from the previous head is closed: #3259 removed the mechanism upstream, so this branch invalidates Pi caches normally and carries no Pi changes.