feat(pi): carry thinking_delta as ChatDelta.reasoning so retained_events keeps the model's reasoning - #185
Merged
Merged
Conversation
…nts keeps the model's reasoning The pi backend dropped every thinking_* event, so the retained event log kept none of the model's reasoning (0 of 23,110 rows) while pi's own session files held 1.89 M reasoning characters. ChatDelta gains an optional reasoning field. The pi backend buffers thinking_delta text and flushes one delta at 256 chars, at 250 ms of buffer age, or before any non-thinking yield, so ordering against content and tool_calls stays exact and the row rate does not grow ~18x. The SSE writer passes it through as choices[0].delta.reasoning; the non-streaming body is unchanged. Replay-log byte accounting and the estimated-usage fallback count reasoning text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The pi backend drops every
thinking_*event frompi --print --mode json(src/backends/pi.ts, pinned bytests/pi-backend.test.ts), andChatDeltahas no reasoning field, so the model's reasoning never reaches the retained event log. The reasoning does arrive: in a running fleet (pi 0.84.2, providertangle-router, modelopenrouter/stealth/ox-alpha,--thinking high), pi's own session files hold 556 thinking blocks across 19 of 22 sessions — 1.89 M reasoning characters against 108 K visible-text characters — whileretained_eventsfor the same sessions holds 0 rows with any reasoning key (23,110 rows inspected). The bridge's inference proxy is not the cause; it forwards the request body byte-for-byte and pipes the SSE response through identity and redaction transforms only.Change
ChatDelta.reasoning?: string(src/backends/types.ts): incremental reasoning text, never assistant content.src/backends/pi.ts:thinking_deltano longer hits the drop path. Deltas buffer and flush as one{ reasoning }ChatDelta;thinking_start/thinking_endremain dropped as boundary markers. A reasoning block ends when the next non-reasoning delta arrives.src/streaming/sse.ts: the SSE writer mapsreasoningtochoices[0].delta.reasoning— the field OpenRouter and pi already use.collectNonStreamingoutput is unchanged; reasoning never entersmessage.content.src/runs/replay-log.ts: replay retention accounting counts reasoning bytes.src/routes/chat-completions.ts: the estimated-usage fallback counts reasoning characters as output.appendRetainedDeltaalready persists the delta verbatim under{"type":"raw","backend":"cli-bridge.chat","event":…}.Coalescing rule and cost
Pi emits one
thinking_deltaper provider chunk (a few characters), and reasoning outweighs visible text ~17:1 by characters, so per-chunk rows would multiply the retained-event row rate ~18x (about 400,000 extra rows for the 31 runs measured). The backend buffers reasoning and flushes at 256 characters, at 250 ms of buffer age, or before any non-thinking yield — ordering againstcontent/tool_calls/usagestays exact, a live reader sees reasoning within a quarter second, and the row count divides by roughly 50.Acceptance
reasoningEffort: highproducesretained_eventsrows whose inner event hasreasoning, interleaved before thecontentrows of the same assistant message.collectNonStreamingoutput is unchanged (reasoning never entersmessage.content).contentdeltas are unchanged.Research:
tangle-network/discovery-lab/docs/research/reasoning-deltas.md(on master).