Skip to content

feat(pi): carry thinking_delta as ChatDelta.reasoning so retained_events keeps the model's reasoning - #185

Merged
drewstone merged 1 commit into
mainfrom
feat/pi-reasoning-delta
Aug 22, 2026
Merged

feat(pi): carry thinking_delta as ChatDelta.reasoning so retained_events keeps the model's reasoning#185
drewstone merged 1 commit into
mainfrom
feat/pi-reasoning-delta

Conversation

@drewstone

Copy link
Copy Markdown
Owner

Problem

The pi backend drops every thinking_* event from pi --print --mode json (src/backends/pi.ts, pinned by tests/pi-backend.test.ts), and ChatDelta has no reasoning field, so the model's reasoning never reaches the retained event log. The reasoning does arrive: in a running fleet (pi 0.84.2, provider tangle-router, model openrouter/stealth/ox-alpha, --thinking high), pi's own session files hold 556 thinking blocks across 19 of 22 sessions — 1.89 M reasoning characters against 108 K visible-text characters — while retained_events for the same sessions holds 0 rows with any reasoning key (23,110 rows inspected). The bridge's inference proxy is not the cause; it forwards the request body byte-for-byte and pipes the SSE response through identity and redaction transforms only.

Change

  1. ChatDelta.reasoning?: string (src/backends/types.ts): incremental reasoning text, never assistant content.
  2. src/backends/pi.ts: thinking_delta no longer hits the drop path. Deltas buffer and flush as one { reasoning } ChatDelta; thinking_start / thinking_end remain dropped as boundary markers. A reasoning block ends when the next non-reasoning delta arrives.
  3. src/streaming/sse.ts: the SSE writer maps reasoning to choices[0].delta.reasoning — the field OpenRouter and pi already use. collectNonStreaming output is unchanged; reasoning never enters message.content.
  4. src/runs/replay-log.ts: replay retention accounting counts reasoning bytes. src/routes/chat-completions.ts: the estimated-usage fallback counts reasoning characters as output.
  5. No store, route, run-registry, or replay-log structural change: appendRetainedDelta already persists the delta verbatim under {"type":"raw","backend":"cli-bridge.chat","event":…}.

Coalescing rule and cost

Pi emits one thinking_delta per provider chunk (a few characters), and reasoning outweighs visible text ~17:1 by characters, so per-chunk rows would multiply the retained-event row rate ~18x (about 400,000 extra rows for the 31 runs measured). The backend buffers reasoning and flushes at 256 characters, at 250 ms of buffer age, or before any non-thinking yield — ordering against content / tool_calls / usage stays exact, a live reader sees reasoning within a quarter second, and the row count divides by roughly 50.

Acceptance

  • A fleet run on pi with reasoningEffort: high produces retained_events rows whose inner event has reasoning, interleaved before the content rows of the same assistant message.
  • collectNonStreaming output is unchanged (reasoning never enters message.content).
  • The existing test's content deltas are unchanged.

Research: tangle-network/discovery-lab/docs/research/reasoning-deltas.md (on master).

…nts keeps the model's reasoning

The pi backend dropped every thinking_* event, so the retained event log
kept none of the model's reasoning (0 of 23,110 rows) while pi's own
session files held 1.89 M reasoning characters.

ChatDelta gains an optional reasoning field. The pi backend buffers
thinking_delta text and flushes one delta at 256 chars, at 250 ms of
buffer age, or before any non-thinking yield, so ordering against
content and tool_calls stays exact and the row rate does not grow ~18x.
The SSE writer passes it through as choices[0].delta.reasoning; the
non-streaming body is unchanged. Replay-log byte accounting and the
estimated-usage fallback count reasoning text.
@drewstone
drewstone merged commit f7e1901 into main Aug 22, 2026
1 check passed
@drewstone
drewstone deleted the feat/pi-reasoning-delta branch August 22, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant