Skip to content

Filter reasoning output and add API_EXTRA_BODY passthrough - #797

Open
rodrigolopezguerra wants to merge 2 commits into
TheR1D:mainfrom
rodrigolopezguerra:main
Open

Filter reasoning output and add API_EXTRA_BODY passthrough#797
rodrigolopezguerra wants to merge 2 commits into
TheR1D:mainfrom
rodrigolopezguerra:main

Conversation

@rodrigolopezguerra

Copy link
Copy Markdown

Problem

Reasoning-capable providers (MiniMax-M3, DeepSeek-R1, Qwen QwQ, GLM) leak their
thinking into the streamed output. With OpenAI-compatible APIs that reasoning
arrives in two shapes:

  1. Inline <think>...</think> (or <thinking>...</thinking>) blocks inside
    delta.content
    — most common, what MiniMax-M3 and DeepSeek-R1 do.
  2. A separate reasoning_content / reasoning field in the delta.

With sgpt -s or sgpt --code, that thinking was concatenated to the
generated command or code. Concrete reproduction on MiniMax-M3: the model's
reasoning contained the phrase "if it's behind/ahead" and zsh aborted with
unmatched ' because the apostrophe leaked into the command.

Fix

Two independent changes, both backwards compatible (empty defaults preserve
prior behavior).

sgpt/config.py — new API_EXTRA_BODY key

A JSON string merged into the request body so users can disable thinking at
the provider level when supported:

API_EXTRA_BODY={"thinking":{"type":"disabled"}}

Empty by default → no body changes.

sgpt/handlers/handler.py

  • _parse_api_extra_body / _apply_api_extra_body: parsed once at module
    load. OpenAI branch passes it as extra_body=...; litellm branch merges it
    as top-level kwargs (litellm does not accept extra_body). Invalid JSON or
    non-object raises UsageError naming the key.
  • _make_thinking_filter: a stateful generator that strips
    <think>...</think> and <thinking>...</thinking> from the streamed
    content. State (tag buffer, inside-block flag) survives across chunks so
    tags split mid-stream are handled correctly. Unterminated thinking blocks at
    end-of-stream are discarded. When no tag appears the output is byte-identical
    to the input.
  • get_completion now creates one filter per call and reads content
    defensively per backend (delta.get("content") for litellm, delta.content
    for OpenAI Pydantic). Reasoning-only chunks yield nothing because we only
    read content, not reasoning_content / reasoning.

The tool_calls recursion, the @cache decorator on get_completion, and
the reset of additional_kwargs to {} in the OpenAI branch are all
untouched.

Test coverage

19 new tests in tests/test_handler_filter.py:

  • Pure filter_thinking_tokens: tag split across chunks, two blocks, alt
    <thinking>, unclosed block discarded, byte-identical passthrough, lone
    < flushed, close tag in next chunk.
  • _parse_api_extra_body: empty / valid object / invalid JSON / non-object /
    string — all with explicit UsageError assertions.
  • _apply_api_extra_body: OpenAI gets extra_body=; litellm gets a top-level
    merge; both no-op when body is empty.
  • End-to-end through the CLI runner: OpenAI stream filters think blocks,
    split-tag stream drops content from inside the block, litellm stream with
    reasoning_content in the delta yields only the content.

Verification on a real reasoning provider

Tested against MiniMax-M3 (api.minimax.io/v1) in shell mode. Raw stream
contained 1096 chars of inline <think>...</think> content; the final
filtered output was just the clean shell command — no leakage.

$ sgpt -s 'Calculá cuántos archivos modificados en las últimas 24hs hay en /tmp'
find /tmp -type f -mtime -1 | wc -l
$ API_EXTRA_BODY='{"thinking":{"type":"disabled"}}' sgpt -s '...'
find /tmp -type f -mtime -1 | wc -l

Reasoning providers (MiniMax-M3, DeepSeek-R1, Qwen QwQ, GLM) leak their thinking
into the streamed content either as <think>...</think> blocks inside the
delta.content field, or as a separate reasoning_content / reasoning field.
With --shell / --code that thinking used to be concatenated into the
generated command or code, producing syntax errors at execution time (e.g.
zsh unmatched quote from a real MiniMax-M3 run).

Two changes:

1. sgpt/config.py: a new API_EXTRA_BODY key. JSON object that sgpt forwards
   to the provider so users can disable thinking at the source, e.g.
   API_EXTRA_BODY={"thinking": {"type": "disabled"}} for MiniMax.
   Empty by default -> behavior unchanged.

2. sgpt/handlers/handler.py:
   - _parse_api_extra_body / _apply_api_extra_body: parsed once at module
     load. OpenAI branch passes it as extra_body=; litellm branch merges
     it as top-level kwargs (litellm does not accept extra_body).
     Invalid JSON or non-object raises UsageError.
   - _make_thinking_filter: a stateful generator that strips
     <think>...</think> and <thinking>...</thinking> from the streamed
     content, byte-identical when no tag is present and correctly handling
     tags split across chunks. Unterminated blocks at end-of-stream are
     discarded.
   - get_completion now creates one filter per call so state survives
     across chunks, and drops reasoning_content / reasoning fields.
…tput

Filter reasoning output and add API_EXTRA_BODY passthrough
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant