Filter reasoning output and add API_EXTRA_BODY passthrough - #797
Open
rodrigolopezguerra wants to merge 2 commits into
Open
Filter reasoning output and add API_EXTRA_BODY passthrough#797rodrigolopezguerra wants to merge 2 commits into
rodrigolopezguerra wants to merge 2 commits into
Conversation
Reasoning providers (MiniMax-M3, DeepSeek-R1, Qwen QwQ, GLM) leak their thinking
into the streamed content either as <think>...</think> blocks inside the
delta.content field, or as a separate reasoning_content / reasoning field.
With --shell / --code that thinking used to be concatenated into the
generated command or code, producing syntax errors at execution time (e.g.
zsh unmatched quote from a real MiniMax-M3 run).
Two changes:
1. sgpt/config.py: a new API_EXTRA_BODY key. JSON object that sgpt forwards
to the provider so users can disable thinking at the source, e.g.
API_EXTRA_BODY={"thinking": {"type": "disabled"}} for MiniMax.
Empty by default -> behavior unchanged.
2. sgpt/handlers/handler.py:
- _parse_api_extra_body / _apply_api_extra_body: parsed once at module
load. OpenAI branch passes it as extra_body=; litellm branch merges
it as top-level kwargs (litellm does not accept extra_body).
Invalid JSON or non-object raises UsageError.
- _make_thinking_filter: a stateful generator that strips
<think>...</think> and <thinking>...</thinking> from the streamed
content, byte-identical when no tag is present and correctly handling
tags split across chunks. Unterminated blocks at end-of-stream are
discarded.
- get_completion now creates one filter per call so state survives
across chunks, and drops reasoning_content / reasoning fields.
…tput Filter reasoning output and add API_EXTRA_BODY passthrough
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Reasoning-capable providers (MiniMax-M3, DeepSeek-R1, Qwen QwQ, GLM) leak their
thinking into the streamed output. With OpenAI-compatible APIs that reasoning
arrives in two shapes:
<think>...</think>(or<thinking>...</thinking>) blocks insidedelta.content— most common, what MiniMax-M3 and DeepSeek-R1 do.reasoning_content/reasoningfield in the delta.With
sgpt -sorsgpt --code, that thinking was concatenated to thegenerated command or code. Concrete reproduction on MiniMax-M3: the model's
reasoning contained the phrase "if it's behind/ahead" and zsh aborted with
unmatched 'because the apostrophe leaked into the command.Fix
Two independent changes, both backwards compatible (empty defaults preserve
prior behavior).
sgpt/config.py— newAPI_EXTRA_BODYkeyA JSON string merged into the request body so users can disable thinking at
the provider level when supported:
Empty by default → no body changes.
sgpt/handlers/handler.py_parse_api_extra_body/_apply_api_extra_body: parsed once at moduleload. OpenAI branch passes it as
extra_body=...; litellm branch merges itas top-level kwargs (litellm does not accept
extra_body). Invalid JSON ornon-object raises
UsageErrornaming the key._make_thinking_filter: a stateful generator that strips<think>...</think>and<thinking>...</thinking>from the streamedcontent. State (tag buffer, inside-block flag) survives across chunks so
tags split mid-stream are handled correctly. Unterminated thinking blocks at
end-of-stream are discarded. When no tag appears the output is byte-identical
to the input.
get_completionnow creates one filter per call and readscontentdefensively per backend (
delta.get("content")for litellm,delta.contentfor OpenAI Pydantic). Reasoning-only chunks yield nothing because we only
read
content, notreasoning_content/reasoning.The
tool_callsrecursion, the@cachedecorator onget_completion, andthe reset of
additional_kwargsto{}in the OpenAI branch are alluntouched.
Test coverage
19 new tests in
tests/test_handler_filter.py:filter_thinking_tokens: tag split across chunks, two blocks, alt<thinking>, unclosed block discarded, byte-identical passthrough, lone<flushed, close tag in next chunk._parse_api_extra_body: empty / valid object / invalid JSON / non-object /string — all with explicit
UsageErrorassertions._apply_api_extra_body: OpenAI getsextra_body=; litellm gets a top-levelmerge; both no-op when body is empty.
split-tag stream drops content from inside the block, litellm stream with
reasoning_contentin the delta yields only the content.Verification on a real reasoning provider
Tested against MiniMax-M3 (api.minimax.io/v1) in shell mode. Raw stream
contained 1096 chars of inline
<think>...</think>content; the finalfiltered output was just the clean shell command — no leakage.