fix(pipeline): tolerate a flaky synthesizer instead of 502ing the fusion (#32) - #33
Merged
Merged
Conversation
…ion (#32) The synthesizer is a single call with no quorum, so a slow or flaky synthesizer that returned an undecodable body (observed in a live benchmark: a 26-89s model returning bodies chorus could not decode) failed the entire fusion for every task, even though the panel had produced usable answers. The panel tolerates a flaky member via min_quorum and the judge already degrades gracefully; the synthesizer was the one stage with no tolerance. run_synthesis now retries once on any synthesizer failure (a transient slow or undecodable response gets one more chance). If the retry also fails, the pipeline degrades to the strongest panel answer (longest non-empty response) under the fusion alias rather than erroring, and increments a new chorus_synthesis_degraded_total counter so the degradation is observable. It only surfaces Error::Synthesis when no panel answer is available to fall back to. This follows the "boring failure behavior" operating principle: partial failure degrades predictably, it does not crash a request. Adds tests for the synthesizer retry (recovers after one transient failure; surfaces the error after the retry is exhausted) and for the pipeline degradation (a never-decoding synthesizer yields a panel answer with panel and judge usage still accounted, not a 502). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jannes Stubbemann <jannes.stubbemann@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #32. In a live benchmark a slow, flaky synthesizer (26-89s, returning bodies chorus could not decode) 502'd the entire fusion for every task, even though the panel had produced usable answers. The panel tolerates a flaky member via
min_quorumand the judge already degrades gracefully; the synthesizer was the one stage with no tolerance.Change
run_synthesisretries once on any synthesizer failure. A single transient slow or undecodable response gets one more chance before the error is surfaced (issue option a).fusion/<profile>alias, rather than erroring (issue option b). It only returnsError::Synthesiswhen no panel answer is available to fall back to.chorus_synthesis_degraded_totalcounter (labelled by profile) makes the degradation observable rather than silent.This follows operating principle 4 (boring failure behavior): partial failure degrades predictably, it does not crash a request. It mirrors the existing judge degradation path. Option (c) from the issue (more lenient decoding of specific upstream response shapes) is a separate, backend-client-level change and is not needed for the fusion to stay up.
Tests (same commit)
run_synthesis_retries_once_on_transient_error: a backend that fails the first call and succeeds the second returns the synthesized answer after exactly two calls.run_synthesis_surfaces_synthesis_error_after_retry_exhausted: an always-failing synthesizer yieldsError::Synthesis.degrades_to_panel_answer_when_synthesis_fails: a never-decoding synthesizer yields a panel answer under the fusion alias, with panel and judge usage still accounted (no synthesizer usage on the degraded answer).Verification
cargo test --workspace(45 core + 6 server integration),cargo clippy --workspace --all-targets -- -D warnings, andcargo fmt --all -- --checkall pass locally. No public quality or cost numbers changed, so no benchmark run is required.