fix(orchestrator): preserve conclude DLQ outcomes#426
Draft
albertywu wants to merge 1 commit into
Draft
Conversation
Summary: Intent: - Keep DLQ reconciliation consistent with the terminal outcome already selected by conclude. - Preserve useful failure context in terminal request logs. Changes: - Reuse the parent request termination helper across DLQ reconciliation. - Preserve successful, failed, and cancelled batch outcomes in conclude DLQ fanout. - Record original topic and failure metadata on DLQ terminal logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #424 to reuse shared request termination in DLQ reconciliation and correctly repair requests when conclude itself dead-letters.
Scope: This is a targeted follow-up to the refactor within the existing architecture, so an RFC is not needed. It keeps the existing DLQ topics, payloads, controller topology, stores, and request-log path. It reuses #424's helper to avoid duplicating request termination mechanics. Its behavior changes are limited to repairing conclude DLQ fanout according to the existing batch outcome and preserving structured DLQ context.
reconcileRequest, which callsrequest.ReconcileTerminalState; request-scoped DLQs do this once, while batch-scoped DLQs do it once for each member request.dlq.last_errorintoLastError, but passed nil forRequestLog.Metadata, so structured delivery context was dropped.TerminalOutcometorequest.ReconcileTerminalState, keepingdlq.last_errorinLastErrorand addingdlq.original_topic,dlq.failure_count, anddlq.failed_attoMetadatawhen available.conclude_dlqused generic batch-failure reconciliation. A Succeeded or Cancelled batch was skipped, which could leave member requests non-terminal if conclude failed during fanout.concludeBatchmaps the existing batch outcome to a request terminal state, then callsrequest.ReconcileTerminalStatethroughreconcileRequestonce for each member; the helper updates requests only and leaves the batch outcome unchanged.Helper call path: In every After case above, each request terminal-state reconciliation is performed by
request.ReconcileTerminalState. Request-scoped DLQs call it throughreconcileRequest; batch and conclude DLQs callreconcileRequestonce for each member request. Batch failure still usesBatchStore.UpdateState, while conclude DLQ preserves the batch's existing terminal state.Normal DLQs still reconcile requests to Error and batches to Failed.
ReconcileTerminalStateusagereconcileRequestErrorfailBatchErrorfor each member requestconcludeBatchSucceeded,Failed, orCancelledmaps to requestLanded,Error, orCancelledFuture callers should use this helper only after deciding that a request must enter a terminal state and needs the matching public request log. The caller owns the business decision and supplies
State,LastError, andMetadata; the helper owns the CAS, version advancement, state-to-status mapping, idempotency, and log publication. Do not use it for non-terminal request transitions, batch transitions, or log-only events.Test Plan
✅
make lint && make check-tidy && make check-gazelle && make testStack