Skip to content

fix(orchestrator): preserve conclude DLQ outcomes#426

Draft
albertywu wants to merge 1 commit into
wua/refactor-request-terminationfrom
wua/refactor-dlq-termination
Draft

fix(orchestrator): preserve conclude DLQ outcomes#426
albertywu wants to merge 1 commit into
wua/refactor-request-terminationfrom
wua/refactor-dlq-termination

Conversation

@albertywu

@albertywu albertywu commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Builds on #424 to reuse shared request termination in DLQ reconciliation and correctly repair requests when conclude itself dead-letters.

Scope: This is a targeted follow-up to the refactor within the existing architecture, so an RFC is not needed. It keeps the existing DLQ topics, payloads, controller topology, stores, and request-log path. It reuses #424's helper to avoid duplicating request termination mechanics. Its behavior changes are limited to repairing conclude DLQ fanout according to the existing batch outcome and preserving structured DLQ context.

Before After
DLQs duplicated request terminal-state and log-publication logic. DLQs route request termination through reconcileRequest, which calls request.ReconcileTerminalState; request-scoped DLQs do this once, while batch-scoped DLQs do it once for each member request.
Terminal DLQ logs copied dlq.last_error into LastError, but passed nil for RequestLog.Metadata, so structured delivery context was dropped. DLQs pass a TerminalOutcome to request.ReconcileTerminalState, keeping dlq.last_error in LastError and adding dlq.original_topic, dlq.failure_count, and dlq.failed_at to Metadata when available.
conclude_dlq used generic batch-failure reconciliation. A Succeeded or Cancelled batch was skipped, which could leave member requests non-terminal if conclude failed during fanout. concludeBatch maps the existing batch outcome to a request terminal state, then calls request.ReconcileTerminalState through reconcileRequest once for each member; the helper updates requests only and leaves the batch outcome unchanged.

Helper call path: In every After case above, each request terminal-state reconciliation is performed by request.ReconcileTerminalState. Request-scoped DLQs call it through reconcileRequest; batch and conclude DLQs call reconcileRequest once for each member request. Batch failure still uses BatchStore.UpdateState, while conclude DLQ preserves the batch's existing terminal state.

Normal DLQs still reconcile requests to Error and batches to Failed.

ReconcileTerminalState usage

DLQ call path Terminal outcome Purpose
Request-scoped DLQ through reconcileRequest Error Terminates the affected request and records available DLQ context.
Batch-scoped DLQ through failBatch Error for each member request Fails the batch and reconciles its request fanout.
Conclude DLQ through concludeBatch Batch Succeeded, Failed, or Cancelled maps to request Landed, Error, or Cancelled Repairs incomplete conclude fanout without changing the batch outcome.

Future callers should use this helper only after deciding that a request must enter a terminal state and needs the matching public request log. The caller owns the business decision and supplies State, LastError, and Metadata; the helper owns the CAS, version advancement, state-to-status mapping, idempotency, and log publication. Do not use it for non-terminal request transitions, batch transitions, or log-only events.

Test Plan

make lint && make check-tidy && make check-gazelle && make test

Stack

  1. refactor(orchestrator): centralize request termination #424
  2. @ fix(orchestrator): preserve conclude DLQ outcomes #426

Summary:
Intent:
- Keep DLQ reconciliation consistent with the terminal outcome already selected by conclude.
- Preserve useful failure context in terminal request logs.

Changes:
- Reuse the parent request termination helper across DLQ reconciliation.
- Preserve successful, failed, and cancelled batch outcomes in conclude DLQ fanout.
- Record original topic and failure metadata on DLQ terminal logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant