tower: HOTFIX — echo-verify re-write loop re-injects a delivered message unboundedly (3.3.2 regression from #1573; zero re-writes after a completed write)

[waleedkadous]

Problem (3.3.2 regression — hotfix)

#1583 (shannon, 2026-09-02): one afx send --file message was re-injected into a builder's prompt dozens of times, byte-identical; a separate architect→architect message was delivered twice. Both on 3.3.2. Root cause confirmed at current main:

• mailbox-delivery.ts:694: after the gated write has completed (every byte + Enter accepted by the PTY), a failed echo-verification returns hold('busy').
• The next clean-prompt pass re-runs the row with a full re-write. No per-row attempt cap exists anywhere in the module.
• busy holds are deliberately excluded from escalation streaks (isClassifierStuck), so the loop is silent to the owner.
• Field trigger is the common case, not the agy corner tower: bound redelivery when echo verification never confirms (#1573 residual) #1578 predicted: a recipient that starts responding immediately scrolls the header out of the sampled screen (or a long write evicts the pre-write copy from the mirror), so the occurrence count does not increase, verification fails, the message is re-injected, the recipient responds again — self-sustaining. afx interrupt worsens it (each interrupt = fresh clean prompt = another pass).

Verify-then-rewrite was the dangerous half of #1573. The honest half (settle-before-write, 48KB cap, "tell the sender when we could not verify") stays.

Fix (prescribed — owner-approved design 2026-09-02: ZERO re-writes)

A row whose write completed is at-least-once delivered and must never be written again.

1. At the verify site (mailbox-delivery.ts ~694): on verification failure after a completed write, do not hold. Allow one bounded additional verify window (no bytes written — e.g. one more watch-style sample after a short delay, ≤ ~1.5s total) to accommodate slow renderers; if still unconfirmed, mark the row delivered and:
   • log WARN (delivered-unverified, row id, to_agent, terminal id, header needle length);
   • set the row's escalated flag (existing column) so the dashboard/SSE surface it;
   • return a result that carries verified: false.
2. Thread verified (boolean) additively onto the immediate-delivery send response (tower-routes.ts handleSend, sdk/tower-client.ts typing, commands/send.ts prints e.g. [ok] Message delivered (unverified — header not seen on the terminal)). Additive field; older clients unaffected.
3. Remove/neutralize any path by which a completed-write row can be re-written: audit every hold(...) reachable after submitMessagePaced resolves successfully; only pre-write prechecks may hold.
4. isClassifierStuck / recordStreak: no change needed once (1) lands (no busy-hold after a completed write exists anymore) — confirm with a test rather than assume.

Out of scope: smarter verification (scrolled-header tolerance), attempt columns/migrations, #1567 truncation class, agy measurement. Those remain under #1578 (retitle it to "smarter verification" follow-up, keep open).

Testing

• Control test (must fail on current main, pass after): fake session whose mirror never shows the header → the message is written exactly once; the row ends delivered + escalated, never held; a second drainer tick writes nothing.
• Immediate-responder simulation: mirror shows the header, then scrolls it out before verification samples → written once, delivered (verified or unverified, but never re-written).
• Verified path unchanged: header count increases → delivered, verified: true, not escalated.
• Send response / CLI: verified:false surfaces as the unverified wording; verified:true/absent prints today's line.
• Keep the tower: delivery write edge — verify-or-retry 'delivered', settle-before-write, loud size limit (silent-loss residuals of #1564/#1521) #1573 and Serializer convergence: route mailbox write edge through submitToSession (serialize gated deliveries vs interrupt/escape) #1365 suites green; update any tower: delivery write edge — verify-or-retry 'delivered', settle-before-write, loud size limit (silent-loss residuals of #1564/#1521) #1573 test that asserted the hold-and-redeliver behavior to assert the new contract instead of deleting it.
• Field verification before close (architect-driven, post-release): long --file send to a builder that responds immediately → exactly one copy in its transcript.

Release

Hotfix release 3.3.3 to @latest immediately on merge. Note in the PR that this supersedes the residual recorded in #1578.

Refs: #1583 (field report), #1578 (residual this replaces), #1573 / PR #1577 (regression source), #1564/#1521 (the verification's original purpose).

[waleedkadous]

On it! Working on a fix now.

[waleedkadous]

Completion statistics

Fixed by PR #1585, merged 2026-09-02 16:13Z (merge commit 8813051, no squash).

Protocol	BUGFIX (strict / porch-driven)
Wall clock	~2h25m, investigate → merge
Commits	7 substantive + 6 porch state
Net source diff	266 lines (228 added / 38 deleted) — inside the <300 LOC BUGFIX ceiling
Test diff	488 added / 21 deleted
Files touched	6 source, 4 test
CMAP	2 rounds, 6 invocations, 14.0 min, $4.72
CI	7/7 green
Local suite	5377 passed, 0 failed, 48 skipped (273 files)

Phase timeline

Phase	Span	Notes
Investigate	06:58–07:05	Root cause traced in-code; no repro script needed — the #1573 suite already pinned the re-write as intended behaviour (writes == [FORMATTED, FORMATTED])
Fix	07:05–07:16	Fix + regression suite + the verified-scoping tightening
PR + CMAP r1	07:16–07:24	gemini=APPROVE, codex=REQUEST_CHANGES (2), claude=COMMENT (4)
CMAP r2	07:24–07:33	After substantial changes: gemini=APPROVE, codex=COMMENT (2), claude=APPROVE (3 non-blocking)
Gate → merge	07:33–16:13	Awaiting human approval

What CMAP caught that solo review did not

Nothing was rebutted across either round — every finding from both non-approving lanes was real, which is the whole argument for not skipping the pass.

The load-bearing one: my first implementation verified before committing the row, leaving it held for ~1.2s. Codex identified that this is not durable — a Tower crash in that window, or an echo.verify() that rejects rather than answering, leaves the row held and re-writable, reinstating the very loop. My own audit of issue item 3 had grepped for hold(...) call sites and never considered the throw path. Reordering markDelivered ahead of verification is what actually closes the issue; without that round the PR would have shipped a partial fix that looked complete and passed its own tests.

Claude caught three doc blocks left asserting the inverted safety property (watchEcho, EchoWatch.verify, watchEchoOnScreen all still promised "hold for redelivery — a duplicate rather than a silent loss", when those residuals are precisely the #1583 trigger).

Regression coverage

bugfix-1584-no-rewrite-after-write.test.ts, 12 tests. 6 of the original 7 fail against the unmodified module at HEAD; 3 of the 5 added after review fail against the pre-review commit — both verified by swapping the module back in and re-running, not asserted. The two #1573 tests that pinned hold-and-redeliver were rewritten to the new contract rather than deleted.

Process notes

• The worktree arrived with no node_modules and no packages/codev/skeleton/; 12 test files fail environmentally until pnpm install + pnpm build. Identical per-file failure counts confirmed at HEAD before building, so none were caused by this change.
• consult did not auto-detect the project from builder context — it listed every project and exited 0 without reviewing. --project-id bugfix-1584 was required. Worth an issue if it recurs.

Follow-ups

• Tower/afx: one sent message re-injected into a builder prompt dozens of times (byte-identical re-delivery loop), plus same-day truncation and stuck-composer incidents #1583 stays open for post-release field verification (long --file send to a builder that responds immediately → exactly one copy in its transcript).
• tower: bound redelivery when echo verification never confirms (#1573 residual) #1578 stays open for smarter verification; three non-blocking residuals from Claude's round-2 review recorded there as a comment.
• Hotfix release 3.3.3 → @latest is the owner's call.