fix: fence DDL visibility across CN admission

[ck89119]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Fixes #27743

What this PR does / why we need it:

A successful DDL commit on one CN could be followed by a fresh connection on another CN whose catalog logtail had not applied that commit. This PR adds a generation-bound, fail-closed cross-CN DDL visibility fence.

Key contracts:

• MORPC v44 uniquely gates SyncCommitV2 and the distributed DDL activation cut; v43 remains the pre-activation baseline and carries the merged scalar-predicate runtime-filter terminal-state contract.
• HAKeeper keeps CN generation ownership monotonic independently of admission activation, and atomically validates exact (ServiceID, Generation, QueryAddress) membership and Prepared/Fenced proofs before committing the monotonic v44 epoch.
• Public DDL synchronously publishes commit T to the HAKeeper RSM before acknowledgement. The response must prove authoritative durable frontier >= T; an exact generation normally succeeds, while a replaced already-admitted producer revokes itself but still force-fans out the durably accepted T before returning an error. Legacy RSMs fail closed.
• The cluster-lifetime frontier survives CN restart, replacement, and store removal. Every activated/restarted CN applies it before publishing Fenced or opening ingress.
• Markerless startup publishes ingress through one atomic heartbeat and synchronously applies/validates the returned authoritative generation. A UUID takeover therefore revokes the stale process before it can open public DDL or ingress.
• Every direct CN heartbeat transfers returned schedule commands to the shared command dedupe owner. Final shutdown withdrawal occurs while TaskRunner and gossip dependencies are alive, so legacy destructive command delivery is applied before subsequent drains.
• Generation revocation synchronously seals admission while asynchronous drains wait for already-admitted DDL and stale-publication fan-out completion before stopping frontend or outbound dependencies; shared runner ownership avoids self-wait and normal-Close races.
• Downgrade below the committed v44 epoch is rejected.

Validation:

• deterministic generation takeover, replacement, membership drift, crash-before-periodic-heartbeat, legacy-HAKeeper frontier drop, destructive command delivery, restart/frontier synchronization, activation, downgrade, shutdown ordering, and TaskRunner ownership tests;
• full affected package tests for CN service, frontend, query client, HAKeeper/logservice protobuf and metadata, control path, and transaction client;
• focused race tests with repeated execution;
• relevant go vet and git diff --check;
• protobuf generated with make pb;
• production-path two-CN embedded test activates v44 through mo_ctl, proves SyncCommitV2 through fault injection, and verifies the peer's first fresh read sees CREATE without retry or SYNCCOMMIT.

Design

Approved revision 6: c630154888

[XuPeng-SH]

Deep re-review of exact head a4e9bd85664eb786f40beb7a2990fddd7ce2913a against base cf6bcf79edffa68c7bb09a1d39dbc2eeba6afa65, including the complete prior review history, the semantic increment after 6c788c437e82, the current design, mixed-version behavior, HAKeeper response ownership, CN generation takeover, frontend commit ordering, asynchronous revocation, performance, and Q1-Q3 lifecycle paths.

The previous crash-before-periodic-heartbeat blocker is closed for an all-current HAKeeper and the accepted CN generation: a successful public DDL now proposes T before acknowledgement, HAKeeper retains a monotonic cluster frontier, a stale generation is rejected, and the rejected TaskRunner path drains asynchronously without self-waiting.

Two P1 blockers remain in the new synchronous publication boundary:

1. [P1] A legacy HAKeeper can acknowledge the generation while silently dropping T. publishDDLCommitFrontier at pkg/cnservice/server_heartbeat.go:193-216 ignores its T argument after building the heartbeat and treats a matching ViewMetadataAdmission.Generation as proof of durable frontier publication. Current main/base already understands that generation field but does not contain DDLVisibilityFrontier; protobuf-compatible old HAKeeper RSMs therefore ignore the new heartbeat field and can still return the matching generation. Reachable mixed-version ordering: CN A is upgraded before every HAKeeper RSM, commits DDL T at the v42 baseline, an old HAKeeper leader accepts the heartbeat but drops T, the method returns nil and v42 skips SyncCommitV2, the client receives success, then A crashes before any upgraded RSM observes T. Later activation sees the old/empty global frontier and can reproduce the linked stale-CN failure. The activation-time LogStore capability check is too late for these pre-cut successful DDLs. Require an RSM response proof that the durable cluster frontier is at least the exact T, or fail public DDL closed before commit until every HAKeeper replica that can host the RSM advertises the frontier schema. Add the old-RSM/new-CN commit -> success attempt -> crash -> activation counterexample.

2. [P1] The frontier heartbeat discards destructively delivered schedule commands. SendCNHeartbeat is not a write-only API, but this path reads only batch.ViewMetadataAdmission and never transfers batch.Commands to handleHeartbeatResponse. Before acknowledged command delivery enters Preparing/Enabled, pkg/hakeeper/rsm.go:1022 calls legacy getCommandBatch, which deletes the pending command from replicated state before returning it. A mixed/fresh cluster with a pending CreateTaskService/JoinGossip command plus a public DDL can therefore have this non-periodic heartbeat consume the only command copy and drop it. The same response-ownership risk exists at the other direct SendCNHeartbeat sites added for startup, ingress, activation, and withdrawal. Fix this systemically: either every direct heartbeat must safely process/retain its CommandBatch under the existing dedupe owner, or frontier/admission publication must use an RSM operation that cannot take schedule-command ownership. Add a deterministic command-delivery-disabled regression.

No additional confirmed generation race, revocation self-deadlock, cleanup leak, or unbounded-growth blocker was found. The steady-state path adds one serialized HAKeeper consensus round plus sequential O(N) SyncCommitV2 fan-out per public DDL; DDL frequency makes that design plausible, but the documented 3-CN/larger-cluster p50/p95 rollout evidence should remain required. Exact-head substantive CI and git diff --check are green. Local focused runtime execution was blocked by the known missing exact-head macOS libmo.dylib, so no local test pass is claimed.

The PR body also still links approved design revision 7eb451dbed (revision 1), while the head document is revision 3 and materially adds synchronous durability, generation rejection, and asynchronous drain semantics. Update the body to the final design revision and re-review that revision with these ownership and mixed-version contracts represented.

[iamlinjunhong]

[P1] MORPCVersion43 is not a unique capability allocation. This PR assigns v43 to the DDL visibility activation and SyncCommitV2 contract at pkg/defines/const.go:81, but the currently open #27553 at head 889d92833751bc08ea1a9aa11eba107114e3b874 independently assigns v43 to its validated MongoDB explicit-query payload (same file/line).

A rolling deployment containing a #27553-only CN and a CN built from this PR therefore has both sides advertise version 43. This branch treats >= MORPCVersion43 as proof of the DDL sender/receiver capability (pkg/frontend/txn.go:1102-1105) and can send SyncCommitV2; it also treats v43 as the activation fence threshold (pkg/cnservice/server_query.go:248-260). The #27553-only CN has neither this command handler nor the durable activation-epoch semantics, so it can be admitted into the cut or receive an unsupported RPC, breaking the cross-CN DDL visibility guarantee.

Please coordinate a unique cumulative MORPC allocation: rebase on the merge-first owner and move every DDL version gate, generated protobuf contract, method map, test, and rollout documentation to the next unclaimed version. The previous reservation of v41/v42 is insufficient now that #27553 currently owns v43.

[XuPeng-SH]

Deep re-review of exact head 73787cd84066c595c599ee47f793d4c8065b58ea found one remaining P1 generation-takeover correctness hole.

The latest generation fencing closes the old-process admission path, but the fence is discovered after the DDL has already committed. A reachable ordering is:

1. CN A generation 1 has an already-admitted user DDL; generation 2 heartbeats and becomes authoritative for the same UUID before generation 1 sends its publication heartbeat.
2. Generation 1 passes the process-local DDL gate and commits the DDL to TN at pkg/frontend/txn.go:996-1008.
3. Its post-commit publication reaches updateCNViewMetadataAdmission, which returns before CNState.Update for the stale generation (pkg/hakeeper/view_metadata_admission.go:449-455). Therefore the monotonic cluster frontier at pkg/pb/logservice/logservice.go:178-180 is not advanced.
4. publishDDLCommitFrontier sees generation 2 and returns an error (pkg/cnservice/server_heartbeat.go:223-229). Because err != nil, frontend skips syncDDLCommitToBarrierReadyCNs at pkg/frontend/txn.go:1063-1069.

The result is not merely a safe client error: the DDL is already durable, while the authoritative frontier remains old and the other still-routable CNs are not synchronously advanced. They can serve a fresh connection with stale catalog metadata, recreating the class of failure this PR is intended to close. The existing exact-head tests independently encode both halves of this counterexample: TestStaleGenerationCannotAcknowledgeDDLFrontierPublication requires the stale heartbeat not to advance the frontier, and the publisher-failure commit test requires zero fan-out requests.

Please make generation replacement linearize with already-admitted commits. One viable invariant is that every successfully committed public DDL—also from a just-replaced incarnation—must durably advance the cluster frontier and complete the visibility fan-out before that commit path exits, while the stale incarnation remains forbidden from reopening ingress or admitting another DDL. Add a deterministic generation-1 Enter/commit -> generation-2 takeover -> generation-1 publication regression that asserts both authoritative frontier >= T and all eligible CNs applied T.

The previous legacy-HAKeeper proof and destructive schedule-command ownership blockers are materially fixed at this head. I found no additional confirmed P0/P1 leak, deadlock, unbounded-growth, or steady-state performance blocker; substantive exact-head CI and git diff --check are green.

[ck89119]

Protocol allocation declaration: PR #27756 uses MORPC v44 for the cross-CN DDL visibility fence and SyncCommitV2 contract. The branch has been merged with the latest main; v43 remains owned by the already-merged scalar-predicate runtime-filter terminal-state contract. All #27756 runtime gates, QueryService method mapping, activation paths, tests, and rollout design now consistently use v44.

[XuPeng-SH]

Deep re-review of exact head c630154888d21aaf81cdb9ef77fada8e37c308db against base 0db2d5ea9a39d281914275fa83108029fe30bfba, including the complete prior review history, the semantic increment after 73787cd84066, the revised generation ownership and revocation lifecycle, frontend commit ordering, HAKeeper state, protocol compatibility, performance, and Q1-Q3 unhappy paths.

The previous generation-takeover blocker is materially closed on its normal, known-commit path. HAKeeper now monotonically absorbs T from the stale generation without accepting incarnation-scoped state; the old CN seals admission synchronously, keeps frontend/QueryService alive, force-fans out T, and only then releases asynchronous drain. The focused HAKeeper/CN/frontend race regressions each passed 10 runs.

One systemic P1 correctness blocker remains:

1. Post-commit failure is not actually fail-closed, and ErrTxnUnknown bypasses the visibility fence entirely. TxnHandler.commitUnsafe irreversibly commits at pkg/frontend/txn.go:996-1008, then performs both HAKeeper publication and CN fan-out with the same request-derived ctx2 at lines 1050 and 1078. If the client context is cancelled, HAKeeper times out/loses its response, or a still-required CN fails after TN committed, the code only returns an error and releases DDLCommitGate; it cannot roll the DDL back, does not durably prove T, and leaves other CNs routable. The existing publication failure prevents successful acknowledgement test actually codifies this state: the transaction has committed, publication returns an error, and no fan-out occurs.

   The ErrTxnUnknown branch is an even stronger reachable counterexample. The txn operator records commitNeedsResolution precisely because TN may already have committed, but no CommitTS is available. Therefore the !commitTs.IsEmpty() condition at line 1042 skips publication, and the unconditional early return at lines 1064-1070 skips fan-out and disconnects DDL visibility from the existing unknown-commit resolver. A retry on another CN can consequently observe no such table or inconsistent catalog state even though the original DDL later resolves committed.

   Please close the invariant at the irreversible commit boundary, not merely the SQL acknowledgement boundary. Acceptable designs include a lifecycle-owned, cancellation-independent bounded completion/retry that obtains authoritative commit outcome/T and durably publishes plus fans it out, or an equivalent admission fence that remains closed until uncertainty is resolved. Returning an error after commit is not itself fail-closed. Add deterministic regressions for (a) commit T succeeds, request cancellation/HAKeeper response loss occurs before publication, then CN B's first fresh read, and (b) DDL commit resolves from ErrTxnUnknown to committed, then CN B's first fresh read. The design failure matrix and PR body must describe the resulting availability tradeoff.

Performance/lifecycle notes: the new completion barrier closes the prior TaskRunner self-wait and I found no new double-stop or unbounded-memory path. However, steady-state fan-out is sequential (SendMessage inside the ordered target loop), so latency is the sum of per-CN waits rather than the design document's apparent slowest-CN bound. This is not a separate correctness blocker for a small CN set, but the promised N-CN p50/p95 evidence is still needed before claiming the stated performance behavior; otherwise use bounded parallelism or document the actual bound.

Cross-PR protocol audit: current main ends at MORPC v43, while both this PR and open #28002 currently assign independent semantics to v44. #27756 declared/used v44 first, so I am treating it as the merge-first owner and will require #28002 to move to a fresh version. If merge order changes, this PR must instead rebase and reallocate every v44 gate/contract.

Validation at exact head:

• substantive GitHub CI: green
• git diff --check: pass
• go test -race ./pkg/hakeeper -run 'Test(ViewMetadataAdmissionRejectsStaleGeneration|CNGenerationCannotRollbackBeforeAdmissionActivation|ViewMetadataAdmissionReplacementKeepsOldGenerationUntilTimeout)$' -count=10: pass
• go test -race ./pkg/cnservice -run 'Test(StaleGenerationCannotAcknowledgeDDLFrontierPublication|MarkerlessCNIngressRejectsGenerationTakeover)$' -count=10: pass
• go test -race ./pkg/frontend -run '^TestCommitSyncsDDLCommitToBarrierReadyCNs$' -count=10: pass

[iamlinjunhong]

Reviewed exact head c630154 against freshly synchronized origin/main 20b43ed (merge-base 0db2d5e). I re-read the complete change set and the increment since my prior review, inspected the CN startup/activation/restart/replacement/heartbeat/admission/shutdown lifecycle, DDL commit gate and fan-out, HAKeeper epoch/frontier ownership, generated protobuf and protocol compatibility, changed tests, prior review threads, author replies, and PR conversation. The author explicitly replied to my prior P1 protocol-allocation finding and the current implementation consistently uses the uniquely allocated MORPC v44 for the DDL visibility deployment gate. I found no remaining confirmed P0/P1/P2/P3 issue. Current CI checks are terminal with no failure.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →