docs: cover gateway clustering and the proxy modes left untested

[kvinwang]

Follow-up to #1134. I tested a two-node gateway cluster after that merged; this adds what it turned up and corrects a claim the first version made about proxy coverage.

Gateway clustering (new section 6)

Two things about clustering are easy to get wrong and hard to diagnose.

Sync is already on in a single-node run. entrypoint.sh sets SYNC_ENABLED=$([ "$NODE_ID" -gt 0 ] && ...), so a node with NODE_ID=1 logs WaveKV: detected certificate changes and looks clustered while nothing has ever replicated. I had assumed the opposite — that sync was off in my single-node runs — and said so before checking the entrypoint. Neither reading is safe without looking, so the section says which it is and why the log line is not evidence.

Every node must deploy under the same --name. Cluster members require the peer's app_id to equal their own (kv/https_client.rs), which is deliberate — it confines WaveKV replication to instances of one authorized compose rather than letting any CVM join. app_id is the compose hash and name is part of it, so a second node deployed under a different name fails discovery with:

bootnode discovery retry failed: failed to fetch peers from bootnode
  app_id mismatch: expected c90cc75a6ceb…, got aa04b7bd7875…

That reads like a trust or connectivity fault and is neither. Per-node values (NODE_ID, WireGuard addressing, ports, MY_URL, BOOTNODE_URL) belong in --env-file, whose values are encrypted at deploy time and excluded from the hash — only the key names are recorded, as allowed_envs. Diffing the two compose files showed name as the only differing field, which is what made the cause obvious; the section suggests comparing the two hashes before deploying.

The section also lists what actually proves a cluster works, because agreeing peer lists are the weakest of the available signals. The one that matters is traffic entering node2 and reaching an app whose WireGuard tunnel terminates on node1 — that shows what replicated is usable routing state, not metadata that happens to match.

Correction to what is now section 7

The merged version listed "curl through the proxy to the app — end to end, including TLS termination and routing". That is true but invites the reading that the proxy is covered, and it is not.

Ingress maps as <id>[-[<port>][s|g]].<base_domain>, and the suffixes select different code paths: s is TLS passthrough through proxy/tls_passthough.rs with its own _dstack-app-address TXT resolution, g is HTTP/2. The 0.6.0-rc0 round exercised only the no-suffix path, single-node and then across a cluster, and only with a small request — no sustained transfer, long-lived connection or streaming. The table row now says so and a subsection spells out what remains untested.

One diagnostic note went in alongside it: a passthrough-resolution error while testing the terminated path usually means the gateway had no certificate and fell through to SNI routing. I spent a while treating that as a DNS problem when the missing certificate was upstream of it.

Housekeeping

Added the pgrep -f self-match trap, which cost more time than anything else in the run despite being the least interesting thing in it. A waiter looping on pgrep -f "build-image.sh ..." never exits while you check progress with a command whose own command line contains that string — checking kept it waiting, and an image that had been ready for half an hour sat unpushed. The same self-match reported a finished build as still running and, earlier, made a bare pgrep -af qemu-system match the grep itself and report a VM that did not exist. The advice is to check for the artifact rather than the process.

Also noted that a cluster run leaves a WireGuard interface per node plus a second set of ufw rules and ports, all of which outlive the CVMs.

Verification

prek run --files docs/rc-testing-runbook.md passes. Every command and error message quoted here came from the two-node cluster brought up on the TDX host, not from reading the code.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.