fix(gateway): register the shared ACME account under the rotation lock

[kvinwang]

Problem

A cluster registers its shared ACME account lazily, on whichever renewal first
finds the credentials record empty:

let stored_creds = self.kv_store.get_acme_credentials()?;
if let Some(creds) = stored_creds { /* ... load and return ... */ }

// Create new global ACME account
let client = AcmeClient::new_account(acme_url, dns01_client, ...).await?;
self.kv_store.save_acme_credentials(&CertCredentials { acme_credentials: creds_json.clone() })?;
if let Some(account_uri) = extract_account_uri(&creds_json) {
    self.generate_and_save_acme_attestation(&account_uri).await?;
}

Nothing serializes that. The only lock held here is try_acquire_cert_lock,
which is per domain, so it does not order two domains against each other at
all:

• one node, two domains — the periodic renewal task and an admin
  RenewZtDomainCert run on separate tasks and take separate locks
• several nodes — each takes the lock for a different domain and proceeds

And the empty-record state is exactly what a fresh cluster starts in: every
node runs init_all at boot, so the first issuance of each domain races the
others.

Both registrations succeed at the CA. save_acme_credentials is a WaveKV write,
last-writer-wins with no compare-and-swap (try_acquire_rotation_lock's own
docs say so), so the record keeps one account and the other is lost. The cost:

1. Every loser spends a registration the CA rate-limits — Let's Encrypt allows
   10 new accounts per IP per 3 hours.
2. generate_and_save_acme_attestation writes under its own key and races
   separately, so the surviving attestation can be the other account's. The
   cluster then issues with an account it cannot prove it holds, which is
   visible to anyone verifying the attestation and not visible locally at all.
3. Issuance itself converges — the next renewal reads the winner — but a
   SetCaa run in between pins CAA to the loser, and issuance stays broken
   until SetCaa is rerun.

Fix

Registration takes the lock that already exists for this key, the one
RotateAcmeCredentials uses, and re-reads the record under it:

let _serialized = self.caa_lock.lock().await;
let Some(rotation_lock) = self.try_acquire_rotation_lock() else {
    bail!("another node is registering or rotating the ACME account; retry after it finishes");
};
let client = self.register_or_adopt_account(domain, &dns_cred, acme_url).await;

The re-read is the point: a waiter let through after the holder finishes adopts
the account that appeared rather than registering a second one. Lock order
matches rotation's (caa_lock then the KV lock), so the two cannot deadlock,
and the lock is released whether registration succeeded or failed.

The DNS provider client is now built after the lock is granted rather than in
the shared prologue. Constructing it resolves the zone through an authenticated
provider API call, and a run that is about to be refused should not spend one.
The fast path — credentials already present — is unchanged, and still builds it.

This narrows the window to WaveKV's replication latency; it does not close it.
The KV lock is last-writer-wins without compare-and-swap, so two nodes can still
both acquire during a replication gap. That is the same guarantee rotation has
had, and closing it properly means compare-and-swap in WaveKV, which is a
different change.

A node refused by the lock returns an error from that renewal. try_renew_all
logs it and continues, and the periodic task retries on its next pass, by which
time the account exists and is adopted.

Verification

cargo test -p dstack-gateway — 290 pass, including a new hermetic regression
test: with the rotation lock held, a first-use registration fails with
registering or rotating rather than proceeding. The test configures a DNS
credential pointed at an unreachable provider (http://127.0.0.1:1), so
reaching the lock error at all is the assertion — before this change the same
run failed while building the DNS client, which happened first.

cargo fmt and cargo clippy -p dstack-gateway --all-targets -D warnings are
clean.

Scope

Pre-existing on next; found while reviewing the dns-persist-01 work in
#1132, where the consequence is sharper — a _validation-persist record names
one account, so a silently replaced account fails every order until an operator
republishes. #1132 touches the same function and will rebase onto this.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.