feat(certbot): issue certificates via dns-persist-01 - #1132
Open
kvinwang wants to merge 23 commits into
Open
Conversation
kvinwang
changed the title
dns-persist-01
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 25, 2026 11:07
aba2699 to
9c9da5e
Compare
dns-01 rewrites the zone on every order, so certbot holds a DNS provider credential for the life of the deployment. dns-persist-01 (draft-ietf-acme-dns-persist-01) proves control with a `_validation-persist` TXT record published once, out of band: the record names the CA and the ACME account, nothing about it changes between orders, and certbot only ever reads DNS. `ValidationMethod` replaces the `Dns01Client` field on `AcmeClient`, so the dns-01-only state -- the provider client and the TXT TTL -- lives in the one variant that has any use for it, and dns-persist-01 cannot be constructed holding a credential it would never call. `check_dns` widens from an exact match on the key authorization to matching whatever the challenge expects, and the cleanup pass skips records certbot did not create: a persistent record is the operator's and outlives every order. The self-check stays advisory for both methods. Our resolver is not the CA's, and under dns-persist-01 our expectation can be stricter than the CA's -- the challenge's `issuer-domain-names` are not exposed by instant-acme -- so a record we cannot see is named in a warning and the order proceeds. CAA content now names the challenge in use, because a record pinned to `validationmethods=dns-01` refuses every dns-persist-01 order. The dns-01 string is unchanged byte for byte, pinned by a test, so records published by earlier releases keep matching. `required_dns_records` renders the whole one-time setup -- validation record plus CAA -- from an account URI rather than a live client, so a caller holding only stored credentials can render it without a round trip to the CA. The record grammar is an RFC 8659 issue-value, and the parser mirrors what CAs run down to the parts that reject rather than ignore (trailing semicolon, repeated tag, whitespace in a value), so certbot never renders a record the CA would refuse or accepts one it would. The gateway keeps its dns-01 behaviour; the call sites move to the new constructor unchanged.
Under dns-persist-01 certbot cannot write the records it needs, so the records are the setup. `certbot dns-records` prints them as zone-file lines for the configured domains, ready to paste into any provider, once `certbot init` has registered the account they name. `challenge` and `issuer_domain_name` join certbot.toml, and `cf_api_token` becomes optional -- a token left configured alongside dns-persist-01 is warned about rather than silently ignored. `auto_set_caa` is refused outright with dns-persist-01: it promises certbot keeps CAA in sync, and without write access nothing can keep that promise.
A gateway CVM running dns-01 holds a Cloudflare token with write access to the operator's whole zone. Attestation covers what the CVM runs, not what becomes of a secret it holds, so that token is the widest credential in the deployment and it exists only to write one TXT record per order. dns-persist-01 removes it: control comes from a `_validation-persist` record the operator publishes once, and the CVM never gets DNS write access at all. `ZtDomainConfig.challenge` picks the method per domain and defaults to dns-01, so records written before the field existed decode as the method those deployments were using -- pinned by a test over both the named and the legacy positional msgpack encodings. Such a domain needs no DNS credential, and `validation_for` never looks one up for it. `GetZtDomain` and `ListZtDomains` return the records to publish in `required_dns_records`, rendered from the stored account URI with no ACME round trip so the listing endpoints stay cheap; it comes back empty rather than failing when no account exists yet. Two operations cannot be self-service for such a domain, and say so rather than failing silently: - `SetCaa` skips it and logs the records instead. There is nothing to reconcile without write access, and one such domain must not make the RPC unusable for the dns-01 domains beside it; the summary reports how many were left to the operator. - `RotateAcmeCredentials` moves the cluster to a new account while every `_validation-persist` record still names the old one, so orders for those domains fail until the operator republishes. The response now carries the new records in `required_dns_records`, rendered after the switch so they name the account the cluster actually moved to, and `domains_updated` counts only the domains whose CAA was re-pinned.
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 25, 2026 12:11
9c9da5e to
f2b1e3c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every dstack certificate today is issued with
dns-01, which requires a fresh_acme-challengeTXT record per order. That means whatever runs certbot holds aCloudflare API token with write access to the operator's zone, permanently —
and in the gateway's case that token lives inside a CVM:
Attestation covers what the CVM is running. It does not cover what becomes of a
long-lived secret the CVM holds. So the widest credential in a dstack deployment
— one that can rewrite
MX,A, anything — exists solely to write one TXTrecord per order and delete it again. It also forces the base domain onto a
provider certbot has an integration for; today that list is exactly
{Cloudflare}.Fix
Add
dns-persist-01(draft-ietf-acme-dns-persist-01, CA/Browser Forumballot SC-088v3) as an opt-in validation method. The zone owner publishes one
record naming the CA and the ACME account:
The account key proves who is asking, the record proves the zone owner agreed,
and nothing about it changes between orders. certbot only ever reads DNS, so the
CVM holds no DNS credential at all and the zone can be hosted anywhere.
Shape of the change
ValidationMethodreplaces theDns01Clientfield onAcmeClient, so thedns-01-only state — the provider client and the TXT TTL — lives in the one
variant that uses it, and
dns-persist-01cannot be constructed holding acredential it would never call:
Opt in per deployment (
challenge = "dns-persist-01"incertbot.toml) or pergateway ZT domain (
ZtDomainConfig.challenge). The default staysdns-01everywhere and existing deployments are untouched; a stored
ZtDomainConfigwritten before the field existed decodes as
dns-01, pinned by a test over boththe named and the legacy positional msgpack encodings.
Because certbot cannot write the records under this method, the records are
the setup, and they are surfaced everywhere an operator would look:
certbot dns-recordsprints them,GetZtDomain/ListZtDomainsreturn them inrequired_dns_records, and the gateway logs them wherever it would otherwisehave written DNS.
Details worth a reviewer's attention
CAA has to move with the method. The value was hardcoded to
validationmethods=dns-01; a record left pinned to that refuses everydns-persist-01 order (and vice versa). It now names the method in use. The
dns-01 string is byte-for-byte unchanged, pinned by a test, so records published
by earlier releases keep matching.
The self-check stays advisory. Our resolver is not the CA's, and under
dns-persist-01 our expectation can be stricter than the CA's —
instant-acmedoes not expose the challenge's
issuer-domain-names, so we compare against theconfigured one. A record we cannot see is named in a warning and the order
proceeds; the check can never fail an issuance that would have worked.
Two gateway operations cannot be self-service, and say so instead of failing
silently:
SetCaaskips such a domain and logs the records. One dns-persist-01 domainmust not make the RPC unusable for the dns-01 domains beside it.
RotateAcmeCredentialsis the sharp edge: it moves the cluster to a newaccount while every
_validation-persistrecord still names the old one, soorders for those domains fail until the operator republishes. The response now
carries the new records in
required_dns_records, rendered after the switchso they name the account the cluster actually moved to, and
domains_updatedcounts only domains whose CAA was re-pinned.
The record grammar rejects rather than ignores. It is an RFC 8659
issue-value, and a CA fails the whole record on a trailing semicolon, arepeated tag, or whitespace inside a value. The parser here mirrors the one CAs
run (Boulder's
va/dns_persist.go), so certbot never renders a record the CAwould refuse nor accepts one it would.
Verification
End-to-end against Let's Encrypt staging, with a real Cloudflare-hosted
zone (
kvin.wang) and no DNS credential in the certbot config at all -- thethree records were published by hand from what
dns-recordsprinted:The single record answered both authorizations at the real CA, which is the
claim this design rests on: the debug log shows the base-name authorization
expecting
letsencrypt.org; accounturi=...and the wildcard one expecting thesame with
; policy=wildcard, both satisfied by the one published record.After issuance and a forced renewal, the zone holds exactly the three records
that were published by hand -- no
_acme-challenge, nothing added, nothingremoved. certbot never wrote DNS, which is the point of the mode.
Failure path at the real CA. A second account (fresh workdir, so a fresh
accounturi) against the same published record:Staging's rejection is more specific than Pebble's, so the doc now quotes this
one; it also notes that a genuinely mismatched record costs the full
max_dns_waitbefore the order is sent, since the advisory check waits out itsbudget first.
End-to-end against Pebble v2.10.1, which implements the draft
(
va.validateDNSPersist01,_validation-persist,policy=wildcard,persistUntil), withpebble-challtestsrvas the zone. Full flow, no DNScredential anywhere in the config:
One record covers the bare name and the wildcard — which is the behaviour the
CA implements, not an assumption: a wildcard authorization strips the
*.andlooks up the base name.
Failure paths, same harness. With a record naming a different account, the
warning names exactly what was expected and the CA rejects the order:
auto_set_caa = true,set-caa, and a leftovercf_api_tokeneach producetheir own message rather than a silent skip:
dns-01 regression, live. The same binary, same Pebble, against the repo's
mock-cf-dnsCloudflare API fromtest-suites/full-stack-compose— a wildcardcertificate still issues through the provider-API path:
Unit tests. 20 in
dns_persistcovering the grammar against the draft's ownexample and each rule a CA enforces (case-insensitive tags, byte-exact
accounturi, wildcard policy in both directions, trailing semicolon, duplicatetag, whitespace in a value,
persistUntilexpiry, several records at onelabel); 6 in
acme_clientpinning the rendered records, including thebyte-identical dns-01 CAA value; 1 in
gateway/src/kvpinning theZtDomainConfigdecode.Rebased onto
nextafter #1129 (theinstant-acmeupgrade this needed),#1130 (authoritative-nameserver DNS checks) and #1133 (the ACME follow-ups)
landed. Both merges kept the incoming structure whole:
check_dnsresolves each challenge's authoritative nameservers andfalls back to the system resolver. Only the "does this answer match" step
widens, from an exact comparison to whatever the challenge expects. Its zone
walk now drops any leading underscore label rather than
_acme-challengespecifically, so
_validation-persisttakes the same path instead of burninga round trip on a name that cannot carry NS records.
challenge_domainhelper — which exists becauseAuthorizedIdentifier'sDisplayrenders the wildcard prefix and wouldpublish at
_acme-challenge.*.example.com— is generalised over the challengekind rather than duplicated. That trap is identical for
_validation-persist,so its test now asserts both methods keep the bare name.
Every e2e run below is on the rebased tree.
dns-01 regression, live. The same binary, same Pebble, against the repo's
mock-cf-dnsCloudflare API fromtest-suites/full-stack-compose— a wildcardcertificate still issues through the provider-API path:
Build/lint.
cargo fmt --all --check,cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables(the CIinvocation),
cargo clippy -p certbot -p certbot-cli -p dstack-gateway --all-targets -- -D warnings,cargo test -p certbot -p dstack-gateway(31 + 290 passing),
prek run --all-files, andreuse lintall pass.Not covered: a live run against Let's Encrypt staging. It offers the
challenge today, but with
accounturiomitted from the challenge object(Boulder's
AccountURIPrefixis unset there), so the account URI comes from ourown
kideither way — which is what this implementation does, sinceinstant-acmeexposes neither field. Pebble is the closer match to the currentdraft.
Review follow-ups
Five findings from review, each verified against the CA's own implementation
rather than taken on description:
The self-check read records more strictly than Boulder does. RFC 8659's
grammar is
parameter = tag *WSP "=" *WSP valueand Boulder trims each halfseparately (
va/dns_persist.go); this trimmed the parameter as a whole, soaccounturi = <uri>parsed its tag as"accounturi "and reported themandatory parameter missing. Boulder also enforces uniqueness only on tags it
recognizes, since the draft has it ignore unknown ones outright, and folds the
issuer name (lowercase, root dot dropped) before deciding whether a record is
its own. All three now match, with the empty-
accounturicase Boulder rejectsadded too.
The cost was not theoretical. Against Let's Encrypt staging, with the published
record padded around its separators -- legal, and issued from -- before and
after:
Issuance succeeded either way -- the check is advisory -- but every renewal
burned the full wait and logged two warnings naming a record that was in fact
correct.
The gateway's
issuer_domain_namecould not be set. It was read by thedns-persist-01 path and documented as the knob for a non-Let's-Encrypt ACME
server, but no proto field, RPC or config path ever wrote it, so it was
permanently empty and the gateway was hardwired to
letsencrypt.org-- againstPebble, every order would have failed with no way to correct it. It is now a
field on
SetCertbotConfigRequest/CertbotConfigResponseand in thedashboard's Certbot Configuration.
A dashboard edit silently downgraded a domain to dns-01.
UpdateZtDomainreplaces the whole record and an empty
challengedecodes asdns-01, but theedit form never round-tripped the field: changing a priority would have moved a
dns-persist-01 domain onto the dns-01 branch, which then fails on the DNS
credential such a deployment deliberately does not have. The form now carries
the challenge forward, the add form offers it (so the UI can create these
domains at all), and the listing shows which challenge a domain uses instead of
naming a credential a dns-persist-01 domain never has.
The advisory wait could not stay advisory.
DNS_PERSIST_MAX_DNS_WAITwas300s, the same as the default
renew_timeoutthat wraps the whole order, andthe wait is measured only after the order and its authorizations are fetched --
so the outer timeout always fired first, aborting with "certificate request
timed out" instead of proceeding to the CA, and the warning an operator is told
to look for was never logged. Fixed for both challenges in the round below, by
deriving the wait from
renew_timeoutinstead of picking a constant.issuer_domain_namewas ignored on dns-01. The key documents itself asnaming the CA "in dns-persist-01 and CAA records", but dns-01 hardcoded
letsencrypt.org, so an operator running dns-01 against another CA withauto_set_caaon would have published CAA forbidding the very CA in use. Bothmethods now read the configured name, whose default is
letsencrypt.org, so anuntouched configuration writes exactly what it wrote before.
Bootstrapping without a domain
RotateAcmeCredentialsused to refuse to run until a ZT domain existed, whichput a fresh
dns-persist-01deployment in a loop: the record an operator has topublish names the ACME account, the gateway only registered one lazily on its
first issuance attempt, and that attempt could not succeed before the record was
published. The way out was to trigger a renewal, let it fail, and read the
records off the account the failure had registered.
Registration asserts nothing about a domain -- it is a POST to
newAccount,with no DNS access and no challenge involved -- so the requirement was an
artifact of
AcmeClient::new_accounttaking aValidationMethodin order toreturn a client.
AcmeClient::register_accountnow returns just the account(
AcmeAccount { credentials, account_uri }),new_accountis that plus aload, and rotation registers before it looks at any domain. A cluster with nodomain registers and stops there; one with domains re-pins their CAA exactly as
before, and the "first domain reuses the registration client" special case is
gone with it -- every domain now takes the same path.
The setup order in the doc is correspondingly straight:
SetCertbotConfig→RotateAcmeCredentials→AddZtDomain→ publish the records it reports →RenewZtDomainCert.Verified against Let's Encrypt staging through the CLI, which registers over the
same path: a fresh workdir registers
acct/329645064anddns-recordsimmediately renders the record naming it, while the existing account still
renews (
DNS:*.persist01.kvin.wang, DNS:persist01.kvin.wang), so theregister-then-loadsplit did not disturb the account it replaces. Thegateway's own no-domain path is covered by a unit test pointed at an unroutable
directory: it fails at the ACME server rather than on a missing domain, which is
what proves the requirement is gone.
Second review round
Two reviews came back on the fixes above; between them they found that one of
those fixes was half a fix, and that another had traded a dormant bug for a
sharper one. Both were right.
The DNS wait now derives from the order budget instead of being a constant.
120sfixed thedns-persist-01arm and leftdns-01— the arm nearly everydeployment uses — with the same 300s-inside-300s collision, and it was a
hardcoded assumption about
renew_timeout, which an operator can lower from thedashboard to 30s. It is now one rule for both:
advisory_dns_wait(configured, renew_timeout)caps the wait at half the budget, so the check always reachesits "proceed anyway" exit and the warning naming the missing record.
The CLI has the same shape and worse defaults —
renew_timeout120s wrapping a300s
max_dns_wait— which neither review looked at, so the clamp lives in thecertbotcrate and both callers use it. Against Let's Encrypt staging, sameconfig (
renew_timeout = 120,max_dns_wait = 300), same missing record:Before, the operator gets
requesting cert timeoutand no idea why. After, theyget the record that was missing and the CA's own verdict.
issuer_domain_nameis validated where it is set. Making it reachdns-01CAA turned an inert config key into a load-bearing one, and nothing checked it:
set_caa_recordsinstalls a;guard, deletes the existing issue/issuewildrecords, then writes the new content — so
"lets encrypt.org", or an explicit""on the CLI path (which passed the empty string straight through where thegateway mapped it to
letsencrypt.org), would leave the zone holding amalformed or empty-issuer
issueproperty and nothing valid behind it. That isCAA denying every issuer, published by certbot itself.
resolve_issuer_domain_namenow does both jobs for every caller: empty meansletsencrypt.org, and a value that is not a DNS name is refused —SetCertbotConfigrejects it, andcertbotrefuses to start with it.The corrupt-record repair path can no longer forget it.
merge_certbot_configrequired four fields to replace an unreadable record and then started from the
defaults, so a deployment on Pebble that repaired its config silently reverted to
letsencrypt.org— the exact hazardacme_urlis in that required set for.issuer_domain_namejoins it.renew_timeout_secs = 0is refused too, since theDNS wait is derived from it.
The dashboard no longer hides the setup.
showAddZtDomainModal()blankedevery field except the challenge select, so a second add silently inherited the
first one's choice; it resets now. And
AddZtDomainreturnsrequired_dns_recordswhich the page discarded — an operator pickingdns-persist-01got "ZT-Domain added" and no hint that nothing would be issueduntil they published a record. The records are now shown after the add, and
again behind a Records button on each
dns-persist-01row, since rotationinvalidates them.
Minor: the KV field's doc comment still said the setting was read only by
dns-persist-01, andset_certbot_config's audit log listed every field exceptthe one just changed. Both fixed. The setup-order doc also lost the
issuer-name guidance when it was rewritten in
4b47a448; it is back, with thenew rules.
cargo test -p certbot -p dstack-gateway: 43 + 296.Third review round
The round above introduced one regression and left three edges; all four are
closed here, verified against Pebble.
The records dialog fired for
dns-01domains, withdns-persist-01copy.Mine, from
f162c07d5b.required_dns_recordsis non-empty fordns-01too --required_dns_recordsskips the TXT record but still returns the two CAA lines-- so adding an ordinary
dns-01domain popped a dialog saying the gateway"holds no DNS credential for it and cannot write these itself", every clause of
which is false there, ending with an instruction to hand-publish records the
gateway writes itself. Gated on the challenge, like the
Recordsbutton in thesame commit already was.
The dashboard now has the button its own message points at. The empty-records
path tells an operator to register an ACME account first, and the documented
setup order makes that step 2 -- but
RotateAcmeCredentialshad no controlanywhere in the dashboard, so a dashboard-only operator hit a dead end. Added to
the Certbot Configuration section, behind a confirmation that says what rotation
costs, and it shows the records the response returns.
new_accountcould drop an account it had just registered. Splittingregister_accountout leftnew_accountas register-then-load, andloadfetches the ACME directory -- so a transient failure in that window returned
ErrafternewAccounthad succeeded, losing credentials neither caller hadpersisted yet and spending one of Let's Encrypt's 10 registrations per IP per 3
hours. The two halves are now taken from one call:
registerreturns the liveaccount and the credentials together,
new_accountbuilds the client from whatit already holds, and
register_accountencodes the same credentials forcallers that want only those. No second request, no window.
The DNS wait overshot the budget it had just been clamped to.
check_dnsslept then checked its deadline, so it overran by up to a full backoff step
(32s). The clamp reserves half of
renew_timeout, which the overshoot could eatwhole: at
renew_timeout = 64the wait is 32s, but the sleep checkpoints landat 31.75s, so the loop slept again and reached its exit at ~63.8s -- past the
order's budget. On Pebble, same config, same missing record:
The deadline is now checked before sleeping and the sleep is cut to what is left
of it, so the exit lands on the cap rather than past it and the order still has
its budget. The clamp also logs when it lowers a configured value, which it
previously did silently -- an operator raising
max_dns_waitand seeing nochange had nothing to read.
The issuer name validator was looser than the grammar it protects. It
accepted
_letsencrypt.org,-letsencrypt.org,letsencrypt-.organd labels ofany length, none of which match RFC 8659's
label = (ALPHA/DIGIT) *( *("-") (ALPHA/DIGIT) )-- so a strict CA reads theissueproperty as malformed, whichis the state the validator exists to keep out of a zone, reached through a value
it waved through. Now held to that grammar, with the 63-octet DNS label bound.
Full
dns-persist-01flow re-verified on Pebble after the registration change:account registered, records printed, published with padded separators (the
parser fix's case), certificate issued for
DNS:p3.e2e.test, DNS:*.p3.e2e.test.cargo test -p certbot -p dstack-gateway: 44 + 296.Fourth review round
Every finding this round was in the dashboard button the previous round added,
which is a fair verdict on where new surface area costs the most.
A partial rotation is now reported, not raised.
do_rotate_acme_credentialsbailed when a domain's CAA could not be re-pinned — after registering the
account and publishing the credentials. The dashboard rendered that as "Failed
to register ACME account", so the obvious next move is clicking again, which
registers another account against a rate-limited quota and rewrites CAA a second
time: exactly what the bail's own text warned against. It also meant
required_dns_recordsnever reached the caller on the one path where therecords matter most.
RotateAcmeCredentialsResponsegainsrepin_failed_domains; the rotation returns success, names the domains, and thelog keeps the full "rerun SetCaa, do not rotate again" line at
error!.The Account URI no longer goes stale. It is server-rendered, and rotation
only refreshed the ZT-domain table — so the one authoritative place an operator
reads the account kept showing the replaced one for the rest of the session,
while the dialog beside it showed records naming the new one. The cell is
updated from the response.
The confirmation says what the button actually does. It warned about
republishing
dns-persist-01records — true — and said nothing about the twothings that can hurt: it registers a rate-limited account, and it rewrites the
CAA of every
dns-01domain by installing a;guard and deleting the existingrecords first, so a provider failure part-way through can leave a domain with
CAA that blocks all issuance until a later
SetCaasucceeds.Polish. The new DNS-budget line fired on every issuance in every deployment
— the stock defaults have both values at 300s, so
configured > cappedisalways true — and was worded as if an operator setting had been overridden; it
is now
debug!and phrased as the budget it is. And the issuer-name validator'sfive rejection reasons had been merged into one message that named no character:
my_ca.example.comwas refused without mentioning the underscore, which is thecharacter this round newly started rejecting and the one most likely to be
reached for. Each reason says what it rejected again, with a test on the
underscore case.
cargo test -p certbot -p dstack-gateway: 45 + 296.Maturity
Deliberately opt-in and documented as experimental. draft-01 is the latest
revision, and Let's Encrypt has stated it will not deploy to production
until an open working-group issue about client-computed record
content is resolved — that change would alter the record format. Nothing here
runs unless an operator sets
challenge, and the default path is untouched.Documented in
docs/certbot-dns-persist-01.md.