Skip to content

doh: pooled connections outlive resolver idle-timeouts — recurring "unexpected EOF" / 502 bursts (Quad9 kills idle conns at ≤30s vs 3m IdleConnTimeout) #241

Description

@ShiroiKuma0

Summary

intra/doh/doh.go h2() configures the DoH http.Transport with IdleConnTimeout: 3 * time.Minute, ForceAttemptHTTP2: true, and no HTTP/2 keep-alive health-check (ReadIdleTimeout unset). Several popular resolvers close idle DoH connections far sooner than 3 minutes. Measured from the same network, same day (raw TLS connection reused after an idle period):

resolver idle time survived
dns.quad9.net EOF at ≤ 30 s
dns.mullvad.net EOF at ≤ 90 s
cloudflare-dns.com alive at > 240 s

With Quad9 (or any resolver whose idle timeout is under 3 minutes), every DNS lull longer than the server's idle window leaves the entire pool populated with dead connections. The next queries are written into them, hang ~8–10 s, and surface as Post "https://dns.quad9.net/dns-query": unexpected EOF or http-status: 502. The maxEOFTries (2) retries often just pick the next dead pooled connection, so bursts of parallel queries (A+AAAA × several apps) fail together. Fresh dials then succeed — until the next lull. The result is a chronic cycle of 10–30 s DNS outages every few minutes.

Downstream effects observed on-device (Android, v055x tag, fd3dbcd): alg mappings expire without refresh (alg: dns64: … realips([]) … until: 0s) so established flows of allowed apps die; OS connectivity probes routed through the tunnel fail, so Android network validation flaps device-wide (even VPN-excluded apps feel it). Log signature:

retrier: read: Base: [<laddr><=9.9.9.9:443]; (rtt: 3.02s / …); b: 0/576 (tee: 0); err: EOF
dns: udp: for <uid> err! tot: 0, t: 10.3s, Post "https://dns.quad9.net/dns-query": unexpected EOF
dns: udp: for <uid> err! tot: 0, t: 3.4s, http-status: 502

Verified it is NOT the network or the resolver being down: from a host on the same LAN, fresh requests to dns.quad9.net answered HTTP 200 in ~120 ms throughout, while the device failed continuously. Only reuse-after-idle is poisoned.

Fix (tested on-device)

--- a/intra/doh/doh.go
+++ b/intra/doh/doh.go
@@ func h2(d protect.DialFn, c *tls.Config) *http.Transport {
-	return &http.Transport{
-		DialContext:         asDialContext(d),
-		ForceAttemptHTTP2:   true,
-		IdleConnTimeout:     3 * time.Minute,
+	tr := &http.Transport{
+		DialContext:       asDialContext(d),
+		ForceAttemptHTTP2: true,
+		// keep pooled conns shorter-lived than any observed server-side idle timeout
+		IdleConnTimeout:     10 * time.Second,
 		TLSHandshakeTimeout: 7 * time.Second,
 		// Android's DNS-over-TLS sets it to 30s
 		ResponseHeaderTimeout: 20 * time.Second,
 		// SNI (hostname) must always be inferred from http-request
 		TLSClientConfig: c,
 	}
+	// health-check h2 conns with PING so half-dead ones are evicted
+	// instead of eating a query first (net/http never does this by default)
+	if t2, err := http2.ConfigureTransports(tr); err == nil && t2 != nil {
+		t2.ReadIdleTimeout = 10 * time.Second
+		t2.PingTimeout = 5 * time.Second
+	}
+	return tr
 }

golang.org/x/net is already a module dependency, so the only new import is golang.org/x/net/http2.

Validation

Same device, same resolver (Quad9), worst-case traffic: bursts of 10 parallel uncached queries after guaranteed > 30 s idle gaps, ~40 cycles over 33 minutes. Stock: query-failure bursts on nearly every cycle. Patched: zero idle-pool failures; the log instead shows the health-check discovering and evicting the server-closed connections (retrier: read … err: EOF with no query error following), roughly one per idle cycle.

One caveat worth noting: with the h2 PING enabled we observed a single rare incident of stream error: stream ID N; PROTOCOL_ERROR resets taking out one burst on one connection (once in ~40 cycles; recovered on redial). If that turns out to be an interaction between the h2 health-check and the custom dialer, the IdleConnTimeout reduction alone — which already removes the corpse-pool state structurally — may be the safer minimal fix.

Happy to send this as a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions