feat(node): rate-limit the miss→DHT-lookup path + explicit proxy fallback for the redirect (#2007) - #182
Merged
Merged
Conversation
…back for the redirect (#2007) Compose the shipped redirect-on-miss (#165) + upstream removal (#1997) machinery with the three residual units of dig_ecosystem#2007: - Unit A: a per-requestor token-bucket limiter (`crate::rate_limit`, a byte-identical mirror of dig-wallet's #1957 primitive) in front of `miss_outcome`'s DHT lookup + proxy fetch, keyed by mTLS peer_id / connection IP / operator. Over-budget → -32009 CONTENT_MISS_RATE_LIMITED; a different requestor is unaffected. Redirect candidate set capped at MAX_REDIRECT_PROVIDERS (= dig-dht MAX_ADDRESSES_PER_RECORD). - Unit B: an explicit `params.proxy` (default OFF) routes a miss through the existing FetchThrough branch + identical chain-anchored merkle-verified fetch_resource; the origin!=Local reshare refusal stays intact (the middle node never becomes a holder). - Unit C: regression proving an unconfigured node (no upstream) redirects a miss and issues zero upstream HTTP, degrading old clients to a well-formed JSON-RPC error. Threads the requestor identity explicitly through dispatch + handle_rpc_as + handle_json_rpc(conn_key) so a peer JSON miss is keyed by the asking peer, not one shared bucket. SPEC §10.4 + the -32009 catalogue entry document the bounds, the proxy semantics, and the privacy note. Minor bump (additive `proxy` field + rate limit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmFawKNiEo1dowZ4mPM19j
The dig.getAvailability not-held → find_providers enrichment was the largest DHT-amplification vector on the miss path and ran with NO per-requestor rate cap: one peer session could send a 512-item batch of distinct valid store ids, none held, and drive 512 independent DHT lookups from a ~50 KB request. The candidate cap bounds the RESPONSE, never the outbound LOOKUP work. Thread RequestorId into availability_batch/availability_answer from both entry points (the dispatch.rs JSON leg's in-scope requestor; the peer typed-frame leg's mTLS-verified conn_key as RequestorId::Peer) and gate each per-item find_providers behind the SAME per-requestor miss-lookup budget the single-item legs use — ONE token per not-held item that would trigger a lookup. When the bucket is exhausted the remaining items answer not-available without a lookup; the availability answer itself (held vs not-held from local inventory) is unchanged, only the best-effort providers hint is dropped. So the number of find_providers lookups one requestor can cause via getAvailability — across any batch size and any call rate — is bounded by its per-requestor token budget, identical to the single-item legs. Adds get_availability_enrichment_is_rate_limited_per_item_per_requestor (two actor, control-peer isolation, budget pinned from both sides, load-bearing against both once-per-batch and no-check mutations). SPEC §10.4 + rate_limit.rs module doc extended to include getAvailability in the bounded-path set. Version: workspace 0.99.0, dig-node-core 0.44.0 (exceed main's independent 0.98.0/0.43.0 so the version-increment gate passes post-rebase onto #179). Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d
force-pushed
the
feat/2007-miss-ratelimit-proxy
branch
from
August 5, 2026 03:23
79aba38 to
7dbb758
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Completes the residual of
dig_ecosystem#2007(a node redirects a cache-miss to a live DHT-discovered holder instead of a dead end). The redirect-on-miss mechanism (#165) and the hardcoded-upstream removal (#1997) were already shipped — this composes them and adds the missing security + NAT-robustness residual. Nothing rebuilt.Changes (Units A/B/C)
crates/dig-node-core/src/rate_limit.rs: a per-requestorMissRateLimiter(byte-identical mirror of dig-wallet's #1957TokenBucket) in front ofNode::miss_outcome's DHT lookup + proxy fetch, for bothcontent_miss_envelopeandrange_miss_envelope. Keyed by mTLSpeer_id(peer), connection IP (anon/gateway). Over-budget → new-32009 CONTENT_MISS_RATE_LIMITED. The redirect candidate set is capped atMAX_REDIRECT_PROVIDERS(≤8) inredirect_error_object/providers_json. Oracle bound already enforced bymiss_content_for's 64-hex guard.proxy:truebounded fallback for NAT asymmetry.proxythreaded intomiss_outcome(echoed likeredirect_depth); when set and under the Unit-A budget, routes to the existingFetchThroughpath reusing the identical verifiedfetch_resource. Theorigin != Localreshare refusal is kept intact — the proxy serves bytes but the middle node never becomes a holder (the amplification boundary). Automatic fetch-on-miss stays OFF.RpcDispatch::dispatch+handle_rpc_as+handle_json_rpc(req, conn_key)) so a peer JSON miss keys by the asking peer, not one shared bucket every peer could exhaust.Tests
dig-node-corelib 754 passed / 0 failed;dig-node-servicetests pass;cargo fmt --check+cargo clippy --workspace --all-targets -D warningsclean. New load-bearing two-actor tests:miss_lookup_rate_limit_is_enforced_per_requestor(N+1 refused, control peer unaffected),proxy_flag_routes_a_redirect_mode_miss_through_fetch_through(redirect vs proxy-served bytes),unconfigured_node_redirects_a_miss…(Unit C),redirect_error_object_caps_the_candidate_set, plus 9rate_limit::primitive tests. The existing real-loopback-mTLSconnected_pool_holder_receives_a_real_fetch_range_rpc_over_mtlsremains the real-wire fetch/serve proof.Version
Workspace
0.97.0 → 0.98.0(minor, additive);dig-node-core 0.42.0 → 0.43.0(new publicrate_limit/RequestorIdAPI).Cargo.lockconsistent. SPEC.md updated in-PR (§10.4 + the-32008/-32009catalogue + a privacy note).Coherence handed to the orchestrator (§1.3b)
SYSTEM.md (the
-32008capped-candidate wire +-32009+proxy:field — a cross-repo client contract), docs.dig.net (error catalog +proxy:semantics + rate-limit note), and client redirect/proxy-awareness siblings (dig-sdk / dig-store / dig-node peer client handle-32009+proxy:) are being filed separately.Closes DIG-Network/dig_ecosystem#2007. Refs #1997, #165.
🤖 Generated with Claude Code
Generated by Claude Code