feat(server): local plaintext content-serve surface + local-first cache (#289, #290) (#19)

* feat(server): local plaintext content-serve surface + local-first cache (#289, #290)

Add a NEW loopback-only HTTP surface, `GET /s/<storeId>[:<root>]/<path>`, that
decrypts store content SERVER-SIDE and returns the real website to a same-machine
browser — distinct from the blind-ciphertext JSON-RPC `POST /` (which stays blind
for node<->node + mTLS node-class clients). A browser cannot present a client cert
to get plaintext from the public gateway, so the trusted local node is the only
place a plaintext read can legitimately happen; plaintext only ever crosses loopback.

#289 — serve surface:
- dig-node-core `content_serve`: `Node::serve_content_plaintext` resolves the
  chain-anchored root (pinned #127, fail-closed), derives `retrieval_key =
  SHA-256(canonical rootless URN)` (empty -> index.html), fetches ciphertext
  local-first -> peer -> public RPC, then verifies inclusion vs the anchored root and
  AES-256-GCM-SIV-decrypts, reusing the ONE digstore-core read-crypto every DIG
  client shares. A local module's DECOY for a missing key (proof does not fold to the
  anchored root) is treated as a miss and falls through, never a served/garbage
  result. `Node::manifest_paths` exposes the PublicManifest paths for the SPA
  decision. Plus `derive_retrieval_key` + `ServeSource`/`PlaintextOutcome`.
- dig-node-service `content` (pure helpers): `/s/...` route parsing, store-root
  `<base>` + `<meta referrer=same-origin>` injection, Referer rerooting of
  root-absolute subresources, the ecosystem ext->MIME map + `is_static_asset_path`
  classifier (ported from the on.dig.net loader), and the hardened store CSP.
- server: `GET /s/*path` + a fallback that reroots via `Referer`; SPA history-fallback
  (route miss -> the store's index.html, known-file/asset miss -> honest 404, #144
  MIME rule); content-type + nosniff; store CSP on HTML; and per-serve
  `X-Dig-Verified` / `X-Dig-Root` / `X-Dig-Source` provenance headers (#292).

#290 — local-first cache: a synced+verified .dig serves local with no network (the
default once cached); an unsynced read serves the immediate resource from the
peer/RPC tier AND triggers the single-flight background whole-.dig sync-down
(`maybe_backfill_capsule`, chain-anchored-root pinned) so the next read is local.
Freshness is inherent to the anchored-root pin (a stale generation is never served
as current). `X-Dig-Source` reflects the tier that served the main resource.

Verified: `cargo fmt --all --check`, `cargo clippy --all-targets --locked -D warnings`
(all crates), and the full workspace test suite (0 failed) — including new unit tests
(verify/decrypt round-trip + tamper/wrong-root/wrong-key fail-closed, retrieval-key
derivation, route parse, Referer reroot, content-type, base injection) and end-to-end
HTTP integration tests (decrypted HTML with injected base + headers, JS asset, SPA
route fallback, asset 404, Referer reroot).

Blast radius: purely ADDITIVE. New `content_serve` module + two `Node` methods in
dig-node-core (no existing symbol changed); new `content` module + additive routes
(`/s/*path` + fallback) in the service `router()` (the only modified existing symbol;
`POST /` / `/health` / discovery / control unchanged). No `.dig` format or wire
change (store-format read via the existing digstore libs; §5.1 preserved). Deps: added
digstore-stage/core/crypto + tempfile as dig-node-service DEV-deps (the integration
fixture) at the same rev as dig-node-core.

Bump: minor (0.14.0 -> 0.15.0 workspace; dig-node-core 0.3.0 -> 0.4.0) — a new
backwards-compatible capability (feat), no breaking change.

Closes #289
Closes #290

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(server): use `?` for the Referer path parse (clippy question_mark)

The newer stable clippy CI runs flags the `match Some/None -> return None`
pattern in `reroot_via_referer` under `-D clippy::question_mark`; rewrite it to
the `?` operator (identical behavior, covered by the existing reroot tests).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: MichaelTaylor3d

Cargo.lock

@@ -31,7 +31,7 @@ edition = "2021"
 # (`[workspace.package].version`), so it MUST be set here for a release to fire
 # (§3.6). The library crates (dig-node-core/dig-runtime/dig-wallet) keep their own
 # independent versions — only the released binary tracks the workspace version.
-version = "0.14.0"
+version = "0.15.0"
 
 # Release hardening, matching digstore: keep integer-overflow checks ON in release.
 # The node parses untrusted serialized input and does offset/length arithmetic over

Cargo.toml

@@ -67,7 +67,10 @@ NOT run this in-process node — it is a pure RPC consumer of an EXTERNAL node o
 Chrome extension's `fetchContentViaRPC` pipeline) MUST work against this node unchanged. Verification and
 decryption happen in the **client** — for the DIG Browser via the native read-crypto FFI (§15.1), for
 webpages via the equivalent `dig-client-wasm`; the node serves blind ciphertext + proofs and MUST NOT
-return plaintext for content reads.
+return plaintext for content reads. The ONE exception is the loopback-only local plaintext
+content-serve surface (§4.6) — a DISTINCT HTTP surface from this JSON-RPC read plane — which decrypts
+SERVER-SIDE for a same-machine browser over loopback; the JSON-RPC `POST /` read plane, `rpc.dig.net`,
+and every peer surface stay blind ciphertext + proof.
 
 1.4. **Canonical RPC interface — `dig-rpc-types` + `dig-rpc`.** The RPC surface this node exposes
 (method names + request/response types, the error-code taxonomy, and the tier classification) is
@@ -294,6 +297,8 @@ method/header-allow behavior above.
 | `/openrpc.json` | GET | The OpenRPC document (§6.3). |
 | `/.well-known/dig-node.json` | GET | The discovery document (§6.4). |
 | `/ws/status` | GET (WS upgrade) | WebSocket status/liveness channel (§4.5). |
+| `/s/<storeId>[:<root>]/<path>` | GET | Local plaintext content-serve — server-side decrypt (§4.6). |
+| *(fallback)* | GET | Root-absolute subresource rerooted via `Referer` into its store (§4.6). |
 
 ### 4.5. `GET /ws/status` — WebSocket status/liveness channel (#239)
 
@@ -346,6 +351,71 @@ reconnect) even if the socket's `readyState` still reports open. A client SHOULD
 exponential backoff + jitter on any close/error and reset that backoff the moment a connection
 succeeds again.
 
+### 4.6. Local plaintext content-serve — `GET /s/<storeId>[:<root>]/<path>` (#289/#290)
+
+A same-machine browser cannot present a client cert to obtain plaintext from the public gateway
+(§5.3), so the LOCAL node — the trusted, key-holding, loopback-only endpoint — exposes a DISTINCT
+HTTP surface that decrypts SERVER-SIDE and returns the real website. This is separate from the blind
+JSON-RPC `POST /` read plane (§1.3, §5): plaintext crosses ONLY loopback; `rpc.dig.net` and peers
+stay ciphertext-only.
+
+**Route.** `GET /s/<storeId>[:<root>]/<path>` on every loopback listener (§4.1: `localhost:<port>`,
+`[::1]:<port>`, and bare `http://dig.local`). `<storeId>` and the optional `<root>` are 64-hex; a
+bare `/s/<storeId>[:<root>]/` (empty `<path>`) serves the store's default view `index.html`
+(`DEFAULT_RESOURCE_KEY`). The Host allowlist (§4.2) + CORS (§4.3) answer only loopback names, so this
+surface is never reachable off-machine.
+
+**Resolution + verify + decrypt (fail-closed).** For `(storeId, path)` the node:
+1. resolves `path` → `retrieval_key = SHA-256(canonical rootless URN)` (`urn:dig:chia:<storeId>[/<path>]`,
+   empty → `index.html`) — byte-identical to `dig-client-wasm`/`dig-runtime`;
+2. resolves the store's chain-anchored tip root and PINS the serve to it (§14.4, #127) — a requested
+   root that is not the tip, an unconfirmable store, or an unreachable chain fails closed;
+3. fetches the resource's ciphertext + inclusion proof + chunk lengths LOCAL-FIRST, then peer, then the
+   public RPC (§4.6 cache order below);
+4. verifies `resource_leaf(ciphertext) == proof.leaf`, `proof.verify()`, and `proof.root ==
+   chain_anchored_root`, THEN AES-256-GCM-SIV-decrypts each chunk under the per-URN key — the SAME
+   `digstore-core` read-crypto every DIG client uses. A tampered chunk, decoy, or non-anchored root
+   never decrypts.
+
+**Store-root scoping (shared-origin best-effort).** Served HTML is rewritten with an injected
+`<base href="/s/<storeId>[:<root>]/">` (RELATIVE links resolve within the store) and
+`<meta name="referrer" content="same-origin">`. A ROOT-ABSOLUTE `/foo` request (the browser drops the
+`/s/...` prefix) lands in the router fallback and is REROOTED via the same-origin `Referer` back into
+its store; an unattributable root-absolute request is a `404` (asset) or the SPA fallback (route).
+Absolute `https://…` URLs bypass the node entirely.
+
+**SPA history-fallback + MIME rule (#144).** A route-like miss (`path` whose final segment has NO known
+static-asset extension) serves the store's `index.html` (`200 text/html`) so a client-side deep link
+boots. The node uses the store's `PublicManifest` (§5.5.1) to distinguish a KNOWN file genuinely missing
+at this root (an honest `404`) from a route (the SPA fallback); a null manifest (old/private store)
+degrades to the extension-less-path heuristic. An ASSET miss (a known non-HTML extension —
+`js`/`mjs`/`css`/`json`/`wasm`/`svg`/images/fonts/media/…) is ALWAYS an honest `404`, never `text/html`
+(a `text/html` body for a service-worker/module fetch is rejected by the browser for a wrong MIME type).
+
+**Content-type + CSP.** The `Content-Type` is the ecosystem extension→MIME map (byte-identical to the
+DIG loader's `contentType()`), with `X-Content-Type-Options: nosniff`. Served HTML additionally carries
+a synthesized hardened store CSP (`object-src 'none'`, same-origin `base-uri`, un-framed, with the
+sanctioned content network legs) attached as a response header, never trusted from the store body.
+
+**Provenance headers (every serve, #292).** `X-Dig-Verified: true|false` (inclusion + chain-anchored-root
+verified server-side — `false` only when the node-side pin is disabled via `DIG_NODE_PIN=off`),
+`X-Dig-Root: <root>` (the resolved root served against), and `X-Dig-Source: local|peer|rpc` (the tier
+that served the MAIN resource). A consumer's DIG Shields / toolbar reads these.
+
+**Local-first store cache (#290).** Resolution order per `(store, root)`:
+1. a synced+verified `.dig` module on disk → serve LOCAL, no network (the DEFAULT once cached);
+2. not held → serve the immediate resource from a peer / the public RPC AND trigger a single-flight
+   background whole-`.dig` sync-down (the deduped `maybe_backfill_capsule` → chain-anchored-root-pinned
+   whole-store pull) into the reserved LRU cache dir, so the NEXT read is local. LRU eviction (§7.10)
+   applies; an evicted-then-re-requested capsule re-syncs. Freshness is inherent to the anchored-root
+   pin (§14.4): a stale locally-cached generation whose root is not the on-chain tip is NEVER served as
+   current — the read resolves the tip and fetches/backfills that generation, so local-default is never
+   local-FROZEN. A synced `.dig` is trusted only after it verifies against the on-chain root at serve.
+
+**Salt.** A private store's secret salt is not yet provisioned to this surface; a private store therefore
+fails closed at decrypt. Public stores (salt = none) serve fully. (Private-store salt provisioning is a
+tracked follow-up.)
+
 ---
 
 ## 5. JSON-RPC surface (read plane)

SPEC.md

@@ -1,6 +1,6 @@
 [package]
 name = "dig-node-core"
-version = "0.3.0"
+version = "0.4.0"
 edition = "2021"
 license = "GPL-2.0-only"
 description = "The canonical DIG node ENGINE library (crate `dig_node_core`): the JSON-RPC dispatch (`handle_rpc`, the same contract as rpc.dig.net), local-first content serve/fetch/redirect from LOCAL .dig store modules (via digstore_host::serve_blind), chain-anchored-root resolution, chain-watch + subscriptions + generation gap-fill, the LRU cache, and the full P2P stack. Shared UNCHANGED by both host shells: the `dig-node` OS-service binary (dig-node-service) and the DIG Browser's in-process cdylib (dig-runtime). Native Rust so the compiled-module serve path works."