Replace full-chunk audit responses with BLAKE3 verified slices

[grumbach]

Linear issue

• https://linear.app/autonominetwork/issue/V2-685 (replace full-chunk byte audit responses with BLAKE3 verified slices)

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

This changes the possession-proof representation itself. A weak slice proof would let under-storing nodes pass audits and keep holder credit, so it sits squarely in T3.

Compatibility

• Wire: changed for the audit message families only; core replication is unchanged and is not partitioned. Details below.
• Storage: none. No on-disk format change; commitments are rebuilt from the existing store.
• API: additive. New replication::slice module and public helpers; metric field names deliberately held stable.

Wire detail. The four subtree-audit bodies move to SUBTREE_AUDIT_PROTOCOL_ID (autonomi.ant.replication.subtree-audit.v1) and the AuditChallenge/AuditResponse pair to POSSESSION_AUDIT_PROTOCOL_ID (autonomi.ant.replication.possession-audit.v2). Round 1's SubtreeLeaf carries content_len + nonced_root instead of a flat nonced_hash; round 2's SubtreeByteChallenge/Response become SubtreeSliceChallenge/Response. A receive guard drops any body arriving on the wrong family id, and responses are routed by the same rule.

Core replication stays on autonomi.ant.replication.v2, so fresh replication, neighbour sync, verification, fetch, repair and commitment fetch keep working across versions for the whole upgrade window. Only cross-version audits pause — see Mitigation / rollback for the trust-accounting effect during that window.

API detail. SubtreeLeaf's field changes are within an already-public replication type. The metric field names (subtree_byte_*, audit_dropped_byte) are held stable on purpose: they are what the V2-685 testnet queried to compare a 0.15.0 cohort against this branch with no field mapping, and renaming them would silently zero existing dashboards.

Semver impact

• breaking
• feature
• fix

Stated precisely, since a review pass raised it: the Rust API is source-breaking for a library consumer. ReplicationMessageBody::SubtreeByteChallenge/Response are renamed to ...Slice... and SubtreeLeaf changes fields, and both are publicly reachable through ant_node::replication::protocol. On a 0.x crate that is a minor bump, which is what "feature" delivers, so the box is right either way. Nothing outside this repo consumes these types today — the only callers are the crate's own tests — so no deprecated aliases are provided.

Test evidence

Testnet (jacderida, 990 nodes / 66 VMs, DEV-01, 07-27): 9/9 acceptance criteria pass. Full data in the testnet registry, row 472.

The mixed-version cohort gave a same-network control — both cohorts in one network, at the same time, under the same client load, differing only in the audit protocol version:

cohort	responses	mean round-2 response
DEV-02 (v2, released 0.15.0)	14,197	6.19 MB
DEV-01 (v3, this branch)	69,903	14.49 KB

About 427x smaller, with every confound held constant. The v2 figure independently reproduces production, measured at 5.79–6.15 MB/response on ant-prod-01 across four separate days.

Other criteria: corrupted node caught by 3 independent auditors in +29 min (consistent with the 30-minute per-peer cooldown); zero false positives (digest_mismatch/key_absent/malformed all 0 across ~43k audits over 5.5 h); 4,049/4,049 uploads and 797/797 downloads; 736/736 and 135/135 mixed-version transfers with zero failures; all 7 v2 hosts grew both chunks.mdb and paid_list.mdb, i.e. the v2 cohort kept accepting paid stores from v3 peers throughout; round-1 CPU 41.3% vs 45.7/45.2 baseline and RSS 908 MB vs 916/901 — no measurable regression despite round 1 now building a nonced Merkle tree over all block bytes; audit-timeout rate 1.57/service-hour vs run 470's ~178.5; 990/990 services up, 0 restarts, 0 failed units (both baselines showed 2/990 and 5/990 panics on 0.15.0).

Detection was validated with a construction that zeroes chunk bytes in place with no restart, so the node keeps its identity, uptime and already-gossiped commitment while its bytes no longer hash to their addresses. Deleting chunks.mdb and restarting is an invalid test — the node rebuilds its commitment from the empty store and then honestly declares holding nothing, which is correct behaviour, not cheating.

What the fleet run does and does not cover. It ran at 78cc1b0, and the branch has moved since. The measurements that carry — round-2 response size, round-1 CPU and RSS, detection of an in-place corrupted node, mixed-version transfer success — are all properties of the proof shape, which has not changed since. What landed after the run and therefore has no fleet evidence: the three digest lanes moving to their own protocol id (with GRACE_POSSESSION_AUDIT_TIMEOUTS covering the resulting cross-version silence), the pre-decode ceiling now classifying every inbound replication message by its own discriminant, the responder-wide round-1 work budget being charged on every exit path, and round 2 revoking commitment-scoped credit on silence. The first two touch paths every message crosses, so a short re-run on the final head before this rides a train is the honest close — happy to spec it narrowly.

Local (on the current head, rebased onto main at 0.16.0):

• cfd clean: clippy -D warnings, fmt, cargo doc --deny=warnings.
• 829 lib tests green, including replication::slice, storage_commitment_audit, the round-2 credit and work-budget tests, the family message ceilings, and the rollout dialect selection.
• poc_commitment_audit_attacks 19/19: relay-holds-addresses, predict-and-fetch under fresh sampling, fabricated-fraction, deleter, substitution / replay / structural.
• poc_audit_handler_live 14/14: the production responder against real LMDB, including a deep-block opening of a multi-block chunk verified on both chains, responder coalescing under duplicate and interleaved openings, and the round-1 work charge on a served proof, a mid-run rejected read, and a first-leaf failure.
• tests/e2e audit + prune 10/10: real multi-node QUIC + LMDB. Honest node passes, data-deleting node is caught, repeated audits produce no false positives.

Rollout compatibility. A peer on the previous release sends its possession challenges on the core protocol id and converts an unanswered one into an audit-severity failure. Those challenges are answered rather than dropped, on the id they arrived on and in the digest construction that peer verifies with, so the release does not penalise the nodes that take it. That is a temporary weakening for old-auditor pairings only, bounded and never below what is deployed today, and it is removed together with GRACE_POSSESSION_AUDIT_TIMEOUTS. ADR-0009 records both halves and the criterion.

New dependency

bao = "0.13.1" — Bao verified-slice encoding over BLAKE3. Shares the blake3 crate already in the tree. This is the mechanism that makes a 1 KiB opening verifiable against a chunk's existing content address, and is what the egress reduction rests on. Needs explicit acknowledgement at review.

ADR

https://github.com/WithAutonomi/ant-node/blob/v2-685-blake3-verified-slice-audit/docs/adr/ADR-0009-audit-proof-shape-and-protocol-families.md — ADR-0009: Audit proof shape and protocol families.

Numbered 0009 rather than 0007: main took 0007 for the Windows LMDB map head-room decision while this branch was in review.

ADR-0002 is left byte-for-byte as it was: it records what was decided then. ADR-0009 records what replaces its round-2 proof shape, the decision to unify freshness derivation across all four audit lanes, and the decision to give each changed message family its own protocol id. Every ADR already on main is byte-identical here; this branch adds one file and touches no other.

docs/REPLICATION_DESIGN.md is updated in the same commit, since it is the normative document and still mandated the flat freshness digest this branch replaces.

Mitigation / rollback

Revert the branch. No stored data changes, so a rollback needs no migration: nodes rebuild commitments from their existing store either way. Because core replication never left autonomi.ant.replication.v2, a partially-rolled-back fleet keeps replicating normally — the only effect is that audits pause between mismatched versions, landing in the timeout lane which GRACE_POSSESSION_AUDIT_TIMEOUTS holds penalty-free for the window. Audit rate and admission caps are config constants, so audit load can also be reduced without a code change.

Follow-up owed: remove GRACE_POSSESSION_AUDIT_TIMEOUTS (set false, delete the constant, inline the guarded branches) once the fleet is past the upgrade window, restoring the timeout penalty on the three digest lanes. ADR-0009 now records the owner and an objective criterion for that: at least 99% of nodes seen in the routing table over a seven-day window on the release carrying this change, and no supported release still on the old audit protocol in the field. While the gate is set the suppression applies to every peer, not only old ones, so this should stay open rather than be closed with this PR.

---

What this changes and why

Round 2 of the storage-commitment audit made the audited node return the complete original bytes of the nonce-sampled chunks, up to two full 4 MiB chunks per response. This replaces that with a verified slice: a few KB per opened 1 KiB block, while proving more than the old check.

Framing the benefit correctly, since the original justification has aged: the "~4.2 TB/day" figure predates 0.15.0. Querying V2-623 counters on ant-prod-01, subtree_byte_response was 5.98 TB/day on 07-22 and 32.3 TB/day on 07-24, then fell to 0.46 and 0.22 TB/day on 07-26/27 as the fleet auto-upgraded to 0.15.0. Response size held at 5.8–6.2 MB throughout; only the rate moved. So the incremental saving on today's fleet is nearer 0.2–0.5 TB/day than 4.

The stronger argument is that this bounds the cost per audit event regardless of rate, which frequency caps cannot do. On 07-24 the audit rate spiked and cost 32 TB/day at 6 MB per response; after this change that same spike costs ~32 GB.

How

A chunk's address is BLAKE3(content), and BLAKE3 is internally a Merkle tree over 1 KiB blocks, so a Bao verified slice proves a block is the real content at a given offset against the address the auditor already knows — content authenticity for free. But the address is public, so authenticity alone is delegable: a node storing nothing could pass by fetching the block on demand from an honest holder. Possession must be bound separately.

So round 1 additionally commits, per leaf, a fresh nonced block tree: a Merkle root over the same 1 KiB blocks whose leaves are keyed BLAKE3 hashes, keyed_BLAKE3(key = f(nonce ‖ peer ‖ key), block_index ‖ len ‖ block). The nonce is fed as the BLAKE3 key, not a message prefix, so it mixes into every chunk compression of every leaf — there is no nonce-independent chaining value to precompute. The auditor draws the block to open with fresh randomness after the roots are committed (cut-and-choose), and round 2 verifies both chains over the same block bytes.

A responder cannot fold an after-the-fact-fetched block to a garbage committed root without a preimage break.

What a pass proves — stated honestly

Against the realistic threat this keeps security close to what main gives, at roughly 1/430 of the egress. It is a sampling check, and the claim should be read that way rather than as a proof of complete possession.

The auditor holds none of the audited bytes, so it cannot tell a correct nonced_root from a responder-chosen one. A peer retaining a fraction p of a chunk's blocks passes whenever every fresh draw lands on a kept block — roughly p^leaves per audit over the 3–5 sampled leaves. The mandatory final-block opening anchors the claimed length but adds no detection, since a partial holder simply keeps the final block. The full-byte round 2 this replaces caught any missing byte of a sampled chunk with certainty.

That difference is inherent: verifying every byte means moving every byte. Stating the comparison in both directions:

• Against a node under-storing in bulk — the realistic case, a node dropping data to save disk — detection is close to what the full-byte audit gave, at ~1/430 of the egress. The fleet run caught a 256 MB in-place corruption on the first audit that reached the node, by three independent auditors, against zero false positives across ~43,000 audits in the preceding 5.5 hours.
• Against a fine-grained partial deleter, one shaving a little off every chunk, this is strictly weaker per audit than serving every byte. The compensating lever is audit frequency, which is what the cost reduction buys — the old shape made frequent auditing unaffordable.

partial_holder_passes_on_kept_blocks_and_fails_on_dropped_ones in replication::slice makes the property executable rather than asserted.

For p^leaves to describe detection rather than merely a pass, it also has to cost something not to answer — see the next section.

Round 2 changes: silence is no longer free

Round 2 names the blocks to open only after the round-1 roots are committed, which is what makes the sample a cut-and-choose. The same ordering means the responder sees the draw before deciding whether to reply. Silence was fully graced and left holder credit in place, so an unfavourable draw cost nothing while credit from an earlier favourable one survived. The published probability then describes an answered pass rather than detection.

An unanswered round 2 following a valid round-1 proof now revokes the holder credit carried by the commitment under audit — the same treatment an explicit Transient reject already had. It stays in the graced timeout lane and takes no trust penalty, and the revocation is scoped to the pinned commitment, so credit re-earned under a newer commitment survives. An honest peer that drops a reply keeps its standing and re-earns credit on its next audit; a peer on the older protocol cannot reach this state, because it never completes the new round 1.

The decision is a small predicate (round2_revokes_pinned_credit) rather than an inline match arm, so it is pinned by a direct test, with a companion test on the scoping.

Responder work bounds

Round 1 had a two-slot pool and a per-peer responder cooldown. Both are keyed by peer identity or by concurrency, and neither bounds sustained work: a slot refills the moment it frees, and a per-peer cooldown refills by presenting a different peer id.

Round 1 is now also charged against a responder-wide budget over the chunk bytes it reads and hashes, refilled at a fixed rate and keyed by nothing at all. The balance carries debt, so an expensive proof is admitted proportionally less often than a cheap one and the sustained rate settles at the refill rate whatever the caller's identity. It is sized above honest demand even at the largest commitment the system allows, since starving honest audits would cost more than it saves — at a mid-sized commitment the headroom over measured honest load is more than an order of magnitude.

Charging happens after the proof is built, because the cost is not known until the pinned commitment has been resolved and its subtree selected. Only the proof path needs charging: every rejection a caller can provoke returns before a chunk is read, and the two paths that do read first turn on this node's own storage rather than on anything the caller sends.

Bounding work before admission

Family, session and admission checks all read fields of a decoded message, so none of them can run before decoding. That made decoding the first thing an unknown peer could demand, under a 10 MiB ceiling sized for hint batches that no audit body carries.

The encoded length needs no parsing, so the audit families now take their own wire ceiling, checked before decode and sized against the largest legitimate audit body — the round-1 proof at the commitment key-count cap. A test builds that worst case and pins the headroom, so a later change to leaf size or commitment encoding cannot quietly start dropping honest proofs.

Binding the content length

content_len and nonced_root travel in the round-1 leaf but are not covered by the signed commitment root (it hashes only key ‖ bytes_hash), so a malicious responder can claim any length. Unchecked, that forges possession two ways: a deflated length collapses the block count so only block 0 is ever opened, and Bao does not bind the slice length header for a prefix slice that never reaches EOF.

Both are closed on the auditor side: verify_block_slice requires the slice's own length header to equal the claimed content_len, and the auditor also opens the claimed final block per sampled leaf. The final-block decode reaches EOF, where Bao authenticates the encoded length against the address, so a forged-short length fails there. Signing content_len into the commitment would not help — the attacker would just commit the false length from the start; only anchoring it against the address does.

Both verifiers now also reject a block index past the end of the chunk. Nothing the auditor generates is ever out of range, so this is about failing cleanly rather than a live hole: the block range clamps an over-large index to an empty range, which would otherwise read back as a verified empty block.

Unifying the audit lanes

There are four lanes that ask a peer to prove it holds bytes: the subtree audit, plus three digest-based callers — the periodic responsible-chunk audit, the post-replication possession probe, and prune confirmation. The digest lanes derived their freshness by prefixing the challenge material to the hashed stream, which binds the nonce to the leading bytes rather than the whole content. All lanes now use one construction: derive a BLAKE3 key from the challenge material under a versioned domain-separation context, then hash the content under that key.

That changes what an older peer computes for the same bytes. Left on the shared core id, a cross-version exchange would read as a confirmed digest mismatch and penalise an honest peer for the whole upgrade window — hence the dedicated protocol ids described under Compatibility.

Also: the responsible-chunk audit no longer reports a pass when a challenged key's own local copy is unreadable. That key cannot be checked, so the tick yields no verdict rather than a pass reflecting only the readable keys.

Why not the literal proposal

The ticket's original wording (verify the slice only against the chunk address, with a nonce-derived offset) is unsound: it drops the possession binding entirely, so a storage-less relay farms audit passes. That defeats proof-of-storage and lets one warehouse back many replica identities, breaking replication multiplicity. The offset must also be fresh post-proof randomness, not nonce-derived, or the cut-and-choose collapses. This PR implements the corrected dual-chain design.

Commits

1. Replace full-chunk audit responses with BLAKE3 verified slices — the dual-chain slice audit.
2. Authenticate slice-audit content_len to defeat possession forgery — the content-length binding (header check + final-block open).
3. Harden slice audit and isolate its protocol id — dedicated subtree-audit protocol family.
4. Address audit review: resource bounds, geometry, contract — admission caps, canonical sibling depth, response contract.
5. Tidy slice-audit comments and lock responder coalescing with a live test — coalesced/order-independent round-2 contract, locked by a live responder test.
6. Close two audit holes: key-content forgery and round-1 work amplification — bytes_hash == key enforcement and round-1 work bounds.
7. Bind every audit lane to the fresh nonce and isolate the audit protocols — the unified keyed digest and the possession-audit protocol id.
8. Take the responder permit before consuming the round-2 session — admission ordering fix so a transient capacity drop no longer permanently kills a round-2 exchange.
9. Lock the round-2 accounting boundary with direct tests — round-2 classification as a pure function, tested at the production boundary.
10. Grace audit timeouts through the rollout, and state the proof's real strength — the rollout gate, and the corrected strength claim.
11. Grace only silence in the prune audit lane — split "no answer" from "undecodable answer" at the source.
12. State the strength comparison in both directions — both-directions framing in the docs.
13. Cite the PR and issue in the ADR's Related field.
14. Renumber the audit ADR and align the replication design doc — ADR-0009, corrected lane count, grace-removal criterion and owner, REPLICATION_DESIGN brought into line.
15. Cost an unanswered round 2 the pinned commitment's credit.
16. Bound audit responder work by rate and by message size.
17. Reject out-of-range block openings, and restate the egress figures.

Related

Supersedes #186, which pursued the same egress goal by having the auditor verify co-held leaves against its own copy. Its keyed-digest and Idle-on-unreadable changes are carried here (commit 7); its local-verify optimisation is not, because once round 2 is 14.49 KB the auditor-side cost of reading and hashing its own 4 MiB chunk exceeds the wire bytes it would save. Reasoning recorded on that PR.

The ADR-0008 payment-economics document that briefly appeared on this branch has been dropped: it belongs to #188, which has since merged to main. This branch now carries no ADR content other than its own.

[Copilot]

Pull request overview

This PR updates ant-node’s storage-commitment audit (ADR-0002) to replace round-2 “serve full chunk bytes” proofs with Bao/BLAKE3 verified 1 KiB slice openings plus a nonced per-block Merkle commitment, drastically reducing audit bandwidth while strengthening possession binding. It also wires in ADR-0005 audit-tally reporting so quote responses can carry a signed, nonce-bound audit report.

Changes:

• Replace round-2 full-chunk byte challenges with slice challenges (SubtreeSliceChallenge/Response), including block-index sampling and dual-chain verification (Bao slice + nonced block-tree opening).
• Introduce replication::slice (Bao slice helpers + nonced block-tree build/open/verify) and update subtree leaf commitments to carry content_len + nonced_root.
• Add ADR-0005 audit tally plumbing: record subtree-audit outcomes, persist them, and attach a signed audit report to quote responses; bump replication protocol ID to v3.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/poc_commitment_audit_attacks.rs	Updates attack PoC to the slice + nonced-root audit model.
tests/poc_audit_handler_live.rs	Live-handler tests for slice openings, including deep-block proof on multi-block chunks and cap enforcement.
src/storage/handler.rs	Adds optional signed audit-report sidecar to quote responses and test coverage for report verification.
src/replication/subtree.rs	Changes SubtreeLeaf to include content_len and nonced_root; removes old nonced_hash helper.
src/replication/storage_commitment_audit.rs	Implements slice round-trip request/response, random block selection, and responder slice-opening handler.
src/replication/slice.rs	New module: Bao slice extract/verify + nonced block-tree root/open/verify helpers and tests.
src/replication/protocol.rs	Wire types updated for slice challenges/responses; adds SubtreeSliceOpening; adjusts sizing test.
src/replication/mod.rs	Exposes new modules; integrates ADR-0005 tally persistence and recording hooks.
src/replication/config.rs	Adds MAX_SLICE_OPENINGS, slice response timeout, and bumps replication protocol ID to v3.
src/replication/audit.rs	Extends AuditTickResult::Passed to carry optional commitment_key_count for ADR-0005 tally.
src/replication/audit_tally.rs	New module: per-peer tally, snapshot tagging, report-row building, and quote-source adapter.
src/payment/quote.rs	Adds AuditReportSource and builds/sizes/signs nonce-bound audit reports for quote responses.
src/node.rs	Wires replication engine’s tally into the quote generator as an AuditReportSource.
docs/adr/ADR-0005-earned-reward-eligibility.md	New ADR describing earned eligibility model and report semantics.
Cargo.toml	Adds bao dependency and updates ant-protocol version/pinning.
Cargo.lock	Locks bao and updates ant-protocol resolution.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/replication/slice.rs:257

• extract_block_slice copies the entire chunk (Cursor::new(content.to_vec())) before extracting a small Bao slice. For multi‑MiB chunks this adds an avoidable allocation and extra memcpy per opening; Cursor<&[u8]> works here and avoids duplicating the content buffer.

pub fn extract_block_slice(content: &[u8], index: u32) -> std::io::Result<Vec<u8>> {
    let (start, end) = block_range(content.len() as u64, index);
    let len = end - start;
    // The outboard carries the BLAKE3 tree hashes separately from the content, so
    // the extractor reads the real chunk bytes plus just the parent hashes on the
    // block's path — no need to materialise a full Bao encoding of the chunk.
    let (outboard, _hash) = bao::encode::outboard(content);
    let mut extractor = bao::encode::SliceExtractor::new_outboard(
        Cursor::new(content.to_vec()),
        Cursor::new(outboard),

src/replication/storage_commitment_audit.rs:1207

• handle_subtree_slice_challenge reads chunk bytes from LMDB once per opening (get_raw_retrying inside the openings loop). Because the auditor can request two openings per sampled leaf (random + final), the same key can appear twice and will be read twice, doubling disk I/O and work for the common case. Consider de-duplicating openings by key (read each chunk once, then build all requested openings for that key while preserving response order).

    let mut items = Vec::with_capacity(challenge.openings.len());
    for opening in &challenge.openings {
        let key = opening.key;
        // Open ONLY keys committed under this pin. A key not in the pinned tree
        // is `Absent` — never served from local storage just because we happen to
        // hold it (§15: serving an uncommitted-but-held key would let a forged
        // challenge harvest data and muddy the possession proof for THIS commit).
        if built.proof_for(&key).is_none() {
            items.push(SubtreeSliceItem::Absent { key });
            continue;
        }
        match get_raw_retrying(storage, &key).await {

src/replication/storage_commitment_audit.rs:821

• This success log still says "byte-checked", but checked is now the number of slice openings (block proofs) verified. The message is misleading when diagnosing audit behavior.

        "Audit: peer {challenged_peer} passed subtree audit ({} leaves, {checked} \
         byte-checked)",

Cargo.toml:207

• ant-protocol = "3.0.0" is still being overridden via [patch.crates-io] to a Git rev, with a comment saying "STRIP BEFORE MERGE". This makes the build source ambiguous and can accidentally ship a git override to production. Either remove the patch when 3.0.0 is available on crates.io, or switch the dependency itself to the git source (and drop the patch) until the crates.io release exists.

# An immutable rev (not a mutable branch) keeps the build reproducible; the
# committed Cargo.lock records the exact resolution. STRIP BEFORE MERGE —
# replace with the published ant-protocol 3.0.0 once it lands on crates.io.
[patch.crates-io]
ant-protocol = { git = "https://github.com/grumbach/ant-protocol", rev = "2015113270bbde8204a1740674bed2ada52d2ca6" }

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/replication/slice.rs:257

• extract_block_slice clones the entire chunk with content.to_vec() just to create a Cursor, doubling memory and adding an extra copy per opening. Since Cursor<&[u8]> works, pass the slice directly to avoid the allocation/copy (especially important when openings require full 4 MiB chunks).

    let mut extractor = bao::encode::SliceExtractor::new_outboard(
        Cursor::new(content.to_vec()),
        Cursor::new(outboard),

src/replication/storage_commitment_audit.rs:1199

• handle_subtree_slice_challenge reads the same chunk from LMDB once per opening. Since the auditor can legitimately request two openings per sampled leaf (random + final block), this causes redundant reads for the same key and makes disk-read amplification easier for a forged auditor (even within MAX_SLICE_OPENINGS). Consider grouping openings by key and reading each chunk at most once, while still returning items in the original requested order.

        match get_raw_retrying(storage, &key).await {
            // Committed key, bytes present → build the two-chain opening (or a
            // terminal reject for an out-of-range index / surprise extraction error).
            Ok(Some(bytes)) => {
                match build_slice_item(challenge, key, opening.block_index, &bytes) {

[dirvine]

Reviewed exact head bae97e9c867b5b1e3b00159cf316ef68a0708eaf with a six-seat panel (two groups of three), then independently reproduced the blocking cryptographic finding and ran the focused checks locally.

Verdict: changes requested. The bandwidth direction is good, and the Bao length-forgery fix is well tested, but the current construction does not yet establish the possession property claimed by the PR.

Blocking findings

1. High — nonce-independent BLAKE3 CV still permits deterministic under-storage

src/replication/slice.rs:51,99-115

The leaf input is:

domain(38) || nonce(32) || peer(32) || key(32) || index(4) || len(4) || block(1024)

The 142-byte prefix plus the first 882 block bytes fill BLAKE3 chunk 0. The final 142 block bytes form chunk 1 and are nonce-independent. A node can retain that chunk's 32-byte chaining value instead of the 142 original bytes, saving 110 bytes per full block / 10.7421875%, while reconstructing the exact round-one leaf/root for every fresh nonce.

I reproduced this against the production layout using BLAKE3's low-level subtree API: merge_subtrees_root(chunk0_cv, precomputed_chunk1_cv) exactly matched blake3::hash(prefix || block).

Round two still needs the missing suffixes, but at most ten openings require only about 1.4 KiB of missing data. The generous slice deadline therefore becomes security-relevant again, contrary to the comments claiming the preprocessing gap is closed and all bytes are required.

A fresh nonce-derived BLAKE3 key (new_keyed(nonce) or a domain-separated key derived from the nonce) makes every BLAKE3 chunk nonce-dependent and closes this specific attack. Merely prefixing the nonce does not.

2. High — round-two requests amplify full-chunk reads and synchronous hashing

src/replication/storage_commitment_audit.rs:1148-1203; src/replication/slice.rs:144-207,248-263; src/replication/mod.rs:2527-2594

Each opening independently:

1. reads the full chunk from LMDB;
2. creates a full Bao outboard and clones the full content;
3. rebuilds the complete nonced tree.

The normal random-plus-final pair does this twice for the same key. Duplicate openings are accepted, so one ten-opening request can cause up to 40 MiB of reads plus repeated full scans/copies. Sixteen globally admitted responder tasks bound concurrency, but there is no rate/work budget or binding of round two to responder-held round-one state; repeated requests can keep the pool and Tokio workers busy.

Please coalesce by key, read each chunk once, reuse Bao/nonced tree state for all openings, reject/coalesce duplicates, and add byte/work-aware admission or bind the request to live round-one state. The CPU-heavy construction should not run unbounded on Tokio worker threads.

3. Medium — correlated mid-audit Bootstrapping preserves holder credit

src/replication/storage_commitment_audit.rs:351-357,782-785; src/replication/mod.rs:4471-4507

After serving a valid round-one proof, a responder can answer round two with a matching Bootstrapping response. It is mapped to Timeout; subtree timeouts are fully graced and preserve existing recent_provers credit. This responsive contradiction bypasses both confirmed-failure handling and bootstrap-claim-abuse accounting.

Treat it as a protocol failure, or at minimum revoke the pinned commitment's credit and route it through bootstrap-claim accounting.

4. Rollout gate — v3 partitions all replication operations

src/replication/config.rs:247-262; src/replication/mod.rs:1237-1257

The single global protocol ID is used for fresh replication, neighbour sync, verification, fetch, repair/pruning/possession audits and commitment fetches—not just the changed subtree audit. During a staggered v2/v3 rollout, DHT routing still mixes cohorts while cross-version peers exchange no replication traffic at all.

This needs either a dedicated/dual-registered audit protocol or a quantified and tested coordinated-rollout plan. The current test only asserts the string constant; it does not test mixed-version bootstrap, fresh replication or repair.

Design documentation

ADR-0002 is unchanged and still specifies full chunk responses, flat freshness hashes and deadline-based possession. Several code comments also still describe full bytes/one opening or incorrectly imply the Bao header check alone authenticates prefix-slice length. These are material security-invariant mismatches, not wording nits; please update the ADR and comments with the implementation.

Verified locally

• cargo fmt --all -- --check — pass
• cargo test -q replication::slice --lib --no-fail-fast — 15/15 pass
• cargo test -q storage_commitment_audit --lib --no-fail-fast — 16/16 pass
• cargo test -q --features test-utils --test poc_commitment_audit_attacks — 19/19 pass
• cargo test -q --features test-utils --test poc_audit_handler_live — 9/9 pass
• cargo clippy --all-targets --all-features -- -D warnings — pass
• Max-chunk measurement: Bao slice 1,800 bytes + 12 nonced siblings/384 bytes per opening; about 22 KiB for ten openings before small envelope overhead.

Review-team consensus

All six seats completed. Both cryptographic reviewers independently reproduced the BLAKE3 preprocessing gap. Both runtime reviewers confirmed duplicate full-chunk work amplification. Both rollout/accounting passes confirmed the global v3 split and round-two bootstrap-credit path. No dissent on the blockers; the main qualification is that the CV attack still needs retrieval/delegation for the few opened suffixes.

CI is green except the Windows test job is still pending at review time.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/replication/storage_commitment_audit.rs:1125

• nonced_siblings should never be None for an in-range block_index (you already check block_index < count). Using unwrap_or_default() would silently build an invalid Present item if an internal inconsistency ever occurs, causing honest nodes to fail audits and take penalties. Prefer failing the whole response with a Transient rejection so the auditor routes it to the timeout lane.

            }
        };
        let nonced_siblings = opener.nonced_siblings(block_index).unwrap_or_default();
        items.push(SubtreeSliceItem::Present {
            key,

[dirvine]

Reviewed exact head 712b4c994ecb04522d357f6330f7e398a5a13e2d (incremental from bae97e9c867b5b1e3b00159cf316ef68a0708eaf).

Verdict: changes still requested. The concrete CV attack, round-two duplicate work, responsive-bootstrap credit path, and global protocol partition are fixed. Two material items from the prior review remain, plus focused regression gaps.

Resolved

• Nonce-independent suffix CV: closed. nonced_block_key binds domain/nonce/peer/key and every leaf uses keyed BLAKE3, so every BLAKE3 chunk compression is nonce-dependent (src/replication/slice.rs:60-83,128-142).
• Round-two repeated work: substantially fixed. Openings are coalesced per key, the chunk is read once, Bao/nonced state is reused, and CPU work runs through spawn_blocking (src/replication/storage_commitment_audit.rs:1079-1131,1207-1298).
• Responsive round-two Bootstrapping: now revokes the pinned commitment's credit and becomes a confirmed rejection (src/replication/storage_commitment_audit.rs:360-367,796-808).
• Rollout partition: core replication remains on v2; the four subtree-audit bodies use a dedicated protocol ID with symmetric body/ID guards (src/replication/config.rs:245-282; src/replication/mod.rs:175-208,2342-2358,3360-3377).

Remaining blockers

1. High — expensive audit work is concurrency-bounded, but still not rate- or session-bounded

src/replication/config.rs:132-158; src/replication/mod.rs:2242-2309; src/replication/storage_commitment_audit.rs:995-1067,1171-1298

The 16-global/4-per-peer admission caps prevent unbounded fan-out, but a peer can continuously refill its four slots; a few identities can keep the shared pool saturated. Round two is still accepted from any source presenting a retained commitment hash, without a short-lived/single-use responder-side round-one session.

Round one is the heavier path and still builds each leaf synchronously on Tokio workers after the LMDB read. At the maximum legal subtree hint, a request may process 1,000 × 4 MiB = 3.906 GiB of chunk content; four in-flight requests from one peer are 15.625 GiB, and sixteen globally are 62.5 GiB of worst-case concurrent work. This can starve honest audits and turn dropped honest requests into timeout/failure accounting.

Please add a per-peer rate/work budget, bind round two to a live single-use round-one exchange, and move/bound round-one hashing off Tokio workers. Separate budgets for the much heavier subtree round one versus responsible/slice audits would also avoid one work class consuming the whole shared pool.

2. High — ADR-0002 still specifies a different protocol

docs/adr/ADR-0002-gossip-triggered-contiguous-subtree-audit.md:118-150,152-158,208-218,233-245

The ADR remains unchanged and still specifies flat freshness hashes, original full-chunk responses, deadline-led possession, “last two” retention, and a generic whole-network breaking rollout. The implementation now uses content_len + nonced_root, keyed block leaves, Bao random+final openings, TTL-based retained gossiped commitments, and a dedicated subtree protocol with bounded mixed-version trust effects.

Several source comments also retain removed names/semantics, including nonced_hash, verify_byte_response, full-byte serving, and one opening per leaf (src/replication/storage_commitment_audit.rs:49-53,478-487,1134-1141; src/replication/subtree.rs:235-242; src/replication/protocol.rs:892-895). For a possession protocol, this is normative design drift rather than a wording nit. Please update ADR-0002 and the stale comments in this PR.

3. Medium — the fixed production paths lack direct regressions

There are no focused tests driving:

• responder coalescing/one-read/one-build behavior and duplicate/interleaved requests;
• ResponsiveBootstrap through scoped credit revocation and confirmed-failure classification;
• an actual old-v2 subtree wire fixture being dropped on the core ID;
• dedicated request/response routing and correlation.

The helper tests are useful, but these were explicit prior findings and should be locked at the production-path boundary. Also reconcile the declared response contract (“one item per requested opening, in requested order”, src/replication/protocol.rs:927-942) with grouped/deduplicated emission (src/replication/storage_commitment_audit.rs:1207-1245): either restore original order/cardinality after reusing work, or document and test the coalesced order-independent contract.

Non-blocking hardening

verify_nonced_block accepts any sibling-vector depth that folds to nonced_root (src/replication/slice.rs:254-273). This does not recreate the deterministic under-storage attack—the root precedes fresh sampling—but enforcing the canonical depth from block_count(content_len) (including odd-tail self-pair semantics) would bind the claimed left-packed tree geometry and reject malformed work early.

Verification

Locally on this exact head:

• cargo fmt --all -- --check — pass
• cargo test -q replication::slice --lib --no-fail-fast — 16/16 pass
• cargo test -q storage_commitment_audit --lib --no-fail-fast — 16/16 pass
• cargo test -q --features test-utils --test poc_commitment_audit_attacks — 19/19 pass
• cargo test -q --features test-utils --test poc_audit_handler_live — 9/9 pass
• git diff --check — pass

Current-head CI: builds, format, clippy, docs, security audit, and no-logging tests pass; Ubuntu/macOS/Windows test jobs remain pending.

Review-team result

Five delegated seats plus independent verification reviewed the exact SHA. The cryptographic/adversarial seats agree the concrete CV attack is closed; runtime/release seats agree the rate/session bound and ADR drift remain blockers. The only dissent was severity of canonical sibling-depth validation (low/non-blocking). A required direct Z.AI GLM-5.2 seat was attempted repeatedly but the direct API returned HTTP 429, so this is not represented as full six-model consensus.

[dirvine]

Follow-up review on head 712b4c9: the keyed-BLAKE3, round-two coalescing, responsive-bootstrap, and dedicated-protocol fixes are confirmed. Changes remain requested for sustained audit work lacking rate/session bounds, unchanged ADR-0002/design comments, and direct regression coverage of the fixed production paths. Full review: #181 (review)

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/replication/slice.rs:362

• ChunkOpener::new says it “hashes the chunk once”, but it computes a Bao outboard plus keyed nonced leaves, which are separate hashes/passes. This is misleading for readers trying to reason about round-2 CPU cost.

    /// Hash the chunk once: build its Bao outboard and its nonced block-leaves.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/replication/protocol.rs:1055

• The response is explicitly described as coalesced (items are per distinct (key, block_index), and Absent is at most one per key), so this field doc is inaccurate: the response is not necessarily one item per requested opening (duplicates and key-level Absent can reduce it).

        /// One entry per requested opening.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

tests/poc_commitment_audit_attacks.rs:250

• nonced_block_siblings(..) returning None here would indicate an invariant break (e.g. block_index out of range). Using unwrap_or_default() would silently turn that into an empty sibling chain, which can mask fixture mistakes and make failures harder to diagnose. Prefer failing the test explicitly.

        .unwrap_or_default();

src/replication/config.rs:294

• PR description says REPLICATION_PROTOCOL_ID advances to v3 for the slice-audit cutover, but the implementation keeps core replication on v2 and introduces SUBTREE_AUDIT_PROTOCOL_ID for the subtree audit. Please reconcile the PR description/rollout notes with the code so reviewers/operators don't follow the wrong upgrade assumptions.

pub const REPLICATION_PROTOCOL_ID: &str = "autonomi.ant.replication.v2";

src/replication/protocol.rs:263

• AuditDropKind::Byte is now documented as the round-2 slice challenge. The variant name no longer matches semantics, which can make audit-drop metrics harder to interpret. Consider renaming it to Slice (updating call sites), or explicitly documenting that Byte is a legacy name kept for metric stability.

    /// Subtree audit round-2 slice challenge.
    Byte = 2,
}

[jacderida]

Testnet evaluation — 990-node DEV run, 9 acceptance criteria, all pass

Ran this branch (78cc1b0, = PR head) on a full-scale testnet against the V2-685 evidence plan. 9/9 criteria pass. Summary below; full data is recorded in the testnet registry as row 472.

Treatment	DEV-01, 990 nodes / 66 VMs, 7 bootstraps, 10 uploaders + 2 downloaders, 30% NAT, 4 providers
Control cohort	DEV-02, 105 nodes on released 0.15.0 (v2 audit), extension sharing DEV-01's bootstraps
Profile	staging-short @ 47165ba — identical to baseline runs 470 and 471 (both ant-node 0.15.0)
Phases	T0 = 2026-07-27T14:44:35Z · warm-up 1h · measurement 15:44:35–18:44:35Z · mixed-version 18:51:22–19:51:22Z

Results

#	Criterion	Verdict	Evidence
1	Round-2 response size	PASS	14.49 KB mean over 69,903 responses
2	Corrupted node detected	PASS	3 independent auditors, digest_mismatch, +29 min
3	No false positives	PASS	digest_mismatch/key_absent/malformed all 0 across ~43k audits / 5.5h
4	Transfers	PASS	4,049/4,049 uploads, 797/797 downloads = 100.000%
5	Mixed-version transfers	PASS	736/736 and 135/135 during the phase, zero failures
6	v2 nodes storing	PASS	all 7 v2 hosts grew chunks.mdb and paid_list.mdb
7	Round-1 cost	PASS	CPU 41.3% vs 45.7 / 45.2 baseline; RSS 908 MB vs 916 / 901
8	Audit-timeout rate	PASS	1.57 per service-hour vs run 470's ~178.5
9	Node stability	PASS	990/990 services + 7/7 bootstraps, 0 restarts, 0 failed units

The size reduction, with a same-network control

The mixed-version cohort turned out to be the cleanest possible measurement. Both cohorts ran in one network, at the same time, under the same client load, differing only in the subtree-audit protocol version:

cohort	responses	mean round-2 response
DEV-02 (v2, released 0.15.0)	14,197	6.19 MB
DEV-01 (v3, this branch)	69,903	14.49 KB

417× reduction with every confound held constant. The v2 figure independently reproduces what production is currently doing — I measured subtree_byte_response on ant-prod-01 at 5.79–6.15 MB/response, stable across four separate days, before the run.

14.49 KB rather than the ~6 KB implied by "~1000x" is expected and correct: round 2 now opens two blocks per leaf with two hash chains each.

Round 1 costs nothing measurable

This was the regression I most expected, since the responder still reads every byte of each opened chunk off disk (per the audit_response_timeout sizing comment) while round 1 now builds a nonced Merkle tree over all block bytes rather than the old flat nonced_hash. On 15 services per VM, CPU came out ~9% below both baselines and per-service RSS was flat. No cost visible at this scale.

Detection parity — note the test-design trap

My first attempt was an invalid test, and it is worth recording so nobody repeats it: deleting chunks.mdb and restarting the service lets the node rebuild its commitment from the empty store, so it then honestly declares holding nothing. No detection occurs, and that is the correct behaviour — the node isn't cheating, it's just smaller.

The valid construction zeroes chunk bytes in place with no restart, so the node keeps its identity, uptime and already-gossiped commitment while its bytes no longer hash to their addresses. (LMDB is mmap'd, so deleting the file under a live process achieves nothing — it keeps serving from the unlinked inode.)

Done that way: 256 MB zeroed at 20:26:29Z → caught by three independent auditors at 20:55:31Z (+29 min, consistent with the 30-min per-peer cooldown), against a counter that had read 0 fleet-wide across ~43,000 round-2 responses over the preceding 5.5 hours. That is the Bao authenticity chain doing exactly what it is designed to do.

What this run does not establish

Node egress fell to 34.6 GB per GB uploaded (baselines: 82.0 and 85.7) on aligned windows, while moving more data. I am not attributing that to this PR. Round-2 audit egress this run was 1.04 GB out of 3,644 GB of total replication egress — 0.03%. At the old ~6 MB size those same responses would have been ~256 GB, so this PR accounts for roughly 6% of a ~4 TB gap. This branch carries 25 commits of main beyond v0.15.0 (audit scheduler, cooldown and admission fixes) that runs 470/471 predate, which is the more likely explanation. The same caveat applies to criterion 8's timeout improvement.

Three notes on the PR itself

1. The PR description is stale in one important respect. It says REPLICATION_PROTOCOL_ID advances to v3 and that "v2 and v3 nodes simply don't exchange replication traffic". At 78cc1b0 the subtree audit rides its own SUBTREE_AUDIT_PROTOCOL_ID, so core replication is not partitioned. The run confirms the current behaviour: all 7 v2 hosts grew paid_list.mdb as well as chunks.mdb, i.e. the v2 cohort was accepting paid stores from v3 peers throughout, with zero transfer failures. Worth updating the description — the current text understates the change.

2. The "~4.2 TB/day out" figure predates 0.15.0. Querying the V2-623 counters on ant-prod-01: subtree_byte_response was 5.98 TB/day on 07-22 and 32.3 TB/day on 07-24, then fell to 0.46 TB/day on 07-26 and 0.22 TB/day on 07-27 — the collapse coincides with the fleet auto-upgrading to 0.15.0. Response size held at 5.8–6.2 MB throughout; only the rate moved. So the incremental saving on today's fleet is nearer 0.2–0.5 TB/day than 4. That is not an argument against this PR — arguably the reverse: on 07-24 the audit rate spiked and cost 32 TB/day at 6 MB per response; this change makes that same spike cost ~32 GB. It bounds the cost per event regardless of rate, which frequency caps alone don't. Might be worth reframing the justification that way, since the headline number invites a "we already fixed that" response.

3. Minor naming inconsistency. SubtreeByteResponse → SubtreeSliceResponse in the enum, but log_traffic_summary still emits subtree_byte_response_* at variant index 14. Convenient for us — it made the counters directly comparable across 0.15.0 and this branch with no field mapping — but the name is now misleading.

Method notes

• Counters are cumulative from process start, so figures are per-(host, service) max − min; that undercounts across a restart and cannot overcount. There were no restarts.
• Whole-run deltas are used in preference to timestamp-ranged queries because the profile includes a deliberately +3h clock-skewed VM that a range filter displaces.
• Criterion 9 was checked via systemd, not Elasticsearch — panics reach journald only, so ES-based crash queries read clean regardless.
• Stability was better than both baselines, which showed 2/990 and 5/990 JoinHandle polled after completion panics on 0.15.0. This run had none.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

[dirvine]

Re-reviewed current exact head 2021de9f3c1cf4d45c0ab6a59a6fa86baa0bb0ea with six completed delegated seats plus independent code and runtime verification.

Verdict: changes requested. The three findings from the f64f662 review are addressed: wrong-protocol audit requests now take the body-family ceiling before decode, partial successful round-one leaf work is charged on rejected outcomes, and direct prune/possession rollout tests now distinguish silence from malformed replies. One response-side resource-bound gap remains.

Blocking finding

Medium — solicited audit responses still decode under the generic 10 MiB ceiling

src/replication/protocol.rs:65-84; src/replication/storage_commitment_audit.rs:211-232,326-353; src/replication/audit.rs:300-315; src/replication/possession.rs:325-347; src/replication/pruning.rs:1540-1565

The new 512 KiB family-aware guard is only in handle_replication_message. Request-response replies are intercepted by saorsa-core and delivered directly to each send_request caller, where all five audit paths call the generic ReplicationMessage::decode before checking the expected response type or collection count.

I reproduced this on the reviewed tree with a valid oversized AuditResponse::Digests:

encoded response = 6,400,007 bytes
audit cap       =   524,288 bytes
generic cap     = 10,485,760 bytes
ReplicationMessage::decode = accepted (200,000 digests allocated)

The responder must be the authenticated peer that was challenged, so this is less exposed than the earlier unsolicited/core-ID bypass. It is nevertheless material in this T3 PR because it contradicts the stated audit-family wire bound, and prune confirmation can have up to 32 solicited responses in flight. Apply MAX_AUDIT_MESSAGE_SIZE before every audit-response decode, preferably through one shared bounded audit decoder, and add a response-side regression using an oversized AuditResponse or SubtreeSliceResponse.

Review-team dissent / non-blocking follow-up

A persistent LMDB error on the first selected round-one leaf performs three attempted reads and two 200 ms backoffs but reports content_bytes == 0, so the responder-wide byte budget is not debited. Two seats treated this as blocking; one treated it as bounded unhealthy-storage behaviour because no chunk bytes are returned or hashed, and I agree with the latter classification under a material bar. A fixed attempted-read charge plus a first-leaf-error regression would still make the documented resource guarantee more complete.

Verification

• cargo fmt --all -- --check — pass
• git diff --check — pass
• cargo test -q --lib --no-fail-fast — 822/822 pass on the reviewed resource-control tree
• cargo test -q --features test-utils --test poc_audit_handler_live — 13/13 pass
• focused protocol/prune/possession/storage-audit filters — pass
• current-head CI — completed jobs pass; macOS/Ubuntu/Windows tests and Windows build were still pending at final check

The cryptographic construction, selective-silence handling, protocol-family routing, ADR/design alignment, partial-success work accounting, and prune regression coverage now look sound. This remaining issue is the other half of the same pre-decode boundary: replies need the audit ceiling too.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/replication/config.rs:274

• The constant BYTE_AUDIT_RESPONSE_FLOOR_SECS is now documented as the floor for the round-2 slice challenge, but its name still implies the old byte-challenge semantics. If you want to keep the historical name, it would help to explicitly call that out here to avoid future confusion (especially since there is also AUDIT_RESPONSE_FLOOR_SECS).

/// The round-1 4 s floor is sized for a hashes-only reply; round 2 keeps a
/// slightly larger 5 s base for the disk-read envelope, with the per-byte scaled
/// term (one full chunk per opening) added on top.
const BYTE_AUDIT_RESPONSE_FLOOR_SECS: u64 = 5;

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/poc_commitment_audit_attacks.rs:250

• nonced_block_siblings(..) returning None for an in-range block_index would currently be silently treated as “no siblings” via unwrap_or_default(). Because these fixtures are single-block chunks (tree depth 0), that would let this test keep passing even if sibling extraction regressed to always return None, weakening the audit gate coverage this PoC is meant to pin.

Prefer failing loudly here (e.g., .expect(...)) so a regression in sibling generation is caught by this test.

        let bao_slice = extract_block_slice(&bytes, block_index).unwrap();
        let nonced_siblings = nonced_block_siblings(
            nonce,
            &commitment.sender_peer_id,
            &leaf.key,
            &bytes,
            block_index,
        )
        .unwrap_or_default();

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/poc_commitment_audit_attacks.rs:377

• This comment says the relay fabricates the proof from public data only, but the loop below reads fixture bytes via honest_bytes(&k) to derive content_len. That’s fine for isolating the possession failure, but the comment should reflect it to avoid implying the relay knows the full content.

    // The lazy node fabricates the proof from PUBLIC data only: it knows each
    // leaf key and its bytes_hash (== address), but NOT the bytes, so it forges
    // every nonced_root.

[dirvine]

Re-reviewed exact head 56210cc707b75bce2311004c5042bcac26caacf5 with six completed delegated seats plus independent local verification.

Verdict: changes requested. The response-side ceiling blocker is fixed, the previous zero-charge first-leaf path is now non-zero, and the legacy challenge compatibility path is correctly scoped. One material sustained-work issue remains.

Blocking finding

High — the 4 KiB leaf floor does not make fixed per-leaf work hit the global budget

src/replication/config.rs:162-246; src/replication/mod.rs:3159-3225,3269-3305,3773-3816; src/replication/storage_commitment_audit.rs:67-99,1220-1300

The new charge is correctly placed before the read and totals max(content_len, 4 KiB) per attempted leaf. That fixes the literal zero counter. The calibration, however, leaves the identity-independent budget non-binding for the fixed work it now claims to cover: LMDB lookups/retries, permit occupancy, and one blocking-task dispatch/join per successful leaf.

Using the shipped constants:

work refill                         64 MiB/s
attempt floor                        4 KiB
persistent first-leaf failure       3 reads + 2 × 200 ms backoff
2 heavy slots / 400 ms              ~5 requests/s
charge at that rate                 20 KiB/s
refill / charge                     3,276.8×

The two heavy slots can therefore remain occupied while the budget stays full. The healthy-storage variant has the same problem: a legal 1,024-leaf tiny-record proof is charged 4 MiB, so the bucket permits 16 such proofs/s — 16,384 LMDB lookups and blocking-task round trips/s — before it binds. In practice the two-slot pipeline saturates first, so rotating identities can continuously capacity-drop honest subtree audits without consuming the identity-independent allowance.

The new first-leaf test covers Ok(None) by deleting the record. It does not exercise persistent Err and its retries/backoffs, and the budget tests prove debt mechanics rather than that the chosen floor constrains this work.

Please keep the byte bucket for content read/hash, and add a separately calibrated responder-wide attempt/IOPS budget (charging each read attempt/retry and blocking dispatch), or otherwise justify and test a floor that becomes limiting before the heavy pool can be held continuously. Add regressions for sustained fresh-identity tiny-record work and persistent first-leaf Err, not only one Ok(None) attempt.

Resolved

• All five solicited audit response paths now use the 512 KiB audit decoder before postcard allocation.
• The exact 200,000-digest / 6.4 MB reproducer is rejected by both the audit decoder and the generic family-aware decoder.
• First-leaf Ok(None)/Err and later failures can no longer report zero work.
• New nodes always ask on the dedicated possession ID; legacy core-ID challenges alone are accepted during the rollout gate and answered using the old digest on the old ID. Core-ID possession responses remain refused.
• ADR-0009 now states the compatibility weakening and proof delegability honestly.

Review-team dissent

The completed panel was split on calibration: three focused/adversarial seats treated the underpriced operation work as blocking; one broad seat approved; the compatibility seat approved; the ceiling seat found only an unknown-discriminant hardening point. I independently classify the unknown-discriminant point as non-blocking because postcard rejects an unknown enum arm before decoding any trailing collection.

Verification

• cargo fmt --all -- --check — pass
• git diff --check origin/main...HEAD — pass
• cargo test -q --lib --no-fail-fast — 829/829 pass
• cargo test -q --features test-utils --test poc_audit_handler_live — 14/14 pass
• cargo test -q --features test-utils --test poc_commitment_audit_attacks — 19/19 pass
• exact response-ceiling, first-leaf and compatibility filters — pass
• current-head CI — all completed checks pass; macOS/Ubuntu/Windows test jobs were still pending at final check