Setup: curio self-built from recent main (includes #1270 termination handling, #1220 chain-state sync, #1309 batched DeletePiece), mainnet, ezpdpz-main. Data set 1245 — Covalent's Storacha-migration data set, ~186k live pieces (206,250 piece ids allocated), active client payment rails.
As part of the Storacha migration we needed to remove 19,780 duplicate pieces from Covalent's data set 1245. On 12 June we scheduled the first 1,503 removals via schedulePieceDeletions — 500 per call per the handler cap, well under the contract's ~2,000-per-proving-period limit. Every call was accepted.
Scheduled removals are executed inline by the next nextProvingPeriod call, and on a data set this size that combined transaction exceeds the block gas limit. Estimation reverts out of gas, so curio never submits it — no failed message on-chain, no FaultRecord, nothing for chain-event monitoring to see. Proving for this data set silently stopped on 13 June. Both caps passed, the state they permitted was un-mineable, and we found no way to un-schedule removals once accepted.
Next challange window scheduling failed … failed to estimate gas: chain: message
execution failed (exit=[33], … out of gas (7)) (task_next_pp.go:143)
On 1 July at 15:05 UTC, task_chain_sync.go:435 reset the data set's proving-failure state (reset PDP data set proving failure state after confirmed PDPVerifier progress, lastProvenEpoch: 6102318 — unchanged since 13 June), arming a clean failure counter. Five consecutive estimation failures followed with backoff (failure_handling.go:106, 100→800 blocks). At 03:47 UTC on 2 July the data set was marked terminated due to repeated failures (failure_handling.go:87/:155), and at 05:00 UTC curio sent and confirmed FWSS terminateService: tx 0x3b973bf3fb06342c71fdcb5fe3f1254736f3eba630ed902c579eec179cbea8d1 — ServiceTerminated + PDPPaymentTerminated emitted, approver = the SP session wallet. Covalent did not request termination and the operator was not involved or notified.
Result: a paying migration customer's data set was terminated on-chain by node automation, three weeks downstream of a duplicate-removal operation that stayed inside every advertised limit. Termination is irreversible — we are recommitting the full data set (~186k pieces) from backup.
Remaining concern: the remaining ~18k duplicates mean any future bulk removal on a large data set can recreate the wedge, and any ProvPeriod failure that persists for 5 attempts — whatever the cause — ends in terminateService. The stall phase produces no on-chain evidence, so an SP sees nothing until the termination lands. Same failure family as #1296 (calibnet, oversized cleanupPieces tx wedging proving); this is the mainnet variant with the #1270 termination path on the end.
Full logs, harmonydb state, and the failing estimation call available on request.
Setup: curio self-built from recent main (includes #1270 termination handling, #1220 chain-state sync, #1309 batched DeletePiece), mainnet, ezpdpz-main. Data set 1245 — Covalent's Storacha-migration data set, ~186k live pieces (206,250 piece ids allocated), active client payment rails.
As part of the Storacha migration we needed to remove 19,780 duplicate pieces from Covalent's data set 1245. On 12 June we scheduled the first 1,503 removals via
schedulePieceDeletions— 500 per call per the handler cap, well under the contract's ~2,000-per-proving-period limit. Every call was accepted.Scheduled removals are executed inline by the next
nextProvingPeriodcall, and on a data set this size that combined transaction exceeds the block gas limit. Estimation reverts out of gas, so curio never submits it — no failed message on-chain, no FaultRecord, nothing for chain-event monitoring to see. Proving for this data set silently stopped on 13 June. Both caps passed, the state they permitted was un-mineable, and we found no way to un-schedule removals once accepted.On 1 July at 15:05 UTC,
task_chain_sync.go:435reset the data set's proving-failure state (reset PDP data set proving failure state after confirmed PDPVerifier progress,lastProvenEpoch: 6102318— unchanged since 13 June), arming a clean failure counter. Five consecutive estimation failures followed with backoff (failure_handling.go:106, 100→800 blocks). At 03:47 UTC on 2 July the data set was markedterminated due to repeated failures(failure_handling.go:87/:155), and at 05:00 UTC curio sent and confirmed FWSSterminateService: tx0x3b973bf3fb06342c71fdcb5fe3f1254736f3eba630ed902c579eec179cbea8d1—ServiceTerminated+PDPPaymentTerminatedemitted, approver = the SP session wallet. Covalent did not request termination and the operator was not involved or notified.Result: a paying migration customer's data set was terminated on-chain by node automation, three weeks downstream of a duplicate-removal operation that stayed inside every advertised limit. Termination is irreversible — we are recommitting the full data set (~186k pieces) from backup.
Remaining concern: the remaining ~18k duplicates mean any future bulk removal on a large data set can recreate the wedge, and any ProvPeriod failure that persists for 5 attempts — whatever the cause — ends in
terminateService. The stall phase produces no on-chain evidence, so an SP sees nothing until the termination lands. Same failure family as #1296 (calibnet, oversized cleanupPieces tx wedging proving); this is the mainnet variant with the #1270 termination path on the end.Full logs, harmonydb state, and the failing estimation call available on request.