fix(consensus/XDPoS,core,eth): repair missing V2 gap snapshots at startup - #2508
Open
gzliudan wants to merge 1 commit into
Open
fix(consensus/XDPoS,core,eth): repair missing V2 gap snapshots at startup#2508gzliudan wants to merge 1 commit into
gzliudan wants to merge 1 commit into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
gzliudan
requested review from
AnilChinchawale,
anunay-xin,
benjamin202410,
liam-lai and
wanwiset25
August 10, 2026 05:30
This was referenced Aug 10, 2026
anunay-xin
approved these changes
Aug 10, 2026
…rtup A node can lose its persisted V2 gap snapshot when the process exits between writeHeadBlock and UpdateM1 in writeBlockWithState: the head markers are already on disk, the snapshot is not, and nothing recreates it. Every consumer of that snapshot then fails for a whole epoch. A node that keeps up with the chain head still imports blocks, because the fetcher path verifies headers with fullVerify disabled and trusts header.Validators, but it can no longer mine or vote. A node that falls behind takes the downloader path, where fullVerify is enabled, and stalls on the epoch switch block while reporting it as a bad block. Repair the hole once at startup, before the chain is used: for each gap block whose snapshot is missing, rebuild it from the state committed at that block and store it. Only the last two gap blocks at or below the head are considered. A gap block G is among them exactly while the head is in [G, G+2*Epoch), and G's snapshot is consulted exactly while the head is in [G+Gap, G+Gap+Epoch), so every hole that can still affect the running chain is covered. Checking a single gap block is not enough: a node killed at G restarts with the head on G itself, while a node that stalled on the following epoch switch restarts several hundred blocks later, and no single derivation yields both. The derivation is shared with core.BlockChain.UpdateM1 and the downloader through the unstable xdc_sort ordering, which must not drift, since a different equal-stake order yields a different masternode set. It also refuses to produce an empty snapshot, which would load back fine and permanently mask the missing masternode list. The rebuild is skipped when a snapshot is already stored, so a decode or I/O error never overwrites a masternode set persisted through a reorg, and at or below the V2 switch block, where the snapshot still comes from Initial(). It needs the gap block's state root to still be readable, which the targeted case implies: a node only keeps that root when it was committed, and this trie database never prunes what it has written. Otherwise loadLastState finds no head state, repair() rewinds to an ancestor that has one, and re-importing the gap block runs UpdateM1 and writes the snapshot again. Roots that are gone for good, such as after offline pruning or below a fast sync pivot, cannot be recovered, so every failure is only logged and startup continues.
gzliudan
force-pushed
the
repair-snapshot-at-startup-v28
branch
from
August 11, 2026 09:01
cffe418 to
05d3d8f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
A node can lose its persisted V2 gap snapshot when the process exits between writeHeadBlock and UpdateM1 in writeBlockWithState: the head markers are already on disk, the snapshot is not, and nothing recreates it. Every consumer of that snapshot then fails for a whole epoch. A node that keeps up with the chain head still imports blocks, because the fetcher path verifies headers with fullVerify disabled and trusts header.Validators, but it can no longer mine or vote. A node that falls behind takes the downloader path, where fullVerify is enabled, and stalls on the epoch switch block while reporting it as a bad block.
Repair the hole once at startup, before the chain is used: for each gap block whose snapshot is missing, rebuild it from the state committed at that block and store it.
Only the last two gap blocks at or below the head are considered. A gap block G is among them exactly while the head is in [G, G+2*Epoch), and G's snapshot is consulted exactly while the head is in [G+Gap, G+Gap+Epoch), so every hole that can still affect the running chain is covered. Checking a single gap block is not enough: a node killed at G restarts with the head on G itself, while a node that stalled on the following epoch switch restarts several hundred blocks later, and no single derivation yields both.
The derivation is shared with core.BlockChain.UpdateM1 and the downloader through the unstable xdc_sort ordering, which must not drift, since a different equal-stake order yields a different masternode set. It also refuses to produce an empty snapshot, which would load back fine and permanently mask the missing masternode list.
The rebuild is skipped when a snapshot is already stored, so a decode or I/O error never overwrites a masternode set persisted through a reorg, and at or below the V2 switch block, where the snapshot still comes from Initial(). It needs the gap block's state root to still be readable, which the targeted case implies: a node only keeps that root when it was committed, and this trie database never prunes what it has written. Otherwise loadLastState finds no head state, repair() rewinds to an ancestor that has one, and re-importing the gap block runs UpdateM1 and writes the snapshot again. Roots that are gone for good, such as after offline pruning or below a fast sync pivot, cannot be recovered, so every failure is only logged and startup continues.
Types of changes
What types of changes does your code introduce to XDC network?
Put an
✅in the boxes that applyImpacted Components
Which parts of the codebase does this PR touch?
Put an
✅in the boxes that applyChecklist
Put an
✅in the boxes once you have confirmed below actions (or provide reasons on not doing so) that