feat(chain): add snapshot loading support to database and chain plugin - #87
Merged
Merged
Conversation
On1x
commented
Apr 12, 2026
On1x
commented
Member
- Implement database::open_from_snapshot to initialize DB from snapshot state
- Add database::initialize_hardforks to setup hardfork data post-snapshot
- Extend database interface with snapshot-related methods and fork database accessors
- Reflect new objects: content_type_object, content_vote_object, witness_vote_object
- Modify chain plugin to accept --snapshot option and open DB in snapshot mode
- Integrate snapshot plugin into vizd application and update CMakeLists accordingly
- Include snapshot plugin header and register it with appbase plugin system
- Update submodules chainbase and fc to newer commits
- Implement database::open_from_snapshot to initialize DB from snapshot state - Add database::initialize_hardforks to setup hardfork data post-snapshot - Extend database interface with snapshot-related methods and fork database accessors - Reflect new objects: content_type_object, content_vote_object, witness_vote_object - Modify chain plugin to accept --snapshot option and open DB in snapshot mode - Integrate snapshot plugin into vizd application and update CMakeLists accordingly - Include snapshot plugin header and register it with appbase plugin system - Update submodules chainbase and fc to newer commits
- Add flush() call after writing snapshot data to file to ensure buffered data is written - On non-Windows systems, open the file and call fdatasync() to persist data to disk - Enhance crash safety by guaranteeing snapshot file integrity after write operation
…lugin - Provide standard Docker run command with port mappings and volume mounts - Explain volume mappings for config files and blockchain data - Describe use of VIZD_EXTRA_OPTS for passing extra CLI arguments - Detail how to create periodic snapshots via config and container restart - Include instructions for one-shot snapshot creation with container run - Outline steps to create snapshot at specific block number - Explain loading snapshot during new node bootstrap with environment variables - Emphasize restarting container without extra opts after initial snapshot load - Suggest host-side cron job for managing snapshot retention - Add quick reference table for common snapshot-related Docker commands
- Updated fc subproject commit from 628882d to be9d885 - Ensured thirdparty/fc points to latest commit for sync
…plugin - Added FC_REFLECT for required_approval_object with id, account, proposal fields - Removed unused type alias in import_objects_generic function - Implemented custom serialization of proposal_index objects to handle shared allocators - Converted bip::flat_set fields to fc::flat_set in proposal_object export - Serialized buffer_type field to std::vector<char> for JSON compatibility - Preserved all approval-related sets and public key approvals in exported state - Replaced default EXPORT_INDEX macro usage with manual export for proposal_index
- Add conditional assertion checking vesting withdraw rate only after hardfork 4 - Prevent unnecessary assertion failure before the specified hardfork - Ensure compatibility with chain rules for vesting withdrawals
… plugin - Add snapshot_load_callback and snapshot_create_callback to chain plugin - Load snapshot state during chain plugin startup before on_sync event - Create snapshot after full DB load but before on_sync during startup - Introduce _dlt_mode flag in database to support snapshot-only mode without block log writes - Skip block log validation and fork DB seeding in DLT mode; P2P seeds fork DB after restart - Modify database code to conditionally bypass block log append and ID updates in DLT mode - Remove vesting withdraw rate hardfork conditional due to logic simplification - Register snapshot plugin callbacks in plugin_initialize for snapshot loading and creation - Improve docs to clarify snapshot creation and loading processes occur inside chain plugin startup - Ensure P2P and witness plugins never start until snapshot loading or creation completes - Add missing FC_REFLECT annotations for proposal_object related types to support snapshot serialization
…ng block_log - Add snapshot plugin documentation explaining usage, config, and protocol - Support snapshot creation, loading, and periodic snapshots in nodes - Implement TCP snapshot serving with trust model and anti-spam protection - Add snapshot P2P client to download and load snapshots from trusted peers - Introduce DLT mode detection and enable rolling block_log with configurable window - Implement block_log truncation to maintain rolling window in DLT mode - Enhance chain plugin to handle snapshot load and P2P sync callbacks safely - Add snapshot network message types and serialization helpers - Provide snapshot server logic handling connections, requests, and data streaming - Add client logic to query peers, select best snapshot, and download in chunks - Ensure clean state by wiping shared memory before importing snapshots - Clear genesis objects before snapshot import to avoid conflicts - Add CLI and config options for periodic snapshot and DLT rolling block_log size - Improve error handling on snapshot load failures with logging and app exit
- Add dlt_block_log as a separate append-only block store for recent blocks - Design offset-aware index file with 8-byte header for arbitrary start block number - Modify database to open, read, write, and truncate dlt_block_log alongside block_log - Update irreversible block handling to write to dlt_block_log in DLT mode - Implement automatic fallback to dlt_block_log for block retrieval not in main block_log - Update config and documentation to describe dlt_block_log behavior and settings - Remove old block_log truncation logic, replacing it with dlt_block_log truncation - Add new dlt_block_log.cpp and header with thread-safe memory-mapped file access - Enhance P2P layer fallback to serve blocks from dlt_block_log if missing from main log
- Changed dlt_block_log_impl to detail::dlt_block_log_impl in dlt_block_log.cpp - Ensures correct scoping of the implementation class - Improves code clarity and namespace management
- Skip past already-irreversible blocks when dlt_block_log is empty - Trust the snapshot by advancing dlt_head_num to last irreversible block - Applied this fix in multiple locations to ensure consistency - Prevent potential reprocessing of irreversible blocks after snapshot load
- Introduce snapshot_max_age_days config to specify max snapshot age - Implement cleanup_old_snapshots function to remove snapshots older than set days - Call cleanup_old_snapshots after creating periodic or block-based snapshots - Log removal actions and errors during cleanup process - Add snapshot-max-age-days option with default value 90 days - Enable snapshot rotation logging during plugin initialization when configured
- Added timing start marker before serving data requests - Accumulated bytes sent during data request handling loop - Calculated elapsed time after serving completes - Logged total bytes served and elapsed time using ilog - Updated thirdparty fc submodule commit reference
- Changed subproject commit from 1c1165a to 728d23aa - Updated thirdparty/fc to point to latest commit
…ndling - Add snapshot caching to avoid reading file on each info request - Implement async connection handling with timeout in separate fibers - Limit max concurrent connections to prevent overload (max 5) - Add atomic counter to track active connections safely - Enforce max snapshot download size of 2 GB for safety - Stream checksum calculations in 1 MB chunks to reduce memory use - Add 64 KB max payload limit for control messages (info requests) - Implement rate limiting and cleanup for connection history map - Reject duplicate sessions from the same IP concurrently - Switch snapshot serving to use cached snapshot info and file path - Stream file chunks during snapshot data serving to minimize memory - Improve snapshot download verification by streaming checksum check - Handle socket closing and error logging properly on connection end
…ails - Enhance `--snapshot` option description with restart safety and file rename after import - Add new CLI and config options for snapshot sync from trusted peers and snapshot serving over TCP - Document built-in snapshot rotation and disk space management configurable by `snapshot-max-age-days` - Introduce detailed explanation on DLT mode, rolling block log, and P2P snapshot sync mechanism - Provide restart safety scenarios and clarify snapshot import skip conditions - Update recommended production config with new snapshot and DLT block log settings - Add usage examples for P2P auto-bootstrap and force re-import scenarios - Expand component changes overview reflecting core system modifications for snapshot and DLT block log - Improve connection handling logic in snapshot server plugin for synchronous fiber-based operation - Refactor chain plugin startup to skip snapshot import if shared memory exists or snapshot file missing - Rename snapshot file to `.used` post-import to prevent duplicate imports on restart
…nus one - Update logic to set dlt_head_num to last irreversible block num minus one when it is zero - Ensure the current newly-irreversible block is written during snapshot processing - Prevent skipping of the latest irreversible block in dlt_block_log initialization - Apply the same fix consistently in multiple locations within database.cpp
…lock sync - Replace assertion with conditional check for missing blocks in fork database - Add a flag to track if any blocks were appended before flushing the log - Avoid flushing the deferred log if no new blocks were added - Document the case of missing blocks as normal after restart before fork_db syncs to LIB - Improve stability on restart by breaking loop instead of asserting when block not found
- Introduce DLT rolling block log for selective block retention and archival - Enhance database with open_from_snapshot method for snapshot-based startup - Implement snapshot plugin with comprehensive state restoration and validation - Integrate DLT mode flag and block retention limits in database logic - Update chain plugin startup sequence to handle snapshot import and fallback - Improve error handling during snapshot operations with detailed logging - Document new snapshot loading, DLT mode, and plugin interactions in architecture overview - Adjust core components and dependency diagrams to include DLT and snapshot modules
- Added console output for P2P snapshot sync start, progress, and completion in chain plugin - Enhanced snapshot plugin with detailed stderr logging during snapshot file loading stages - Added logs for connection attempts, snapshot info querying, downloading progress, and checksum verification when downloading snapshots from peers - Changed default of sync-snapshot-from-trusted-peer option to true to enable P2P snapshot sync on empty state by default - Logged enabling/disabling status of P2P snapshot sync based on user configuration - Included timing and block number info in snapshot sync and loading debug output
- Create snapshot directory if it does not exist before writing snapshots or downloads - Add log messages when snapshot directories are created - Prevent snapshot creation while syncing from P2P by checking block timestamp age - Add warning logs and console messages when node has no blocks and no P2P snapshot sync configured - Log when no trusted snapshot peer is configured and P2P snapshot sync is disabled
…s and features - Mark `--sync-snapshot-from-trusted-peer` option default as true and explain auto sync behavior - Add notes that snapshot directory is auto-created if missing - Clarify periodic snapshots only trigger on live blocks to avoid waste during syncing - Improve P2P sync section with detailed progress logging and blocking behavior - Add instructions to disable auto sync and show console warning if no trusted peer configured - Update examples to remove explicit sync flag and recommend config usage instead - Enhance chain and snapshot plugin summaries with new options, logging, and safety checks - Document console progress display during download and import operations - Note diagnostic warning when node at 0 blocks lacks snapshot sync configuration
…andling - Implement sophisticated conditional block fetching with graceful fallback across storage layers - Integrate signal_guard for improved error handling during restart sequences - Update restart sequence handling to enable graceful failure instead of assertions - Strengthen DLT mode integration with runtime parameter control and conditional error propagation - Document enhanced lifecycle methods including open, open_from_snapshot, reindex, close, wipe with DLT awareness - Expand core component coverage to include enhanced error and signal handling features - Add detailed diagrams and flowcharts illustrating block fetch process and restart logic
- Set database _dlt_mode flag to true after loading snapshot - Ensure block_log remains empty to reflect DLT mode - Maintain existing snapshot loading and hardfork initialization flow
…directory management - Add automatic P2P snapshot synchronization for empty blockchain nodes - Implement real-time logging and progress feedback during snapshot operations - Introduce automatic directory creation and intelligent cleanup for snapshot files - Enhance chain plugin integration with callback-based automatic snapshot sync - Update plugin architecture to support new features and improve error handling - Optimize peer selection process for snapshot downloads based on quality metrics - Improve snapshot loading and validation processes with comprehensive updates - Extend system capabilities to include automatic empty node snapshot synchronization
…n integration - Improve block identification and verification processes during snapshot operations - Update database layer to seamlessly handle DLT mode with enhanced error handling - Strengthen fallback mechanisms from primary block log to DLT rolling block log - Enhance snapshot plugin integration with better checksum validation and error reporting - Add detailed documentation of selective retention policies and automatic pruning capabilities - Incorporate runtime parameter documentation and updated architecture diagrams - Include troubleshooting guidance for DLT-specific failure scenarios - Document proper setting of _dlt_mode flag during snapshot loading for seamless DLT mode operation - Enhance P2P snapshot synchronization with automatic empty node detection and progress tracking - Add automatic directory management and improved logging in snapshot plugin system
… gracefully - Change log message to indicate snapshot is being offered, not just served - Add comments explaining normal client disconnect after info query (Phase 1) - Wrap message reading loop in try-catch to handle EOF as clean disconnect - Log informational message on early client disconnect without data transfer - Log warning if client disconnects during snapshot data transfer - Ensure file reading and chunk sending follow updated pattern fix(witness): correct default required participation percentage - Change implicit value of required witness participation option to 33% - Fix calculation of participation rate capture value to reflect correct percentage
- Changed log level from info to debug for rejecting unlinkable blocks - Removed an info log for applied blocks to reduce noise - Removed redundant info log for sync block received in p2p plugin - Used consistent debug logging in block sync process
…and base image - Added libbz2-dev, liblzma-dev, libzstd-dev, and zlib1g-dev dependencies for compression support - Upgraded base image to phusion/baseimage:noble-1.0.3 for all Docker variants - Included sed command fixes to exclude test subdirectories during build - Expanded troubleshooting guide with compression library related issues - Updated runtime and build stage documentation with new dependencies and image references docs(build): improve build helper tools documentation with header inclusion - Documented addition of missing <algorithm> header for code quality improvement - Clarified compilation correctness requirements in build helper tools - Added best practices section for managing headers in C++ build tools
- Promote LIB to head_block_num after snapshot import for correct P2P synopsis - Seed fork database on fresh import and attempt seeding from dlt_block_log on restart - Implement early block rejection in _push_block to handle duplicates, far-ahead blocks, and ensure immediate successor acceptance - Enhance is_known_block() in DLT mode to verify blocks on preferred chain, preventing false positives to peers - Fix fork_database bugs: move _unlinked_index.insert() before throw, add _push_next() call, add duplicate check, and clear _unlinked_index on reset - Update snapshot plugin to rename snapshot files to prevent re-import and improve restart safety - Add detailed documentation on P2P sync process and fixes after snapshot import in DLT mode
- Add a guard to log sync start message only once per sync session - Log sync start with block number at the beginning of syncing - Increase block log frequency from every 10000 to every 500 blocks for syncing - Reset sync start logged guard when not syncing anymore - Retain detailed block info logging during sync for better monitoring
- Enhance DLT mode fork database seeding for automatic seeding on fresh snapshot import - Improve block availability checking logic with DLT mode awareness and better error handling - Add stalled sync detection for DLT nodes with automatic recovery and snapshot reload - Strengthen P2P fallback mechanisms with detailed logging and enhanced block serving - Improve out-of-order block caching and duplicate block detection in fork database - Update fork resolution logic with early rejection, better error handling, and reliability - Integrate automatic fork database seeding with database initialization in DLT mode - Enhance snapshot plugin integration for DLT mode, including stalled sync detection - Refine block retrieval chain with improved fallback and validation strategies
Lib boost 1.71
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.