Skip to content

design.md

Latest commit

 

History

History
225 lines (171 loc) · 13.9 KB

design.md

File metadata and controls

225 lines (171 loc) · 13.9 KB
Raw

readcon-db design notes

Problem

readcon-core streams and validates CON/convel files. NEB and long-timescale campaigns produce corpora: (10^3)–(10^6) frames across many trajectories. Loading every frame as a ConFrame exceeds RAM; scanning every file for “all frames with H and Cu and (N<200)” wastes I/O.

Approach

Treat the corpus as an embedded key-value database with hand-built secondary indexes, backed by LMDB via Heed so the OS page cache supplies RAM residency without a custom buffer pool.

Embedded multi-process SOTA patterns (what we implement)

These are standard patterns for local high-performance multi-process KV stores (LMDB / Kyoto Cabinet / RocksDB-class embedded engines; Gray & Reuter MVCC lineage)— not distributed cluster consensus (Raft/Paxos), which is out of scope for a workstation campaign corpus.

Pattern Chemistry-store gap (ASE.db / SQLite) readcon-db
Mmap primary storage Row pages + BLOB deserialize OS page cache holds CON text
MVCC multi-reader, single-writer Writers/readers contend on SQLite locks LMDB COW pages; max_readers=512
Cross-process shared env One connection model in Python Separate OS processes each open() same dir
Authoritative value + secondary indexes Pickled Atoms + ad hoc KV CON blob authoritative; B-tree indexes rebuild via reindex
Selective index scan + set intersection Often full table / formula scan Smallest-first BTreeSet intersect
Batch read txn for trajectory extract Per-row toatoms() touch_trajectory_blobs / get_frame_texts one RoTxn
Exact content addressing Row id / UUID xxHash3-128 on stored blob

Cluster / network patterns (not implemented): multi-node sharding, Raft leader election, gRPC query plane. Those do not match the NEB-on-one-host threat model.

Why Heed/LMDB (not SQL, not SQLite)

  • Mmap-first; readers do not copy the whole DB into a heap arena.
  • Concurrent read transactions are first-class.
  • Predictable latency for point lookups and ordered scans.
  • We control indexes (natoms, symbols, energy, composition, fmax, section flags) for exact access patterns of MD post-processing.
  • SQL would invite ad hoc joins that encourage copies and planner variance.

Secondary indexes

DB name Key layout Select predicates
idx_natoms BE u32 n_atoms ‖ FrameKey natoms_range
idx_symbol symbol UTF-8 ‖ 0xff ‖ FrameKey require_symbol
idx_elem_count symbol ‖ 0xff ‖ BE count ‖ FrameKey element_exact / element_min
idx_formula canonical Sym:count|... ‖ 0xff ‖ FrameKey exact_composition
idx_topo / topo_by_frame topology-key utf-8 ‖ 0xff ‖ FrameKey; FrameKey → key topo_key, find_by_topology_*
idx_energy order-preserving BE bits of finite energy ‖ FrameKey energy_range
idx_fmax order-preserving max (|F_i|) ‖ FrameKey (forces only) fmax_range
idx_flags flag_id (u8) ‖ FrameKey require_forces / require_velocities / require_energy
frame_by_hash / hash_by_frame xxHash3-128 exact_hash, dedup

Formula encoding: sorted non-empty symbols, Sym:count joined by | (e.g. Cu:2|H:2). Finite energies and fmax only; frames without forces never satisfy a finite fmax_range.

Query cost model: each predicate materializes a BTreeSet<FrameKey> from one index scan (or hash point lookup). Final result is set intersection, then sort + optional limit. Cost tracks selective indexes—not full corpus decode. Full-table fallback only when no indexed predicate is set.

Ingest paths

  1. Path / multi-frame CON text (append_trajectory_path / append_trajectory_str) with next_with_raw_span when possible.
  2. In-memory frames (append_trajectory_frames / extend_trajectory_frames) for chemfiles → ConFrame → corpus without a temp file. Python: append_trajectory_str / append_trajectory_frames. C: rkrdb_append_trajectory_str / rkrdb_append_trajectory_frame (RKRConFrame*).
  3. Directory ingest (ingest_directory).

Reindex

ConCorpus::reindex (CLI: readcon-db reindex <corpus_dir>) clears secondary DBs and rebuilds them from authoritative frames blobs—schema upgrade path after adding indexes. Safe to run twice (idempotent key sets). Frames and traj_meta are not deleted. idx_topo is rebuilt from topo_by_frame when every annotated trajectory records the same cutoff/graph/hops/method; mixed parameters error without a half-rebuild. Unannotated corpora leave idx_topo empty.

Invariants

  1. Frame blob is authoritative for fidelity; indexes are derived and rebuildable.
  2. Single writer for ingest/reindex; analysis is read-only (ConCorpus::open_readonly / MDB_RDONLY). MPI ranks that all need the same frames should not each open the env: rank 0 of the caller communicator pack_frame (RCSO) / pack_frames (RCSB) and MPI_Bcast on that same handle (include/readcon-db-mpi.h, Python bcast_packed_frame / bcast_packed_frames). The host already owns MPI (LAMMPS lmp->world / a sub-comm, mpi4py Comm); the library never calls MPI_Init and never names the process-wide world communicator. Workers unpack with no handle. Shared mmap (open_readonly on every rank) is the other legal path when ranks touch different keys.
  3. Decode with readcon-core so CON semantics never fork.
  4. Selection returns keys first; callers decode lazily.

Core contracts (readcon-core::index_proj)

Screening scalars and ingest rules live in readcon-core so this crate does not fork CON meaning:

Contract API Role
Index projection FrameIndexProjection::from_frame natoms, formula, finite energy, fmax, mass, volume, sections mask, meta channels
Formula encoding composition_formula / frame_composition_formula idx_formula keys (`Cu:2
Finite policy finite_energy, mass/volume/fmax Non-finite scalars omitted from ordered indexes
Sections mask sections_present_mask / SECTIONS_MASK_* forces / velocities / energies flags
Span ingest ConFrameIterator::next_with_raw_span, frame_byte_spans Store exact multi-frame substrings; no hot-path re-serialize
Canonical write ConFrameWriter::canonical(true) Opt-in stable JSON key order for materialize-from-frames

frame_scalars and corpus prepare/reindex call into these APIs (thin wrappers / delegates).

Ecosystem

Crate Responsibility
readcon-core CON/convel interchange, chemfiles ingress, multi-language hourglass ABI. CPC manuscript main claim.
readcon-db (this repo) Companion campaign store: corpora, indexes, mmap multi-reader, exact dedup, reindex. Not a second CPC paper.

ASE is calculator-only; not the campaign store. Fair ASE.db timings for a CPC appendix, if used, are the freeze in paper/cpc/freeze/.

Security / multi-tenant

Single trusted user on local disk for v1. No network protocol in v1.

Status

Shipped: frames, traj_meta, composition/energy/fmax/flags/natoms/symbol/hash indexes, Select, reindex, append frames, CLI/Python/C campaign select.

Optional cooked SoA tier (frames_soa)

Shipped (derived, non-authoritative): RCSO is not authority. Each FrameKey may have a binary payload in LMDB DB frames_soa (magic RCSO, v1 LE POD header + f64 N×3 positions and optional forces/velocities). Encode/decode: cooked_soa::CookedSoa from a parsed ConFrame.

Why we still carry CON text (not a “fully equivalent blob”)

RCSO is not a complete stand-in for a CON frame: it omits element symbols, masses, cell/angles, constraint bits, JSON metadata, section structure, and exact UTF-8 bytes. Those are required for xxHash3 content addressing, composition/symbol indexes, CON export, and join/split fidelity. The tier’s purpose is to avoid CON parse on numeric extract (get_positions / get_forces / get_velocities) when a valid cooked blob exists—not to drop frames storage. Deleting CON text while keeping RCSO is unsupported and fails get_frame_text / hash; keeping CON and deleting RCSO always works.

Rule Behavior
Authority UTF-8 CON text in frames only; xxHash3 / dedup / join-split / reindex ignore SoA
Opt-in Default ingest does not cook; cook_frame / recook_all / append_trajectory_path_cook(..., true)
Hot path Numeric getters prefer valid cooked (no CON parse on hit); corrupt/missing → parse CON
Discard delete_cooked_soa; reindex and select unaffected; CON text and hash unchanged
Non-equivalence Symbols/metadata/exact bytes live only in CON; RCSO cannot substitute for missing frames
DLPack Still ephemeral in-process views on ConFrame—not an LMDB value format

Roadmap (unchanged): parallel chunked reindex; optional multi-dtype SoA matrix.

ASE.db column ↔ readcon-db (competitive screening set)

Speed only matters if filters users already have in ASE.db exist. Architecture stays non-SQL (secondary LMDB DBs + intersection); the feature claim is campaign-column parity for CON-derivable fields.

ASE.db / common filter CON source readcon-db predicate / index
natoms atom count natoms_range / idx_natoms
formula / species multiset exact_composition, element_* / idx_formula, idx_elem_count
symbol presence symbols require_symbol / idx_symbol
energy metadata energy energy_range / idx_energy
forces present / fmax forces section / ‖F‖ require_forces, fmax_range / idx_flags, idx_fmax
velocities section / data require_velocities
total mass masses_per_type × counts mass_range / idx_mass
cell volume lattice_vectors or boxl+angles volume_range / idx_volume
pbc metadata pbc (explicit only; missing ≠ match) pbc([x,y,z]) / idx_pbc
time, timestep reserved metadata time_range, timestep_range / idx_meta
frame_index reserved metadata frame_index_range / idx_meta
NEB bead/band neb_bead, neb_band neb_bead_range, neb_band_range / idx_meta
charge, magmom optional JSON numbers charge_range, magmom_range / idx_meta
exact structure id CON blob exact_hash / xxHash3
topology up to relabelling seams fingerprint key topo_key / idx_topo
id / row id — FrameKey (traj_id, frame_idx)
unique_id UUID — N/A (ASE bookkeeping); use content hash
ctime / mtime / age — N/A (ASE bookkeeping)
user / calculator — N/A unless stored in CON metadata (not competitive set)
arbitrary key_value_pairs DSL — N/A (no SQL DSL); reserved + charge/magmom cover screening
SQL SELECT — N/A (architecture); use Select / CLI / Python

reindex rebuilds all secondary indexes including mass/volume/pbc/meta.

Writer concurrency

CPU-bound prepare (parse CON spans / serialize ConFrames) runs outside the exclusive LMDB write_txn. Concurrent threads may prepare in parallel; only commits serialize at the engine (single active write txn). FFI handle-table locks do not cover ingest.

HPC multi-writer (millions of ranks)

A single LMDB environment cannot run concurrent write_txns. For site-scale ingest (many SLURM tasks uploading CON), use ShardedConCorpus:

  1. readcon-db shard-init /scratch/campaign --shards 256 once on the shared FS.
  2. Each rank opens only its shard. One writer owns each shard_id across the job (traj_id % n_shards). If many ranks share a shard id, each node keeps a private tree, drains to a unique dest, then join-drained.
  3. Writers on different shards never share a write lock — up to n_shards parallel commits on one filesystem (bounded by FS, not one LMDB mutex).
  4. Global queries: shard-select / ShardedConCorpus::select fans out read-only across shards (multi-reader MVCC per shard).

Assign trajectory IDs so traj_id % n_shards == shard_id (CLI shard-ingest advances start-id accordingly). This is partitioned embedded writers, not Raft multi-master — the right pattern for campaign uploads on Lustre/GPFS.

H5MD interchange

collect_h5md / export_h5md emit one cooked [T][N][3] trajectory (position, optional force and velocity). CON text stays the corpus authority. Dest units are Å / ps / kJ mol^{-1} Å^{-1} / Å ps^{-1} (Angstrom ps-1 for MDA 2.10). Node-local ingest then drain_to + join-drained is the campaign write path (see above).

LMDB model decision (HPC)

KEEP sharded LMDB (Heed). Single-env multi-writer contradicts LMDB SWMR; HPC-scale uploads use independent envs per shard (MDHIM-style local backends / industry sharded-LMDB practice). Full rationale, citations (LMDB docs, MDHIM HotStorage’15, PapyrusKV, RocksDB contrast, HDF5/Zarr/ADIOS2/DAOS contrast), risks, and falsifiers: see the architecture note on sharded LMDB. Do not interpret “one write_txn per env” as “LMDB cannot do multi-process writes”—only that partitioning is mandatory for concurrent commits.

Compaction: reversible join / split for analysis

Mode CLI Use
sharded-lmdb shard-init / compact-split HPC multi-writer; parallel rank ingest
single-env-lmdb compact-join / ordinary ingest Laptop analysis; ConCorpus::open_readonly
extxyz compact-export-extxyz [--sharded] External ML tools (non-LMDB)

Join copies CON blobs by traj (ids preserved; duplicate traj across shards errors). Split routes traj_id % n_shards into a new manifest root. Membership is reversible under that routing (indexes rebuilt via normal prepare/commit ingest).