Compression is prediction, and the compression ratio is the loss function of a schema.
schema-codec is a domain-conditioned compression library built on the Minimum
Description Length principle: given a shared prior (a schema) held by both
encoder and decoder, you don't transmit the data — you transmit only the part the
schema failed to predict. The residual. The surprise.
encode(data, schema) -> residual # what the schema didn't predict
decode(residual, schema) -> data # schema's prediction + residual
The size of the residual is a measurement of how well the schema modeled reality. The codec is therefore also a ruler for schema quality.
- Never worse than plain zstd. Every frame emits the smallest of three
candidates —
zstd(residual),zstd(raw),raw— so a mispredicting or drifted schema can never beat plain zstd by more than an 11-byte header. Property-tested under fuzzing. - Lossless round-trip for any data and any valid schema, including a null schema.
- Schema-hash handshake. A frame carries a CRC32 of the prior; a decoder holding the wrong prior rejects the residual and falls back rather than producing garbage.
- Causality invariant. A streaming schema predicts from already-decoded bytes only — no lookahead — guaranteeing deterministic decode.
Not encryption. A residual is opaque without the schema, but that is obfuscation, not confidentiality (see CRIME/BREACH). Wrap real crypto (WireGuard / TLS /
age) around the payload if you need secrecy.
The prior has to be strong. It wins big where the prior is exact — the receiver already holds a near-identical version — and ties plain zstd where it isn't.
Verified on a real corpus (fleet-memory delta-sync — receiver holds the previous version of each file, sender ships the next):
TOTAL raw 1,126,962 zstd 579,668 schema-codec 11,738
schema-codec vs plain zstd: +97.98% (0.0202x the size)
That is not a claim about arbitrary data — it is the delta-sync case, where you pay for the change, not the file. On data with no useful prior, the floor guarantee holds and you get plain zstd.
Process/swap memory: measured, thin. With an honest disjoint train/test split, a
learned dictionary prior gave +1.5% to −0.3% over zstd, dedup +0.4% to +3.7%,
and blind pointer transposition −33%. The prior over heterogeneous process memory
is too weak. The boundary is the finding: schema-as-loss needs an exact prior, not a
statistical one. See scripts/ for the benchmarks.
| Layer | File | Role |
|---|---|---|
| Predictor | schema_codec/schema.py |
Schema interface, PredictorKind, Expectation |
| Header | schema_codec/header.py |
11-byte frame + schema-hash handshake |
| Residual | schema_codec/residual/ |
Bitwise (XOR), SequenceDelta (edit script), Probabilistic (range coder) |
| Coder | schema_codec/coder.py |
HybridCoder — three-candidate floor |
| Telemetry | schema_codec/telemetry.py |
Loss ratio + schema-drift detection |
v1 Python prototype — thesis validation. A native (Rust/C, armv6-clean) port is a v2
decision, made only once the ratios justify it. Design spec and implementation plan
live under docs/superpowers/.
python3 -m pip install -e ".[dev]"
python3 -m pytest -q
PYTHONPATH=. python3 benchmarks/bench_fleet_memory.pyAGPL-3.0.