Surprise

Research code for testing whether causal predictive surprise can coordinate adaptive computation in sequence models. The repository contains a lightweight NumPy research scaffold, synthetic experiments for segmentation and memory, and the V13 orchestration used to audit a long-horizon token-level compute-allocation study.

Paper result: under the audited V13 architecture and protocol, observed surprisal did not establish a preregistered endpoint advantage and the learned budget policy became nearly constant. The sealed final test was not opened.

Research question

A surprising token is one that the model assigned low probability. Surprise is causal and easy to measure, but it is not automatically the same as the counterfactual value of spending more computation. This project separates three possible uses of predictive state:

1. detecting sequence boundaries;
2. deciding when to write memory; and
3. allocating extra computation to selected positions.

The accompanying preprint tests the third branch with matched controls.

V13 study at a glance

• seven experimental arms;
• three paired warm starts;
• 50,000 adaptation steps per arm;
• 21 long-horizon training runs;
• 84 complete-development evaluations;
• primary dynamic-surprisal arm: 1.958168823 BPB at 50k;
• pure backbone: 1.956480433 BPB at 50k;
• no required endpoint comparison passed;
• mechanism rate CV: 0.000325419 versus required 0.01;
• surprisal-to-next-rate correlation: -0.018967185 versus required 0.10;
• sealed final test used: false.

See docs/RESULTS_SUMMARY.md for the scoped numerical summary and paper/ for the preprint.

Quick start: public NumPy smoke experiment

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e ".[dev]"
python scripts/train_synthetic.py --config configs/smoke.yaml
pytest -q

The smoke experiment writes generated metrics under runs/, which is ignored by Git.

Repository layout

surprise/       NumPy research scaffold and controllers
configs/        small public smoke configurations
scripts/        synthetic experiments and V13 orchestration
slurm/v13/      V13 SLURM workflow scripts
paper/          article-format preprint PDF
docs/           results scope and reproducibility status
tests/          public core tests

Reproducibility boundary

This public export does not contain the exact frozen V13 surprise_torch source snapshot, generated experiment configs, checkpoints, enwik8 bytes, complete result records, or canonical release manifests. Therefore it is not a complete end-to-end reproduction bundle for the paper. The included V13 scripts are useful for audit and workflow inspection but depend on artifacts not present here. Read docs/REPRODUCIBILITY_STATUS.md before making reproducibility claims.

Compute and data safety

Do not commit raw datasets, model checkpoints, cluster logs, credentials, or personal HPC paths. The .gitignore excludes common large artifacts. enwik8 is not redistributed in this repository.

Acknowledgment

The numerical calculations reported in the accompanying paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).

Citation

Use CITATION.cff. Add the Zenodo DOI to the citation metadata once the record has been published.

License

No open-source license has yet been granted. See LICENSE_NOTICE.md. Public visibility on GitHub should not be interpreted as permission for reuse or redistribution.