Skip to content

docs: reconcile arXiv benchmark figures after post-fix review - #348

Draft
cdeust wants to merge 2 commits into
mainfrom
agent/issue-347-post-fix-docs
Draft

docs: reconcile arXiv benchmark figures after post-fix review#348
cdeust wants to merge 2 commits into
mainfrom
agent/issue-347-post-fix-docs

Conversation

@cdeust

@cdeust cdeust commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Closes #347

What changed

  • Align the README LoCoMo headline with the canonical post-fix run: Recall@10 94.35%, MRR 0.8279, n=1,986, code SHA 2f45bcb.
  • Add a supersession warning to the pre-fix results document.
  • Resolve the provenance of the historical 0.794 / 0.926 comparator and label it explicitly as noncanonical.
  • Align the current LongMemEval headline on the clean July run: Recall@10 98.2%, MRR 0.9167, n=500, code SHA 28145f0; retain 98.4% / 0.9124 only as the named historical E1 v3 ablation snapshot.
  • Label BEAM 0.591 as a five-conversation, n=100 retrieval-proxy MRR at code SHA a071d89 and remove the invalid comparison with the paper end-to-end metric.
  • Align the thermodynamic and context-assembly arXiv sources and all publication-facing appendices, then regenerate both PDFs.
  • Add a versioned audit of the arXiv-facing figures and their sources.

Opus 5 review follow-up

Resolved the residual stale 92.6% in the Popper appendix, the LongMemEval 98.4 / 98.2 / 97.8 split, and the missing BEAM metric/protocol audit identified in the issue review.

Why

The public documents mixed current post-fix results with historical or protocol-specific measurements. This made the arXiv draft ambiguous and, for BEAM, risked comparing incommensurable retrieval-proxy and end-to-end metrics.

Validation

  • scripts/check_doc_claims.py
  • git diff --check
  • Relative Markdown-link validation for all changed Markdown files
  • Exact artifact/provenance assertions for LoCoMo and LongMemEval
  • Exact 14-row LoCoMo table parity across the post-fix result, Markdown paper, and LaTeX paper
  • Clean LaTeX builds for both papers, with no undefined citations or references
  • PDF text extraction confirming 98.2%, 94.35%, and 0.591 with their metric framing

@cdeust cdeust changed the title docs: align arXiv figures with post-fix LoCoMo docs: reconcile arXiv benchmark figures after post-fix review Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Three different LoCoMo number pairs published across README / docs/benchmarks / CLAUDE.md-referenced baseline — reconcile before arXiv submission

1 participant