Skip to content

perf: build the RNTuple form once and prune it per field - #1723

Open
kadircanyildirm-crypto wants to merge 2 commits into
scikit-hep:mainfrom
kadircanyildirm-crypto:fix/1637-cache-default-akform
Open

kadircanyildirm-crypto wants to merge 2 commits into
scikit-hep:mainfrom
kadircanyildirm-crypto:fix/1637-cache-default-akform

Conversation

@kadircanyildirm-crypto

Copy link
Copy Markdown

Part of #1637.

to_akform rebuilt the form from the field records on every call, so reading a wide RNTuple one column at a time — which is what coffea does — rebuilt the whole form once per column. For the 16-event, 1491-field NanoAOD skim in the issue that is 1579 builds.

This builds the unfiltered form once per RNTuple, keeps it there, and serves later requests by pruning it to the requested keys. A field takes the same cached form, walks down to its own path and prunes from there, so the per-column path comes off the cache as well — that is the path coffea actually uses, since it asks fields rather than the ntuple. For a field under a collection the form is still built around the outermost collection on its path and field_path is returned as before; ak_add_doc still takes the original path.

Measured on main (c979803), Windows, Python 3.12, awkward 2.13.0, coffea 2026.7.0, reading the skim back through NanoEventsFactory:

before after
read-back 252.0 s 32.1 s / 36.2 s
full form builds 1579 1

The same data written as a TTree reads in 3.8–4.5 s, so this does not close the gap to TTree by itself. What is left is a different effect, measured and described in #1637 (comment).

Checks:

  • all 1491 leaf columns are bit-identical to the ones built the old way (ak.to_buffers, compared byte for byte)
  • pytest -k "rntuple and not dask": 146 passed, 0 failed — 143 on main, plus the 3 added here
  • full suite identical to main: 1032 passed, 29 failed, 117 skipped either way, the 29 all in the dask test files because distributed is missing in my environment
  • the added test asserts that 200 field requests cause one form build, and that the pruned form equals the rebuilt one for filters, subfields and ak_add_doc

Disclosure, per CONTRIBUTING: this was written with Claude Code. The measurements above are from runs on my own machine against this branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6b4SK8hGvDHGAx1W6KqGJ

`to_akform` rebuilt the form from the field records on every call. Reading a
wide RNTuple one column at a time, which is what coffea does, therefore rebuilt
the whole form once per column: 1579 builds for the 1491-field NanoAOD skim in
scikit-hep#1637, and 252 s for a read that takes 4 s from the same data in a TTree.

Build the unfiltered form once per RNTuple, keep it there, and serve later
requests by pruning that form to the requested keys. A field takes the same
cached form, walks down to its own path and prunes from there, so the
per-column path comes off the cache as well. For a field under a collection the
form is still built around the outermost collection on its path and
`field_path` is returned as before; `ak_add_doc` still takes the original path.

The read is now 32-36 s with a single form build, and all 1491 leaf columns are
bit-identical to the ones built the old way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6b4SK8hGvDHGAx1W6KqGJ
@github-actions github-actions Bot added the type/perf PR title type: perf (set automatically) label Sep 17, 2026
@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.89888% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.38%. Comparing base (c979803) to head (8bf0c57).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/uproot/behaviors/RNTuple.py 80.89% 10 Missing and 7 partials ⚠️

❌ Your patch check has failed because the patch coverage (80.89%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
Files with missing lines Coverage Δ
src/uproot/behaviors/RNTuple.py 74.69% <80.89%> (-0.09%) ⬇️

codecov flagged the patch at 80.9%. Two of the uncovered paths turned out to be
dead: pruning only ever returns "nothing survives" for the top-level call with
an empty selection, never for a field below it, because a field is visited only
when its own path is in the keep set. Handle the empty selection where it
happens, in `to_akform`, and let `_prune_akform` always return a form. `exact`
then has no reader left, so it goes too.

The rest is covered by tests: an empty selection, the tuple and
subfields-of-a-record shapes from the STL-container and nested-struct files,
walking to a field that isn't in the form, and the fallback for a form the walk
cannot follow.

Swept for equivalence while doing it: every field of 11 RNTuple test files plus
four filter shapes, 1453 comparisons of pruned against rebuilt, no differences.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6b4SK8hGvDHGAx1W6KqGJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/perf PR title type: perf (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant