Skip to content

feat: Add RNTuple row extension and field addition via update mode - #1687

Merged
ariostas merged 97 commits into
scikit-hep:mainfrom
Yokubas:Yokubas/rntuple-update-pr
Sep 3, 2026
Merged

ariostas merged 97 commits into
scikit-hep:mainfrom
Yokubas:Yokubas/rntuple-update-pr

Conversation

@Yokubas

@Yokubas Yokubas commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements in-place modification of existing RNTuples:

  • f["name"].extend({"x": array1, "y": array2}) — append new rows to existing RNTuple
  • f["name"].add_fields({"z": np.int32, "w": np.float32, ...}) — add one or more new fields back-filled with zeros
  • f["name"].add_fields({"particle.phi": np.float32, "particle.eta": np.float64, ...}) — add subfields to existing untyped structs
  • f["name"].extend({"x": array1, "z": array2}, accept_new_fields=True) — auto-add new fields and extend

How it works

  • Reads existing RNTuple metadata (anchor, header, footer, page lists)
  • Reconstructs a writable NTuple object from the existing file
  • For row extension: writes new pages and adds new cluster group to footer
  • For field addition: uses RNTuple's deferred column mechanism — new fields are added to the footer's extension records with first_element_index = num_entries, marking where new data starts. Old cluster groups are left completely untouched. The reader automatically zero-pads entries before first_element_index. Subsequent extend calls write new cluster groups that include the new column.
  • Only footer and new data are written — existing data never touched
  • Anchor updated in-place since it's always the same size
  • add_fields/extend on an RNTuple in a subdirectory now resolve their key against that subdirectory, not always the file root
  • add_fields on an RNTuple created earlier in the same session (via mkrntuple or directory assignment, not reopened via uproot.update) now lazily loads the state it needs instead of assuming it was already loaded

Tests

42 tests in tests/test_1687_rntuple_update.py covering:

  • Basic extend and add_fields
  • Multiple extends in separate sessions
  • Multiple fields added in a single call
  • Sequential add_fields calls holding the same object
  • add_fields + extend in same session with same object
  • Variable length arrays
  • Mixed types (scalar + jagged)
  • Empty ntuples
  • Multiple ntuples in same file
  • Subfields and deeply nested subfields with correct parent resolution
  • Validation: nonexistent parent, typed parent, collection parent, wrong field types
  • accept_new_fields kwarg behavior
  • ROOT verification reading actual column values via RNTupleReader
  • add_fields/extend on an RNTuple in a subdirectory (regression test)
  • add_fields on a same-session (not reopened) RNTuple (regression test)
  • Repeated add_fields calls in one session don't duplicate field records (regression test)
  • extend() after add_fields() doesn't scramble which data lands in which column, verified against real ROOT's own RNTupleReader as well as uproot's reader (regression test for the corruption below)
  • add_fields dotted-path parent resolution rejects an ambiguous bare name instead of silently picking the wrong parent (regression test)
  • A field's real (non-root) nesting depth is no longer misidentified as root during multi-level parent resolution (regression test)
  • add_fields's duplicate-field-name check no longer false-positives against an unrelated nested field's name (regression test)
  • add_fields checks column-encoding compatibility before mutating shared state, and that check's error message is no longer masked by an unrelated dotted-path error (regression test)
  • WritableNTuple.extend has a docstring again (regression test — see below)
  • add_fields on a genuine ROOT-written fixture (ntpl001_staff) raises cleanly (regression test)

Blocking issues addressed (from review)

  • Silent data corruption on ROOT-written files → column record comparison raises clear ValueError with encoding mismatch instead of writing garbage data
  • Wrong element_offset for jagged fields → per-column element counts recovered from existing page lists instead of assuming all columns have num_entries elements
  • Multi-cluster groups in add_fields → now fully supported — writes one page per new field per cluster sized to that cluster's entry span
  • Stale in-memory state after add_fields → field records and column counts updated directly in memory; footer/page lists/akform reloaded from file
  • Subfield parent resolved by bare name → walks full dotted path using parent-id chain, correctly handles both uproot and ROOT parent-id conventions
  • Extension field records were double-counted on the second and every subsequent add_fields call in one session (footer's cumulative extension list was re-added on top of an already-updated field list each time) — this could corrupt the file badly enough that a third add_fields call raised IndexError on reopen. Fixed by recombining the header's fixed field list with the footer's cumulative list, instead of the previous call's already-merged result.
  • extend() after add_fields() could silently write each field's data under the wrong column: extend()'s dict-to-awkward conversion always sorts fields alphabetically, but a header reloaded from disk reflects the true on-disk (insertion) field order — awkward's Form equality doesn't check field order, so the mismatch went undetected and columns were written under each other's keys. Confirmed independently against ROOT's own RNTupleReader, not just uproot's reader. Fixed by reordering the data to match the header's actual field order before writing.
  • The dotted-path parent-matching used for single-level "parent.field" paths matched by bare name only, with no check that the match was unique — if two fields (e.g. a top-level record and an unrelated nested record) shared the same bare name, add_fields could silently attach the new field under the wrong one. Now raises a clear "ambiguous" error and suggests a fully-qualified path.
  • Fixed a related bug found while fixing the above: multi-level path resolution's "is this a root field" check (parent_field_id == 0 or parent_field_id == i) wasn't a valid signal on its own — it also matched any non-root field whose real parent happened to sit at index 0 — which could make a legitimate 3+-level dotted path fail to resolve. A field is root iff it's its own parent; fixed to check only that.
  • add_fields's duplicate-field-name check compared a new (always top-level) field's name against every existing field's bare name, including nested ones — a legitimate new top-level field could be rejected as "already exists" if any unrelated nested field happened to share its name. Now only compares against actual top-level fields.
  • add_fields checked column-encoding compatibility only partway through the per-field loop, after already mutating the shared, persistent footer object and after dotted-path resolution could raise a more specific but less fundamental error first — unlike extend(), which checks it immediately. Moved to the top of add_fields, matching extend().
  • Found and fixed a related but currently-unreachable bug in the same area: the column-count recovery used when reopening a file only read the first cluster of each cluster group, undercounting for any group with more than one cluster (not possible via uproot's own writer today, but would silently corrupt element offsets the moment that changes).
  • WritableNTuple.extend's docstring had been accidentally placed after a statement instead of as the function's first statement, silently turning it into a dangling string and dropping it from generated docs — restored, and both extend/add_fields now explicitly document that only uproot-written RNTuples are supported for these operations after reopening a file.

Known limitations

  • ROOT-written RNTuples cannot be extended (split encoding not yet supported) — this restriction is now stated explicitly in the extend/add_fields docstrings, and is covered by a test against a real ROOT-written fixture rather than only a synthetic one
  • Only scalar numeric types supported in add_fields
  • Multi-cluster RNTuples not yet supported for add_fields (a latent bug in the read-back accounting for this case was also fixed defensively — see above — even though it isn't reachable via the current public API)
  • File-like objects not supported (requires file path for re-reading metadata)

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.45%. Comparing base (4a41da3) to head (3b1037d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/uproot/writing/writable.py 94.73% 5 Missing and 6 partials ⚠️

❌ Your patch check has failed because the patch coverage (95.45%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
Files with missing lines Coverage Δ
src/uproot/writing/_cascade.py 87.44% <100.00%> (+0.01%) ⬆️
src/uproot/writing/_cascadentuple.py 90.12% <100.00%> (+1.16%) ⬆️
src/uproot/writing/writable.py 84.51% <94.73%> (+2.53%) ⬆️

... and 5 files with indirect coverage changes

@ariostas ariostas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @Yokubas, this is fantastic progress! I left a few comments.

Comment thread src/uproot/writing/writable.py Outdated
Comment thread src/uproot/writing/writable.py Outdated
Comment thread src/uproot/writing/writable.py Outdated
Comment thread src/uproot/writing/writable.py Outdated
Comment thread src/uproot/writing/writable.py
Comment thread tests/test_1687_rntuple_update.py
Comment thread tests/test_rntuple_update.py Outdated
Comment thread tests/test_1687_rntuple_update.py
Comment thread src/uproot/writing/_cascadentuple.py Outdated
Comment thread src/uproot/writing/writable.py Outdated
Yokubas added 15 commits August 24, 2026 13:05
…ontaining directory, not always the file root
…just the first, when recovering column counts
…nd/add_fields only support uproot-written RNTuples after reopening
…plicate-name check to top-level fields, and check column encoding compatibility before mutating shared state
Comment thread tests/test_1687_rntuple_update.py
@ariostas

Copy link
Copy Markdown
Member

This is looking great! I left a comment with a more minor thing, but everything seems to be working well.

…-typed RNTuples, instead of a confusing internal assertion
Comment thread tests/test_1687_rntuple_update.py
Yokubas and others added 2 commits August 27, 2026 18:14
Resolves conflicts with scikit-hep#1690 (in-place TTree branch addition and row
extension), which landed on main and touches the same regions of
writable.py.

The two features are complementary rather than overlapping:

- WritableDirectory._get: main routes preexisting TTrees to
  _load_existing_ttree, this branch routes preexisting RNTuples to
  _load_existing_ntuple. Both branches kept; each side's "cannot view
  preexisting" TypeError is now dead and removed.

- The new methods between _get and _del: main added
  _load_existing_ttree, this branch added _read_ntuple_envelope and
  _load_existing_ntuple. Git interleaved them because they share
  boilerplate (sink.flush, _get_chunk, _ReadForUpdate). All three kept
  verbatim from their respective sides.

Full test suite passes (1106 passed, 94 skipped), including both
tests/test_1687_rntuple_update.py and tests/test_1690_ttree_inplace.py.

Assisted-by: claude-code:claude-opus-5[1m]
@ariostas

ariostas commented Sep 3, 2026

Copy link
Copy Markdown
Member

I added a commit to resolve conflicts with main.

@ariostas ariostas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the hard work, @Yokubas! It looks great. This is some really useful functionality to have.

@ariostas
ariostas merged commit 532baf7 into scikit-hep:main Sep 3, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/feat PR title type: feat (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants