feat: Add RNTuple row extension and field addition via update mode - #1687
Merged
Merged
Conversation
…s/uproot5 into Yokubas/rntuple-update-pr
…structing full header
…s/uproot5 into Yokubas/rntuple-update-pr
…s/uproot5 into Yokubas/rntuple-update-pr
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (95.45%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files
|
…s/uproot5 into Yokubas/rntuple-update-pr
ariostas
reviewed
Jul 21, 2026
ariostas
reviewed
Aug 19, 2026
ariostas
reviewed
Aug 19, 2026
…_fields() calls in one session
…ontaining directory, not always the file root
…ed earlier in the same session
…riting column buffers
…just the first, when recovering column counts
…nd/add_fields only support uproot-written RNTuples after reopening
…plicate-name check to top-level fields, and check column encoding compatibility before mutating shared state
…cated RNTuple deserialization boilerplate
… the resolved parent's existing children
…m in extend, not just top-level fields
ariostas
reviewed
Aug 26, 2026
Member
|
This is looking great! I left a comment with a more minor thing, but everything seems to be working well. |
…-typed RNTuples, instead of a confusing internal assertion
ariostas
reviewed
Aug 27, 2026
This was referenced Aug 27, 2026
Resolves conflicts with scikit-hep#1690 (in-place TTree branch addition and row extension), which landed on main and touches the same regions of writable.py. The two features are complementary rather than overlapping: - WritableDirectory._get: main routes preexisting TTrees to _load_existing_ttree, this branch routes preexisting RNTuples to _load_existing_ntuple. Both branches kept; each side's "cannot view preexisting" TypeError is now dead and removed. - The new methods between _get and _del: main added _load_existing_ttree, this branch added _read_ntuple_envelope and _load_existing_ntuple. Git interleaved them because they share boilerplate (sink.flush, _get_chunk, _ReadForUpdate). All three kept verbatim from their respective sides. Full test suite passes (1106 passed, 94 skipped), including both tests/test_1687_rntuple_update.py and tests/test_1690_ttree_inplace.py. Assisted-by: claude-code:claude-opus-5[1m]
Member
|
I added a commit to resolve conflicts with main. |
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements in-place modification of existing RNTuples:
f["name"].extend({"x": array1, "y": array2})— append new rows to existing RNTuplef["name"].add_fields({"z": np.int32, "w": np.float32, ...})— add one or more new fields back-filled with zerosf["name"].add_fields({"particle.phi": np.float32, "particle.eta": np.float64, ...})— add subfields to existing untyped structsf["name"].extend({"x": array1, "z": array2}, accept_new_fields=True)— auto-add new fields and extendHow it works
first_element_index = num_entries, marking where new data starts. Old cluster groups are left completely untouched. The reader automatically zero-pads entries beforefirst_element_index. Subsequentextendcalls write new cluster groups that include the new column.add_fields/extendon an RNTuple in a subdirectory now resolve their key against that subdirectory, not always the file rootadd_fieldson an RNTuple created earlier in the same session (viamkrntupleor directory assignment, not reopened viauproot.update) now lazily loads the state it needs instead of assuming it was already loadedTests
42 tests in
tests/test_1687_rntuple_update.pycovering:WritableNTuple.extendhas a docstring again (regression test — see below)ntpl001_staff) raises cleanly (regression test)Blocking issues addressed (from review)
ValueErrorwith encoding mismatch instead of writing garbage datanum_entrieselementsadd_fieldscall in one session (footer's cumulative extension list was re-added on top of an already-updated field list each time) — this could corrupt the file badly enough that a thirdadd_fieldscall raisedIndexErroron reopen. Fixed by recombining the header's fixed field list with the footer's cumulative list, instead of the previous call's already-merged result.extend()afteradd_fields()could silently write each field's data under the wrong column:extend()'s dict-to-awkward conversion always sorts fields alphabetically, but a header reloaded from disk reflects the true on-disk (insertion) field order — awkward's Form equality doesn't check field order, so the mismatch went undetected and columns were written under each other's keys. Confirmed independently against ROOT's ownRNTupleReader, not just uproot's reader. Fixed by reordering the data to match the header's actual field order before writing."parent.field"paths matched by bare name only, with no check that the match was unique — if two fields (e.g. a top-level record and an unrelated nested record) shared the same bare name,add_fieldscould silently attach the new field under the wrong one. Now raises a clear "ambiguous" error and suggests a fully-qualified path.parent_field_id == 0 or parent_field_id == i) wasn't a valid signal on its own — it also matched any non-root field whose real parent happened to sit at index 0 — which could make a legitimate 3+-level dotted path fail to resolve. A field is root iff it's its own parent; fixed to check only that.add_fields's duplicate-field-name check compared a new (always top-level) field's name against every existing field's bare name, including nested ones — a legitimate new top-level field could be rejected as "already exists" if any unrelated nested field happened to share its name. Now only compares against actual top-level fields.add_fieldschecked column-encoding compatibility only partway through the per-field loop, after already mutating the shared, persistent footer object and after dotted-path resolution could raise a more specific but less fundamental error first — unlikeextend(), which checks it immediately. Moved to the top ofadd_fields, matchingextend().WritableNTuple.extend's docstring had been accidentally placed after a statement instead of as the function's first statement, silently turning it into a dangling string and dropping it from generated docs — restored, and bothextend/add_fieldsnow explicitly document that only uproot-written RNTuples are supported for these operations after reopening a file.Known limitations
extend/add_fieldsdocstrings, and is covered by a test against a real ROOT-written fixture rather than only a synthetic one