Conversation
Mirrors uproot.dask / dask_write for the graphed task-graph system:
- src/uproot/_graphed.py: uproot.graphed(files, library="ak") returns a deferred graphed Array
(graphed-awkward backend). Metadata-only construction (TTree form via typetracer; no event data
read); compute() reads lazily via uproot, and graphed's necessary-buffer projection means only the
TBranches the analysis touches are read (the dask-awkward column-projection analogue). Plus
necessary_columns() and compute(). graphed imported lazily so importing uproot needs no graphed.
- src/uproot/writing/_graphed_write.py: graphed_write(array, dest, tree_name=...) — compute (with
projection) + write to a TTree, the dask_write analogue.
- surfaced as uproot.graphed / uproot.graphed_compute / uproot.necessary_columns / uproot.graphed_write.
- tests/test_graphed_{for_awkward,column_projection,write}.py: same structure as uproot's dask tests
(skhep_testdata + importorskip), asserting graphed reads match plain uproot bit-for-bit, projection
reads only needed branches (over-touch protection), write round-trips, library='np' -> NotImplemented.
- .github/workflows/graphed.yml: runs the graphed tests on the graphed-mvp branch (installs the
graphed-*-mvp siblings); does not collide with uproot's build-test matrix.
- gated via the graphed orchestrator (scripts/graphed_advance.py + .graphed/), milestone UPROOT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…a python -m pytest Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ressions) Install the full uproot dev dependency-group + the graphed-*-mvp siblings and run all of tests/ (uproot + graphed) with upstream's rerun flags, instead of only the graphed test files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er in this job) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…otd') Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ors, not a .compute() shim Drop the dask-mimicking compute()/graphed_compute convenience. A uproot-read graphed analysis is now executed the way a task graph actually runs: per-partition through graphed_exec_local's ProcessExecutor (and ThreadExecutor) with tree reduction — the path a deferred-array .compute() hides. - _graphed.py: remove compute(); add graphed_partitions(files, steps_per_file) -> [Task(Partition...)]. - writing/_graphed_write.py: inline the projection+materialize it needs (no uproot.compute dependency). - __init__: surface uproot.graphed_partitions; drop uproot.graphed_compute. - tests/graphed_uproot_analysis.py: picklable process/combine/empty reading a Partition via open_once, recording + materializing the analysis on the chunk (mirrors the M7 adl.py executor glue). - tests/test_graphed_executor.py: run via ProcessExecutor + ThreadExecutor; bit-for-bit vs single-pass; invariant to partition count; non-vacuous; n_combines. - trim the read/projection tests to construction-time (metadata-only) + projection-minimality checks. - graphed.yml: also install graphed-exec-local-mvp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… on it) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…xecutor (mirror dask_write)
graphed_write now builds a write task graph instead of materializing one array:
- each output partition is a side-effecting write task (reads its chunk via open_once, writes
{prefix-}part{N}.root, returns nothing) -> a graphed_core.Plan.
- compute=False returns the Plan (the write tasks), writing nothing; compute=True runs it through a
graphed-exec-local executor (ProcessExecutor default, executor="thread" for the pool).
- destination is a directory of part files; steps_per_file splits each TTree; prefix names the files.
Also fixes uproot.graphed for RNTuple (HasFields): use filter_field (not filter_branch) for keys and
to_akform for the form -- surfaced because uproot writes RNTuples for `file[name] = dict`.
Tests (mirror the dask_write cases): one file per partition + contents, prefix, compute=False returns
the task graph and writes nothing (then runs), compute=True via process AND thread executors,
multi-file topology, Zmumu roundtrip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… steps) graphed_partitions now takes step_size / steps_per_file / open_files with uproot.dask's rules: - step_size is incompatible with open_files=False and mutually exclusive with steps_per_file (TypeError); - open_files=True opens each file for exact entry ranges (step_size in entries or a memory string); - open_files=False is BLIND: files are not opened here; each chunk records (step_index, n_steps) as entry_start / -entry_stop and its real range is resolved against the file's own count at read time. New shared reader uproot.read_graphed_partition(partition, columns, tree=...) resolves blind partitions; the executor helper and graphed_write's write task both use it. tests/test_graphed_blind_steps.py mirrors test_0876: the TypeError rules + every step_size/steps_per_file/ open_files combo, blind vs eager, all reduce to the single-pass histogram bit-for-bit (38 graphed tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n_resumable Wire uproot partitions into graphed's own error-harvesting machinery (graphed semantics, not a copy of dask's report tuple): a DurablePlan runs each Partition through a uproot read+histogram; a missing/broken file raises and run_resumable harvests it into the content-addressed dead-letter set (ResumeReport.dead / dead_letters) while the good partitions still reduce. - tests/graphed_uproot_report.py: module-level process/hist_add/hist_zero + build_plan (DurablePlan with OpSpec.from_ref, a small representative IR for content-addressed task_ids). - tests/test_graphed_report.py: bad file -> dead-lettered (reproducible task_id) + good partitions reduce; clean run has no dead letters; kill-then-resume skips completed (no double-count, bit-for-bit, less work); error budget stops the run. - graphed.yml: also install graphed-checkpoint-mvp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…erved plan at alternate inputs - tests/test_graphed_preserve.py: build a graphed-preserve Bundle of a uproot-read analysis (embeds the events array + canonical IR). reproduce() recomputes the histogram bit-for-bit from references alone (incl. a clean reload via Bundle.open with no original file); inspect() renders without executing and a removed payload raises UnresolvedPayload; self-fingerprinting (same data -> same fingerprint, different data -> different). - Demonstrate the PRESERVED PLAN runs on alternate inputs: a DurablePlan's canonical IR is shared, so with_partitions re-targets the same analysis at a different file location and at a different number of partitions (compile once, run on N datasets) -- both reproduce the single-pass result. - graphed.yml: also install graphed-preserve-mvp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ass blind partitions, write hardening
Applies the graphed M10 remediations (superproject mvp-shortcomings.md) to the
integration branch (freeze-UPROOT-1; see .graphed/UPROOT/attempts.md):
- A.2: graphed_uproot_analysis.process no longer builds a Session and
re-records the analysis per partition — the analysis is compiled once per
worker (graphed.compile_ir, module-cached) and each partition evaluates the
reduced serialized IR (graphed.evaluate_ir). Bit-for-bit equality with the
single pass re-pinned.
- A.3: uproot.necessary_buffers + uproot.resolve_read_branches — a count-only
analysis reports {branch: OFFSETS} (where necessary_columns reports the
empty set) and is served from the jagged branch's COUNTER branch (NMuon for
Muon_Px) without reading the payload baskets.
- C.9: graphed_partitions(open_files=False) emits first-class
graphed_core.Partition.blind chunks; read_graphed_partition resolves them
(legacy negative-entry_stop sentinel still honored for pre-M10 plans). The
one frozen-era test pinning the sentinel ENCODING amended under
freeze-UPROOT-1 (same files-not-opened intent, honest representation).
- C.9: graphed_write uses the public Session.sources() accessor, rejects
multi-source arrays loudly, writes only the array's projected branches, and
ships an O(#files) base-index table instead of pickling the whole
per-partition path map into every task (part{N} naming unchanged).
New suite tests/test_graphed_m10.py (11 tests). Local: all 70 graphed tests
green; full uproot suite (xrootd-deselected) 644 passed, 1 pre-existing
environment-only dask failure also present on the base commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- graphed_to_parquet: ROOT in, parquet out, partition by partition — blind partitions, compiled-IR evaluation per partition, specialized on the new graphed.write base, ProcessExecutor default, disabled==enabled write consistency, multi-source rejected. The read list is the graph's SYNTACTIC source-field accesses (the buffer projection under-supplies evaluation: a zip's pz/E legs are replayed even when only .pt is consumed). - uproot.graphed(behavior=...): awkward behaviors (vector Momentum4D) record, evaluate, and project to exactly the branches a property reads; process workers accept an importable module:attr behavior reference (behavior dicts contain lambdas and do not pickle). - uproot.graphed_head: first file's leading entries only, projected branches only, through the compiled IR — witnessed by corrupting every later file. - Fusion witness: per-event (axis=1) reductions live INSIDE stages (graphed M16), with the frozen M4 SingleUse diamond pin intact; maximal fusion collapses to one stage. - Inherited-surface pins over uproot sources: record-subset getitem, axis-0 slices, the ufunc tier, structure ops. - graphed_write deliberately stays OFF the write base: its None-returning tasks and empty-range skipping are frozen pins predating it (documented; alignment = a recorded freeze bump). 19 new tests (3 files); the existing frozen set untouched; 78/78 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rquet removed for the generic entry point (freeze-UPROOT-2) USER-AUTHORIZED frozen amendments to test_graphed_write.py (named in the confirmed plan): workers report their part paths (was: return None); part names follow the base's part_path (part-00000.root); docstrings updated. Freeze tag bumped freeze-UPROOT-1 -> freeze-UPROOT-2. - uproot.graphed_to_parquet is REMOVED: graphed_awkward.io.to_parquet(uproot.graphed(...), ...) is the same functionality through ONE generic writer — _GraphedTTreeSource implements graphed.write.PartitionedSource (blind partitions, open-once partition reads). Efficiency is witnessed: the whole-dataset loader never runs, planning opens no files, read lists pinned. - uproot.graphed_write specializes the graphed.write base: blind partitions (no driver file opens), path-reporting write tasks, base naming, blind_part_index; empty resolved steps are skipped (no empty part files; numbering may gap in that corner case). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…=) — graphed-core removed mark_output USER-AUTHORIZED (graphed-core freeze-M22-1): outputs are per compile request; bytes unchanged for the helper's single-output graph. 78/78 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed branches (freeze-UPROOT-4)
Per-query gak.zip with with_name + vector behaviors (no schema layer): form-correct recording,
truthful projection (px -> exactly {Muon_pt, Muon_phi}), record+record four-vector sums,
jagged-integer-array getitem, and the capstone TTree -> collection -> behavior property ->
hist.graphed fill through a spawned process pool with the behavior by import ref.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 8 graphed-*-mvp prototype packages are now one pip-installable distribution
`graphed`. Rewrite the merged import roots graphed_{core,awkward,debug,numpy,
checkpoint,preserve} -> graphed.{...} in _graphed.py, the write hook, and the
graphed test suite; the frontend `graphed`, graphed_exec_local, and
graphed_orchestrator stay separate/unchanged.
CI graphed.yml installs the consolidated `graphed[awkward,preserve]` (one git URL
in place of the CORE/DEBUG/FRONTEND/AWKWARD/CHECKPOINT/PRESERVE siblings) plus
graphed-exec-local; checkpoint's cloudpickle is a base dep, preserve is an extra.
Assisted-by: ClaudeCode:claude-opus-4.8
…ecation) graphed-exec-local deprecated ProcessExecutor on 2026-06-17 (7855a9d, split into ProcessPoolExecutor + PinnedPoolExecutor); it now emits a DeprecationWarning, which this suite's `filterwarnings = error` turns into a failure. ProcessExecutor was a same-behaviour subclass of ProcessPoolExecutor, so swap to the parent in the write helper's executor map (`"process"` key unchanged) and across the graphed tests. Assisted-by: ClaudeCode:claude-opus-4.8
…graphed-executors rename) The reference executor package was renamed graphed-exec-local -> graphed-executors with the executor under graphed_executors.local. Migrate the graphed src glue + tests to the new import path and point CI at the renamed repo. Assisted-by: ClaudeCode:claude-opus-4.8
The fork's FIRST tests/frozen tree. graphed_write today copies source TBranches verbatim (no compile_ir/evaluate_ir), so a graphed graph carrying a DERIVED column loses it on write. These frozen tests write derived columns via graphed_write, read back with plain uproot.open, and assert the derived column round-trips — FAILING against today's verbatim copy and PASSING once m51/R1 adds per-partition IR evaluation. Non-varied only; varied ROOT write-out is Phase-2. Discriminating: 3 derived-column tests fail today (derived field absent), 1 plain-source test passes today (positive control proving the write/read harness works). Assisted-by: ClaudeCode:claude-opus-4.8
_write_partition now compiles the recorded array once in the driver and evaluates the compiled IR per partition (mirroring graphed_head), so a derived column — a field absent from the source branches — is materialized and written, instead of the pre-m51 verbatim source-branch copy. Read list switches to the SYNTACTIC _evaluation_columns (buffer projection under-supplies evaluation, plan §6.4f). A bare (non-record) expression evaluates to a fieldless array and falls back to its source columns, keeping the pre-m51 behavior (strict superset; fixes the m10 projected-write regression). graphed.yml: m51-vary push trigger + isolated frozen-acceptance step. Frozen 4/4 green + unmodified; 100% frozen diff coverage; ruff clean; deterministic. black/mypy N/A for this fork (untyped module; black baseline non-conformant, absent from CI). Assisted-by: ClaudeCode:claude-opus-4.8
…-up) The fork's first gated-pipeline milestone gains its CI gates (they ride this PR, not deferred infra): - Diff-coverage gate: scripts/diff_coverage_gate.py (stdlib only, no diff-cover dep) requires >=90% line+branch coverage on the m51-changed lines of _graphed_write.py vs origin/graphed-mvp. Whole-module coverage is 79% (pre-existing untested paths), so the gate is diff-scoped. Result 9/9=100%; proven non-vacuous (55.6%, fails, from the frozen suite alone — spawn workers are invisible to plain coverage). tests/extra/m51 drives the worker body and the bare-expr fallback in-process via executor="thread". - Scoped [tool.mypy]: check_untyped_defs + ignore_missing_imports over the m51 source and its test trees — meaningful (bodies checked) without fork-wide --strict on untyped upstream. One boundary fix (evaluated: Any, since evaluate_ir is typed list[object]). Clean; non-vacuous (caught 3 errors pre-fix). - ruff scripts/* T20 ignore (mirrors dev/*); fetch-depth:0 + mypy/coverage steps in graphed.yml. DoD matrix (ubuntu/3.11-3.12) and push trigger recorded as accepted reduced scope in .graphed/m51/attempts.md. Frozen 4/4 unchanged; git diff freeze-m51 -- tests/frozen/ empty. Assisted-by: ClaudeCode:claude-opus-4.8
…ork review MED)
The §6.4f choice of _evaluation_columns (syntactic) over necessary_columns
(buffer projection) was load-bearing but unwitnessed — swapping it back was
caught by zero tests. Add tests/extra/m51 starve witness: zip({"a": x, "b":
y})[["a"]] keeps only field a, so necessary_columns={x} but the zip
syntactically reads y so _evaluation_columns={x,y}; the round-trip asserts
output fields=={a}. Proven discriminating: under necessary_columns the workers
raise "no field named 'y'" (starve), under _evaluation_columns it passes.
Frozen untouched (4/4; diff empty); coverage/mypy/ruff stay green.
Assisted-by: ClaudeCode:claude-opus-4.8
The gate ran python -m coverage without installing it (the mypy step installs mypy inline the same way); No module named coverage failed the step on 3.11/3.12. Passed locally only because the dev venv had coverage. Assisted-by: ClaudeCode:claude-opus-4.8 Claude-Session: https://claude.ai/code/session_01CEvu5gzCDFKYa5EntyXmDi
The full uproot suite hard-failed on test_*_s3 (FileNotFoundError on the public pivarski-princeton picoDst.root, unreachable in CI) — external data rot, not a graphed regression (995 uproot tests pass, 0 graphed/m51 fails). All 4 s3 tests are @pytest.mark.network; excluded on the same basis as xrootd (infra this lightweight job does not provision). Local-http tests keep running with their existing rerun-on-transient logic. Assisted-by: ClaudeCode:claude-opus-4.8 Claude-Session: https://claude.ai/code/session_01CEvu5gzCDFKYa5EntyXmDi
…on branch Once m51-vary merged into graphed-mvp, the m51 lines in _graphed_write.py became part of the graphed-mvp baseline, so diffing the source against origin/graphed-mvp yields an empty diff and the gate's own empty-diff guard false-fails every push to the integration branch (7 tests still pass; only the PR-oriented gate misfires). The gate is a feature-branch concept; guard it to non-graphed-mvp refs so it keeps protecting future feature branches while the frozen suite + mypy + full suite continue to guard graphed-mvp. Assisted-by: ClaudeCode:claude-opus-4.8 Claude-Session: https://claude.ai/code/session_01CEvu5gzCDFKYa5EntyXmDi
lgray
marked this pull request as draft
September 14, 2026 14:21
The graphed.yml workflow, the .graphed/ tracking directory and the scripts/ CLI helpers belong to the graphed-project pipeline, not to uproot; the graphed hooks ride uproot's own CI instead. The pyproject config that only they used goes with them (the scripts/* ruff per-file-ignore and the [tool.mypy] block, whose sole runner was graphed.yml -- upstream runs no mypy). Assisted-by: ClaudeCode:claude-opus-5
tests/frozen/ and tests/extra/ were the graphed-project pipeline's own layout; upstream has one flat tests/ directory. The two modules move to tests/test_graphed_derived_columns.py and tests/test_graphed_write_thread_and_fallback.py, matching the other tests/test_graphed_*.py, and now run under tests/conftest.py like every other test rather than with --noconftest. Docstrings lose their references to the pipeline trees that no longer exist. Assisted-by: ClaudeCode:claude-opus-5
Every graphed test is guarded by pytest.importorskip, so without this the hooks were silently skipped in CI. graphed requires Python >=3.11, hence the marker; the >=0.0.2 floors keep a lowest-versions resolve off 0.0.1. The extras are the ones the tests import (graphed.awkward, and pyarrow for the to_parquet test); vector is used by the NanoAOD behaviour tests. graphed's rolling "7 days" exclude-newer window hides releases newer than the cutoff, so the three packages get the same opt-out awkward has. Assisted-by: ClaudeCode:claude-opus-5
black over the files this PR adds, and pretty-format-toml over
pyproject.toml. pre-commit.ci could not push these itself ("push
permission was denied - pr was made from an organization"). The toml
reformat is the pinned hook's own output; it is semantics-preserving and
main is equally unformatted against that hook today.
Assisted-by: ClaudeCode:claude-opus-5
The docstrings and comments carried freeze tags, work-item numbers and milestone labels from the development pipeline that produced them, which an uproot reader cannot resolve; each now states the behaviour it explains in plain words. tests/test_graphed_m10.py is renamed for what it asserts. Assisted-by: ClaudeCode:claude-opus-5
… runs toml-sort 0.25.0 (2026-09-12) rewrites inline tables with inner spaces; pre-commit.ci's cached hook env still runs 0.24.3 and rejects that layout. Only the graphed test-group lines and exclude-newer-package entries differ from main now. Assisted-by: ClaudeCode:claude-fable-5-1
lgray
marked this pull request as ready for review
September 14, 2026 17:12
Contributor
Author
|
@ariostas this one is ready to go now. |
lgray
added a commit
to graphed-org/hist-graphed-mvp
that referenced
this pull request
Sep 14, 2026
An installed uproot without the graphed reader (any release before scikit-hep/uproot5#1720) raised AttributeError instead of skipping. Assisted-by: ClaudeCode:claude-fable-5-1
lgray
added a commit
to graphed-org/hist-graphed-mvp
that referenced
this pull request
Sep 14, 2026
An installed uproot without the graphed reader (any release before scikit-hep/uproot5#1720) raised AttributeError instead of skipping. Assisted-by: ClaudeCode:claude-fable-5-1
ariostas
reviewed
Sep 14, 2026
``graphed`` and ``graphed-executors`` are optional dependencies, so they now go through ``uproot.extras`` like every other optional import: the new ``extras.graphed()`` / ``extras.graphed_executors()`` import the submodules Uproot uses and raise ``ModuleNotFoundError`` with an install hint. The call sites keep their lazy, in-function position and switch to attribute access. ``_is_graphed_array`` is gone: ``graphed_write`` resolves the package through extras first and checks ``isinstance(array, graphed.Array)``, so a missing ``graphed`` now reports the install hint instead of a misleading TypeError. Assisted-by: ClaudeCode:claude-opus-4-8
The graphed test files take the PR number, as the test suite's convention asks. Their non-test helpers move into a ``tests/graphed`` package (``analysis``, ``report``, ``vector_backend_ref``) and are imported as ``tests.graphed.<name>``. That also retires the ``sys.path.insert(0, os.path.dirname(__file__))`` hack: ``tests/__init__.py`` already puts the repo root on ``sys.path``, so the helpers — and the import refs a spawned worker resolves — are reachable by their dotted names from any cwd, with ``tests/`` itself never on ``sys.path``. Assisted-by: ClaudeCode:claude-opus-4-8
Contributor
Author
|
@ariostas took care of your comments - but it looks like whatever serves the root files in the tests is 403'ing hard. I'll try kicking the tests again tomorrow. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add hooks to uproot that allow it to be used with the delayed execution engine "graphed", it has stabilized enough to where it makes sense to support the code upstream and ease installation.
The graphed hooks ride uproot's own CI:
graphed,graphed-executorsandgraphed-histogram(all on PyPI at 0.0.2) are entries in thetestdependency group behindpython_version >= "3.11"(graphed requires 3.11+), so thetests/test_1720_graphed_*.pysuite runs on the 3.11+ jobs and is skipped on 3.10. The optional imports go throughuproot.extras.graphed()/uproot.extras.graphed_executors(); the suite's non-test helpers live in thetests/graphedpackage and are imported astests.graphed.<name>. The three packages are opted out of[tool.uv] exclude-newerthe way awkward already is, because the 7-day window would otherwise hide the fresh 0.0.2 releases; those keys can go once the window has rolled past.The pre-commit fixes are committed by hand because pre-commit.ci cannot push to a fork owned by an organization.
pyproject.tomlkeeps the layout pre-commit.ci's cachedtoml-sort0.24 produces; a fresh local hook env pulls toml-sort 0.25.0 (2026-09-12), which reformats inline tables and would fail the same hook on pre-commit.ci. The "GPU test with CUDA 13" failure is the self-hosted runner'scuda-bindingspre-release version parse intests-cuda/, which this PR does not touch and whichmainhits too.