Skip to content

Avoid closed/locked HDF5 handles when lazily loading OMX matrices - #83

Open
jpn-- with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-closednodeerror-in-sharrow
Open

Avoid closed/locked HDF5 handles when lazily loading OMX matrices#83
jpn-- with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-closednodeerror-in-sharrow

Conversation

Copilot AI commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Datasets built by from_omx_3d hold lazy dask arrays that reference the source HDF5 file. Computing them later (e.g. to_zarr) can fail with ClosedNodeError when the array wraps a PyTables node whose handle is already closed, or with BlockingIOError (unable to lock file) when the reader tries to reopen a file the caller has open in write/append mode.

ds = sh.dataset.from_omx_3d(omx_path, indexes="one-based", time_periods=TIME_PERIODS)
ds[["DRV_COM_WLK_BOARDS"]].to_zarr(zarr_path, mode="w")  # ClosedNodeError

with openmatrix.open_file(omx_path, mode="a") as h:
    sh.dataset.from_omx_3d(h, time_periods=TIME_PERIODS).compute()  # BlockingIOError

Changes

  • _is_reopenable(filename) (new, sharrow/dataset.py): probes whether the underlying file can be independently opened read-only with h5py. Returns False for unknown/in-memory filenames and for files locked by another open handle.
  • from_omx_3d: uses the deferred reopen-based reader only for reopenable files; otherwise materializes the matrix eagerly (np.asarray(node[:])) instead of wrapping a live file node, so the dataset no longer depends on a handle staying open past graph construction. Reopenability is evaluated once per source file.
  • from_omx: the 2D fast path now gates on _is_reopenable(filename) rather than filename is not None, falling back to reading through the caller's handle when reopening would fail.
  • Tests: test_from_omx_3d_to_zarr covers the reported scenario; test_from_omx_3d_writable_handle covers compute from a handle opened in append mode (fails on the base commit with the lock error).

Note the tradeoff: for non-reopenable sources, data is now read at dataset-construction time rather than lazily. That is the only way to keep the result valid after the handle closes, and it matches the existing eager behavior of from_omx.

Copilot AI linked an issue Jul 30, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Fix ClosedNodeError on from_omx_3d Avoid closed/locked HDF5 handles when lazily loading OMX matrices Jul 30, 2026
Copilot finished work on behalf of jpn-- July 30, 2026 16:08
Copilot AI requested a review from jpn-- July 30, 2026 16:08
@jpn--
jpn-- marked this pull request as ready for review July 30, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClosedNodeError on from_omx_3d(path, ...)

2 participants