Skip to content

Promote reach-end gridpoints to breakpoints, remove boundary - #694

Open
jpalm3r wants to merge 3 commits into
network-loadingfrom
expose-reach-quantities
Open

Promote reach-end gridpoints to breakpoints, remove boundary#694
jpalm3r wants to merge 3 commits into
network-loadingfrom
expose-reach-quantities

Conversation

@jpalm3r

@jpalm3r jpalm3r commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Important

Stacked on #685 (network-loading) — merge that first. This diff will include #685's changes until then; the description below covers only what this branch adds on top of network-loading.

Summary

Reach-end gridpoint data (e.g. EPANET's Flow/Velocity, or MIKE's reach-end WaterLevel, which is a distinct series from the node's own) was previously held in a boundary dict on NetworkNode that nothing ever read. This PR removes boundary and instead promotes a MIKE reach's first/last gridpoint to an ordinary breakpoint, so that data is reachable through find()/recall()/to_dataframe() like any other break point.

  • Res1DReach now includes a reach's own start/end gridpoint in its breakpoints (previously only the intermediate ones), gated by populate_gridpoints like every other breakpoint — reaches=[] now consistently means "no gridpoint data at all," including these.
  • Since a reach-end breakpoint sits at the same location as its node, _generate_graph connects them with a zero-length edge, tagged boundary=True and clamped to exactly 0.0 (the raw value is a difference between two independently-sourced numbers, so floating-point noise could otherwise leave a tiny negative weight and break shortest-path algorithms).
  • NetworkNode.boundary (and BasicNode/Res1DNode's boundary param) are removed — a breaking change for any existing NetworkNode subclass.
  • Reverts an in-progress, broken rewrite of _build_dataframe (25 failed / 29 errors on this branch beforehand) — it turned out not to be needed, since breakpoints already flow into the dataframe through the same mechanism as any other graph node.
  • Docs and the one notebook referencing boundary are updated accordingly.

Deliberately out of scope: EPANET. Its reaches have at most one synthetic gridpoint belonging to neither end, so the existing gate (len(reach.gridpoints) > 2) already excludes them — EPANET keeps today's behavior (no breakpoints, length=None) unchanged.

Closes #599.

Related to #680, which stays open: this only reaches MIKE's reach-end data. EPANET's single-gridpoint case needs its own placement rule and is left for a follow-up.

Test plan

  • uv run pytest tests/ — 814 passed, 6 skipped
  • uv run ruff check / uv run mypy on all touched files
  • Manually verified boundary-edge tagging and the floating-point clamp against the real network.res1d/network_cali.res11 fixtures

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request updates ModelSkill’s network loading and time-series handling to (1) make MIKE reach-end gridpoint data discoverable via standard breakpoint mechanisms (removing the unused NetworkNode.boundary concept), (2) add a quantities filter to Network.from_mike()/from_epanet() for more selective reads, and (3) standardize timezone-aware time series inputs by converting them to UTC timezone-naive datetime64[ns].

Changes:

  • Promote MIKE reach start/end gridpoints to ordinary breakpoints and tag node↔breakpoint connections as zero-length boundary=True edges.
  • Add quantities filtering to network constructors and propagate it through the Res1D adapter read path.
  • Normalize tz-aware time to UTC-naive and extend persistence support to include node-geometry comparers’ raw model data; update docs and tests accordingly.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/test_timeseries.py Adds coverage for tz-aware inputs being converted to UTC-naive.
tests/test_network.py Updates node-count expectations; adds tests for boundary edges, reach-end breakpoint behavior, quantity filtering, and filtered-load correctness.
tests/test_match.py Adds matching tests for tz-aware vs tz-naive combinations and different timezones.
tests/test_comparercollection.py Adds regression test for save/load round-trip of node-geometry comparer raw data.
tests/test_comparer.py Adds regression test for tz-aware from_matched() normalization.
src/modelskill/timeseries/_timeseries.py Replaces ns-only normalization with timezone + ns normalization and applies it during dataset validation.
src/modelskill/timeseries/_point.py Switches to the new _normalize_time() helper.
src/modelskill/network.py Removes NetworkNode.boundary, adds quantities filtering, shared chainage tolerance, and boundary-edge tagging/clamping in graph generation.
src/modelskill/model/adapters/_res1d.py Implements quantities filtering and shared empty-frame usage for topology-only locations (with one inconsistency flagged in review).
src/modelskill/comparison/_comparison.py Uses _normalize_time() and enables save/load of raw model data for node-geometry comparers.
docs/user-guide/network.qmd Updates docs to remove boundary, explain boundary edges, and document quantities.
docs/user-guide/matching.qmd Documents timezone handling and UTC-naive conversion behavior.
Suppressed comments (1)

src/modelskill/model/adapters/_res1d.py:28

  • _simplify_colnames() returns a new empty DataFrame() when node.quantities is empty (e.g. MIKE 11 nodes). This defeats the new “share one empty frame” optimization and causes many unnecessary allocations; it also makes topology-only empties inconsistent with the _EMPTY_DATA sentinel used elsewhere.
    # Some formats keep no timeseries at all on some locations - MIKE 11, for instance,
    # stores everything on reach gridpoints, leaving the nodes empty. Asking mikeio1d
    # for a dataframe there raises, so return an empty one instead.
    if not node.quantities:
        return pd.DataFrame()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@jpalm3r
jpalm3r changed the base branch from main to network-loading August 6, 2026 13:34
@jpalm3r
jpalm3r marked this pull request as ready for review August 10, 2026 13:11
@jpalm3r
jpalm3r requested a review from ecomodeller as a code owner August 10, 2026 13:11
jpalm3r and others added 3 commits August 11, 2026 10:27
…tadata

A reach's first and last gridpoint sit at the same location as its start
and end node, but were read separately and stashed as per-node boundary
metadata. That made them a special case: a second loading rule, a dict on
every node, and an abstract property every NetworkNode subclass had to
implement even when it had nothing to put there.

Make them ordinary breakpoints instead. Each is joined to its node by a
zero-length edge tagged boundary=True, so the distinction survives in the
graph while the loading rules collapse into one — notably, reaches=[] now
skips these reads like any other breakpoint.

Distances within a chainage tolerance of the reach ends are clamped to an
exact 0.0. The two endpoints come from different upstream sources, so
floating-point noise could otherwise leave a tiny nonzero weight where the
true value is analytically zero.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old PNG showed boundary nodes, which no longer exist. Redraw it as an
SVG so it stays legible when scaled and can be edited as text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Document the zero-length boundary edges and the two remaining NetworkNode
properties, and warn that length-weighted networkx algorithms treat a 0.0
edge as a free hop rather than erroring. Drop the boundary property from
the subclassing examples and the notebook.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jpalm3r
jpalm3r force-pushed the expose-reach-quantities branch from 9872826 to 24d4600 Compare August 11, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include boundary in recall

2 participants