Skip to content

Generate regular GRIB coordinates lazily - #1608

Open
michaeldiener wants to merge 2 commits into
Unidata:maint-5.xfrom
michaeldiener:codex/grib-lazy-coordinates
Open

michaeldiener wants to merge 2 commits into
Unidata:maint-5.xfrom
michaeldiener:codex/grib-lazy-coordinates

Conversation

@michaeldiener

@michaeldiener michaeldiener commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opening a GRIB file currently materializes its regular horizontal coordinate arrays, even when the caller only reads the weather field.
Generate these coordinates through a ProxyReader when requested, using normal variable caching behavior.
This applies to regular projected, latitude/longitude and rotated latitude/longitude axes; Gaussian latitude arrays stay on their existing path.

Section reads generate the requested indexes using the original start + index * increment expression.
This preserves the full-array values and rounding, including strided sections, without constructing the whole coordinate first for an uncached axis.
In a 349-field ICON workload that does not read these coordinates, this avoids 3.83 GiB of temporary array allocation.

Validation:

  • Added tests for lazy file opening, exact full/strided values, positive and negative increments, sections of sections, slices, copied variables and mutation isolation for cached and uncached axes.
  • The file-opening regression fails before the builder change and passes afterwards.
  • All 349 real ICON fields retain identical raw float bits and shapes.
  • The new tests pass on Java 8 and Java 21 on Linux, and Java 21 on macOS.
  • Repository-wide spotlessCheck passes.
  • GitHub Actions passes all build, documentation and style checks, plus the Java 8, 11, 17, 21 and 25 test matrix on both Temurin and Corretto.
  • On macOS, the full enabled GRIB suite has 170 passing tests and 20 failures in the two existing S3 test classes.
    Those HTTP 301/400 failures also reproduce using unchanged upstream classes while opening the remote test objects.

@CLAassistant

CLAassistant commented Sep 24, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@lesserwhirls lesserwhirls left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this contribution! The trade-off between memory usage and CPU cycles seems well worth it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants