Skip to content

Reuse decoded arrays for single full-field GRIB reads - #1609

Open
michaeldiener wants to merge 1 commit into
Unidata:maint-5.xfrom
michaeldiener:codex/grib-full-field-reuse
Open

michaeldiener wants to merge 1 commit into
Unidata:maint-5.xfrom
michaeldiener:codex/grib-full-field-reuse

Conversation

@michaeldiener

@michaeldiener michaeldiener commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GribDataReader.DataReceiver allocates and fills a result array before reading a record, then copies the decoded floats into it even for a single complete field.
Defer the result allocation and transfer the decoder's fresh array when the requested shape contains exactly one full field, both ranges are contiguous and start at zero, and the decoded array has exactly the required size.

Subsets, strides, composite/scattered ranges, multiple records and oversized decoder output keep the copy path.
Missing records still produce NaNs, and debug zero filling remains supported.
The original result shape and rank are preserved.

This change can be merged independently.
For multi-byte CCSDS fields, the decoder buffer-size fix in #1607 lets more reads use the fast path; oversized arrays safely fall back to copying until then.
With correctly sized decoded arrays, avoiding the second array saves 3.83 GiB of allocation in the 349-field ICON workload.

Validation:

  • Added seven tests covering storage reuse, rank, missing records, out-of-order records, subsets/strides, composite/scattered ranges, oversized input and debug zero filling.
  • The storage-reuse test fails on unchanged upstream; all seven pass with the change.
  • All 349 real ICON fields retain identical raw float bits and shapes, both with this change alone and with the decoder fix.
  • The new tests pass on Java 8 and Java 21 on Linux, and Java 21 on macOS.
  • Repository-wide spotlessCheck passes.
  • GitHub Actions passes all build, documentation and style checks, plus the Java 8, 11, 17, 21 and 25 test matrix on both Temurin and Corretto.
  • On macOS, the full enabled GRIB suite has 173 passing tests and 20 failures in the two existing S3 test classes.
    Those HTTP 301/400 failures also reproduce using unchanged upstream classes while opening the remote test objects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant