Skip to content

Reduce transient allocations when decoding CCSDS GRIB2 fields - #1607

Merged
lesserwhirls merged 1 commit into
Unidata:maint-5.xfrom
michaeldiener:codex/grib2-ccsds-buffer-size
Sep 24, 2026
Merged

lesserwhirls merged 1 commit into
Unidata:maint-5.xfrom
michaeldiener:codex/grib2-ccsds-buffer-size

Conversation

@michaeldiener

@michaeldiener michaeldiener commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

CCSDS decoding allocates one float per decoded byte, so 16-bit and 32-bit fields return arrays two or four times the grid size, with trailing zeros.
It also creates full-sized Java arrays for both the compressed input and the decompressed native output.

This change keeps the returned array at totalNPoints, transfers compressed input through at most 64 KiB of Java scratch storage, and converts decoded samples directly from a big-endian view of native memory.
The view is consumed before the native allocation is closed; callers still receive an independent Java float array.
The input transfer uses the existing RandomAccessFile.readFully API, preserving partial-read and EOF handling for file-backed, in-memory and remote readers.

All three changes are kept together because they affect the same CCSDS decoding method.
The coordinate and receiver changes in #1608 and #1609 are independent.

Validation:

  • Added 19 direct-reader cases covering 8-, 16-, and 32-bit samples, unsigned values, bitmap gaps, scan order, result independence, multi-chunk input, partial reads, truncated input, constant fields and exact payload boundaries.
  • The multi-byte length regression and bounded-buffer regression fail before their respective fixes.
  • All 19 cases pass on Linux with Java 8 and 21, and on macOS with Java 21.
  • All 349 real ICON fields retain identical raw float bits and shapes; complete application outputs also match bit for bit.
  • Repository-wide spotlessCheck passes.
  • The full enabled GRIB suite on macOS has 185 passing tests and 20 failures in the two existing S3 test classes.
    Those HTTP 301/400 failures were also reproduced using unchanged upstream classes while opening the remote test objects.
  • The 67,574-byte compressed fixture uses deterministic generated samples encoded with libaec; its parameters and expected samples are documented in the test.

Allocation and timing were measured on Linux/Java 21 using captured ICON data.
Against a baseline already containing the array-size, coordinate and receiver fixes, the two buffer changes remove approximately 2.40 GiB of cumulative Java allocation per 349-field import.
In one full application comparison, allocation fell from 6,892 to 4,435 MiB (35.7%), while elapsed time increased from 94.84 to 96.59 seconds (1.8%).
The isolated decode-and-hash benchmark was slower, from a median of 9.92 to 11.46 seconds across three measured runs per variant.
Larger input transfer buffers did not improve those timings, so the scratch limit remains 64 KiB.
These measurements describe allocation churn and a CPU tradeoff, not retained heap or production latency.

@michaeldiener
michaeldiener force-pushed the codex/grib2-ccsds-buffer-size branch from 266adf3 to c5a522a Compare September 24, 2026 08:26
@michaeldiener michaeldiener changed the title Fix CCSDS decoded array length for multi-byte samples Reduce transient allocations when decoding CCSDS GRIB2 fields Sep 24, 2026
@michaeldiener
michaeldiener marked this pull request as ready for review September 24, 2026 08:42
@lesserwhirls lesserwhirls added this to the 5.11.0 milestone Sep 24, 2026

@lesserwhirls lesserwhirls left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are great changes, thank you so much - and thank you for keeping the PR small and targeted (as opposed to combining all three into one larger PR).

@lesserwhirls
lesserwhirls merged commit 0b4a054 into Unidata:maint-5.x Sep 24, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants