Compress datasets exported to netcdf

[jenshnielsen]

Datasets exported to netcdf were written without any compression. Enable the deflate (gzip) compression built into netcdf4/HDF5 when exporting.

The encoding is tuned rather than using the defaults, since the defaults leave a large part of the achievable compression unused:

• HDF5 auto chunking picks chunks of about 20 kB. Each chunk is compressed independently so this compresses noticeably worse than using chunks of a few MB. Chunks are now sized to about 4 MB.
• The shuffle filter is only enabled for non complex data. Complex numbers are stored as an HDF5 compound type of two floats and shuffling the bytes of such a compound type roughly halves the compression ratio, while shuffling is what gives most of the gain for real valued data.
• Dask backed datasets are rechunked to match the HDF5 chunks before writing. Otherwise every dask block triggers a read, decompress, update and recompress cycle of the same HDF5 chunk in the delayed writer.

For typical QCoDeS data this reduces the file size by a factor of 1.9 for gridded data, 3.0 for complex data and 3.3 for non gridded data, without slowing down the export. The compression is lossless and the files remain regular netcdf files readable by any netcdf client.

The level is configurable via the new config option dataset.export_netcdf_compression_level which defaults to 4 and can be set to 0 to write uncompressed files. Higher levels are slower with almost no additional size reduction.

The temporary per row files written by the chunked export of large datasets are written uncompressed since they are immediately recombined into the final file.

[codecov]

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.13%. Comparing base (a997230) to head (bb12f5b).
⚠️ Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
src/qcodes/dataset/exporters/export_to_xarray.py	95.45%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8367      +/-   ##
==========================================
+ Coverage   71.09%   71.13%   +0.04%     
==========================================
  Files         305      305              
  Lines       31913    31977      +64     
==========================================
+ Hits        22687    22746      +59     
- Misses       9226     9231       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.