CCR

The Community Codec Repository supports compression (and other) filters for netCDF/HDF5 files which are not natively supported by the netCDF C library.BZIP2

Bzip2 is a free and open-source file compression program that uses the Burrows–Wheeler algorithm. For more information see https://www.sourceware.org/bzip2/ and https://en.wikipedia.org/wiki/Bzip2.

In C:

• nc_def_var_bzip2()
• nc_inq_var_bzip2()

In Fortran:

• nf90_def_var_bzip2()
• nf90_inq_var_bzip2()

BitGroom

The BitGroom filter quantizes the mantissa of floating point values (integers are unaffected) by the appropriate amount to retain not less than the requested Number of Significant Digits (NSD), usually taken as the intrinsic precision of the measured or modeled data. BitGroomed data remain in IEEE-754 format, and are more accurate than other quantization filters such as BitShaving and BitSetting. Consider BitGroom as a pre-filter for subsequent lossless compression which with the simple mantissas yields better compression ratios. Zender, C. S. (2016), Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199-3211, doi:10.5194/gmd-9-3199-2016. http://www.geosci-model-dev.net/9/3199/2016 For more info see http://nco.sf.net/nco.html#bg.

In C:

• nc_def_var_bitgroom()
• nc_inq_var_bitgroom()

In Fortran:

• nf90_def_var_bitgroom()
• nf90_inq_var_bitgroom()

Granular BitGroom

The Granular BitGroom (GBG) filter quantizes the mantissa of floating point values (integers are unaffected) by the appropriate amount to retain not less than the requested Number of Significant Digits (NSD), usually taken as the intrinsic precision of the measured or modeled data. GBG is a second generation quantization filter that incorporates aspects of the BitGroom, DigitRound, and BitRound algorithms. GBG determines the quantization mask for each value independently using the base-10 logarithm method discussed in the DigitRound paper. GBG computes exact log10() of every number, in contrast to DigitRound which uses a lookup table for speed. This makes GBG slower than DR. However, GBG-quantized values will use the same mask regardless of the sign of the input value, where DR might produce slightly different masks. Another difference is that GBG uses the BitRound rounding method, rather than reconstructing the scalar quantization at bin center. Similar to BitGroom, GBG ignores the value zero, and the _FillValue. GBG quantizes a few more bits than BitGroom from the raw number yet still maintains the precision guarantee, thus the quantization error of GBG is larger than BitGroom. GBG improves compression ratios by ~20% relative to BitGroom for typical climate data with NSD = 3. Consider GBG as a pre-filter for subsequent lossless compression which with the simple mantissas yields better compression ratios. For more info see http://nco.sf.net/nco.html#gbg.

In C:

• nc_def_var_granularbg()
• nc_inq_var_granularbg()

In Fortran:

• nf90_def_var_granularbg()
• nf90_inq_var_granularbg()

Zstandard

From the Zstandard documentation: "Zstandard is a fast compression algorithm, providing high compression ratios. It also offers a special mode for small data, called dictionary compression. The reference library offers a very wide range of speed / compression trade-off, and is backed by an extremely fast decoder (see benchmarks below). Zstandard library is provided as open source software using a BSD license. Its format is stable and published as IETF RFC 8478." For more info see https://facebook.github.io/zstd/.

In C:

• nc_def_var_zstandard()
• nc_inq_var_zstandard()

In Fortran:

• nf90_def_var_zstandard()
• nf90_inq_var_zstandard()