This is the code file for the Community Codex Repository library for netCDF. More...

#include "config.h"
#include "ccr.h"
#include <hdf5.h>
#include <H5DSpublic.h>
#include <stdlib.h>

Include dependency graph for ccr.c:

Macros
#define	MAX_BITGROOM_NSD_FLOAT 7

#define	MAX_BITGROOM_NSD_DOUBLE 15

#define	MAX_GRANULARBG_NSD_FLOAT 7

#define	MAX_GRANULARBG_NSD_DOUBLE 15

Functions
int	nc_def_var_bzip2 (int ncid, int varid, int level)
	Turn on bzip2 compression for a variable. More...

int	nc_inq_var_bzip2 (int ncid, int varid, int bzip2p, int levelp)
	Learn whether bzip2 compression is on for a variable, and, if so, the level setting. More...

int	nc_def_var_bitgroom (int ncid, int varid, int nsd)
	Turn on BitGroom quantization for a variable. More...

int	nc_inq_var_bitgroom (int ncid, int varid, int bitgroomp, int nsdp)
	Learn whether BitGroom quantization is on for a variable, and, if so, the NSD setting. More...

int	nc_def_var_granularbg (int ncid, int varid, int nsd)
	Turn on Granular BitGroom quantization for a variable. More...

int	nc_inq_var_granularbg (int ncid, int varid, int granularbgp, int nsdp)
	Learn whether Granular BitGroom quantization is on for a variable, and, if so, the NSD setting. More...

int	nc_def_var_zstandard (int ncid, int varid, int level)
	Turn on Zstandard compression for a variable. More...

int	nc_inq_var_zstandard (int ncid, int varid, int zstandardp, int levelp)
	Learn whether Zstandard compression is on for a variable, and, if so, the level setting. More...

Detailed Description

This is the code file for the Community Codex Repository library for netCDF.

Author: Ed Hartnett

Date: 12/30/19

Function Documentation

◆ nc_def_var_bitgroom()

int nc_def_var_bitgroom	(	int	ncid,
		int	varid,
		int	nsd
	)

Turn on BitGroom quantization for a variable.

The BitGroom filter quantizes the data by setting unneeded bits alternately to 1/0, so that they may compress well. The BitGroom filter itself is lossy (data are irretrievably altered), and it improves the compression ratio provided by a subsequent lossless compression filter. Call the nc_def_var_bitgroom() function before the function that turns on the lossless compression filter (nc_def_var_deflate(), for example).

A notable feature of BitGroom is that the data it processes remain in IEEE754 format after quantization. Therefore the BitGroom filter does nothing when data are read. However, the Bitgroom filter must still be installed on machines that need to read bitgroomed data.

The BitGroom filter only quantizes variables of type NC_FLOAT or NC_DOUBLE. Attempts to set the BitGroom filter for other variable types through the C/Fortran API return an error (NC_EINVAL). The filter does not quantize values equal to the value of the _FillValue attribute, if any. The main difference between the BitGroom algorithm as implemented in the CCR and in NCO is that the NCO version will not quantize the values of "coordinate-like" variables (e.g., latitude, longitude, time) as defined in the NCO manual, whereas the CCR version will quantize any floating-point variable.

Note: Internally, the filter requires CCR_FLT_PRM_NBR (=5) elements for cd_value. However, the user needs to provide only the first element, NSD, since the other elements can be and are derived from the dcpl (data_class, datum_size), and extra queries of the variable (has_mss_val, mss_val). Hence, the netCDF API exposes to the user and requires setting only the minimal number (1) of filter parameters.

Parameters

ncid	File ID.
varid	Variable ID.
nsd	Number of significant digits to retain. Allowed single- and double-precision NSDs are 1-7 and 1-15, respectively. (Default is 3).

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

◆ nc_def_var_bzip2()

int nc_def_var_bzip2	(	int	ncid,
		int	varid,
		int	level
	)

Turn on bzip2 compression for a variable.

Parameters

ncid	File ID.
varid	Variable ID.
level	From 1 to 9. Set the block size to 100k, 200k ... 900k when compressing. (bzip2 default level is 9).

Returns: 0 for success, error code otherwise.

Author: Ed Hartnett

◆ nc_def_var_granularbg()

int nc_def_var_granularbg	(	int	ncid,
		int	varid,
		int	nsd
	)

Turn on Granular BitGroom quantization for a variable.

The Granular BitGroom (GBG) filter quantizes the data by setting unneeded bits to 0, so that they may compress well. GBG is in all external respects similar to BitGroom. The following paragraph describes the differences between GBG and BitGroom, and the rest this preamble, and the postamble are identical to the BitGroom docs. The filter itself is lossy (data are irretrievably altered), and it improves the compression ratio provided by a subsequent lossless compression filter. Call the nc_def_var_granularbg() function before the function that turns on the lossless compression filter (nc_def_var_deflate(), for example).

Granular BitGroom is a second generation quantization filter that incorporates aspects of the BitGroom, DigitRound, and BitRound algorithms. GBG determines the quantization mask for each value independently using the basse-10 logarithm method discussed in the DigitRound paper. GBG computes exact log10() of every number, in contrast to DigitRound which uses a lookup table for speed. This makes GBG slower than DR. However, GBG-quantized values will use the same mask regardless of the sign of the input value, where DR might produce slightly different masks. Another difference is that GBG uses the BitRound rounding method, rather than reconstructing the scalar quantization at bin center. Similar to BitGroom, GBG ignores the value zero, and the _FillValue. GBG quantizes a few more bits than BitGroom from the raw number yet still maintains the precision guarantee, thus the quantization error of GBG is larger than BitGroom. GBG improver compression rations by ~20% relative to BitGroom for typical climate data with NSD = 3.

A notable feature of GranularBG is that the data it processes remain in IEEE754 format after quantization. Therefore the GranularBG filter does nothing when data are read. However, the Bitgroom filter must still be installed on machines that need to read bitgroomed data.

The GranularBG filter only quantizes variables of type NC_FLOAT or NC_DOUBLE. Attempts to set the GranularBG filter for other variable types through the C/Fortran API return an error (NC_EINVAL). The filter does not quantize values equal to the value of the _FillValue attribute, if any. The main difference between the GranularBG algorithm as implemented in the CCR and in NCO is that the NCO version will not quantize the values of "coordinate-like" variables (e.g., latitude, longitude, time) as defined in the NCO manual, whereas the CCR version will quantize any floating-point variable.

Note: Internally, the filter requires CCR_FLT_PRM_NBR (=5) elements for cd_value. However, the user needs to provide only the first element, NSD, since the other elements can be and are derived from the dcpl (data_class, datum_size), and extra queries of the variable (has_mss_val, mss_val). Hence, the netCDF API exposes to the user and requires setting only the minimal number (1) of filter parameters.

Parameters

ncid	File ID.
varid	Variable ID.
nsd	Number of significant digits to retain. Allowed single- and double-precision NSDs are 1-7 and 1-15, respectively. (Default is 3).

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

◆ nc_def_var_zstandard()

int nc_def_var_zstandard	(	int	ncid,
		int	varid,
		int	level
	)

Turn on Zstandard compression for a variable.

Parameters

ncid	File ID.
varid	Variable ID.
level	From -131072 to 22 (depends on Zstandard version). when compressing. Regular compression levels are from 1 up to 19. Use levels >= 20, labeled `--ultra`, cautiously: they require more memory. Negative compression levels that extend the range of speed vs. ratio preferences. The lower the level, the faster the speed (at the cost of compression).

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

◆ nc_inq_var_bitgroom()

int nc_inq_var_bitgroom	(	int	ncid,
		int	varid,
		int *	bitgroomp,
		int *	nsdp
	)

Learn whether BitGroom quantization is on for a variable, and, if so, the NSD setting.

Parameters

ncid	File ID.
varid	Variable ID.
bitgroomp	Pointer that gets a 0 if BitGroom is not in use for this var, and a 1 if it is. Ignored if NULL.
nsdp	Pointer that gets the NSD setting (from 1 to 15), if BitGroom is in use. Ignored if NULL.

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

◆ nc_inq_var_bzip2()

int nc_inq_var_bzip2	(	int	ncid,
		int	varid,
		int *	bzip2p,
		int *	levelp
	)

Learn whether bzip2 compression is on for a variable, and, if so, the level setting.

Parameters

ncid	File ID.
varid	Variable ID.
bzip2p	Pointer that gets a 0 if bzip2 is not in use for this var, and a 1 if it is. Ignored if NULL.
levelp	Pointer that gets the level setting (from 1 to 9), if bzip2 is in use. Ignored if NULL.

Returns: 0 for success, error code otherwise.

Author: Ed Hartnett

◆ nc_inq_var_granularbg()

int nc_inq_var_granularbg	(	int	ncid,
		int	varid,
		int *	granularbgp,
		int *	nsdp
	)

Learn whether Granular BitGroom quantization is on for a variable, and, if so, the NSD setting.

Parameters

ncid	File ID.
varid	Variable ID.
granularbgp	Pointer that gets a 0 if Granular BitGroom is not in use for this var, and a 1 if it is. Ignored if NULL.
nsdp	Pointer that gets the NSD setting (from 1 to 15), if Granular BitGroom is in use. Ignored if NULL.

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

◆ nc_inq_var_zstandard()

int nc_inq_var_zstandard	(	int	ncid,
		int	varid,
		int *	zstandardp,
		int *	levelp
	)

Learn whether Zstandard compression is on for a variable, and, if so, the level setting.

Parameters

ncid	File ID.
varid	Variable ID.
zstandardp	Pointer that gets a 0 if Zstandard is not in use for this var, and a 1 if it is. Ignored if NULL.
levelp	Pointer that gets the level setting (from -131072 to 22), if Zstandard is in use. Ignored if NULL.

Returns: 0 for success, error code otherwise.

Author: Charlie Zender

Macros

Functions

Detailed Description

Function Documentation

◆ nc_def_var_bitgroom()

◆ nc_def_var_bzip2()

◆ nc_def_var_granularbg()

◆ nc_def_var_zstandard()

◆ nc_inq_var_bitgroom()

◆ nc_inq_var_bzip2()

◆ nc_inq_var_granularbg()

◆ nc_inq_var_zstandard()