CDO

CDO supports Zarr V2 datasets via the NetCDF library.
To ensure that all features are supported, NetCDF version 4.10.0 or newer should be used.

In order to access a Zarr dataset with CDO, the file name normally used is replaced with an URL with a specific format.
URL Format:

protocol://[host:port]/path#mode

protocol: this should be https, s3 or file. The s3 scheme is equivalent to "https" plus setting "mode=s3".

The mode key can have the following values:

• zarr: pure Zarr format
• nczarr: NCZARR format extends the pure Zarr format by adding extra attributes
• consolidated: use consolidated metadata
• file: The file driver stores data in a directory tree.
• s3: The s3 driver stores data using Amazon S3 or some equivalent.
• zip: The zip driver stores data in a local zip file

zip and s3 must be enabled when configuring NetCDF.

You can find full details of the NetCDF NCZarr implementation in the NetCDF NCZarr documentation

Limitations

• Zarr V3 is not supported
• Misleading/general error messages

Compressed Zarr datasets

Zarr datasets are usually compressed. Various compression filters are used for this purpose. For compressed Zarr datasets, the environment variable HDF5_PLUGIN_PATH must point to the directory containing the installed compression filter plugins. If HDF5_PLUGIN_PATH is not set or the required plugin is missing, this may result in error messages that are difficult to understand!

CDO output format

The output format of a CDO command is the same as the input format, unless it is explicitly changed using the -f option. When selecting data from a Zarr dataset, the output format should be set using the -f option. Otherwise, CDO will attempt to write the selected data to a Zarr dataset, which is usually not the intended outcome.

CDO query feature

Using the query feature allows you to reduce the view to the dataset before CDO processes it. This can improve performance and reduce memory requirements. See https://code.mpimet.mpg.de/boards/53/topics/16602
This feature is available with CDO version 2.6.1 and later.

Examples

Here are a couple of examples to access Zarr datasets using CDO.

Create a Zarr dataset using a local directory tree as storage:

cdo -f nczarr copy infile file:///path/dataset.zarr#mode=zarr

Create a Zarr dataset using a zip file as storage:

cdo -f nczarr copy infile file:///path/dataset.zarr.zip#mode=zarr,zip

Create a Zarr dataset using S3 as storage:

cdo -f nczarr copy infile s3://servername/bucketname/dataset.zarr#mode=zarr

Display the content of a Zarr dataset using a file as storage:

cdo xsinfon file:///path/dataset.zarr#mode=zarr

Display the content of a Zarr dataset using S3 as storage:

cdo xsinfon s3://servername/bucketname/dataset.zarr#mode=zarr

The following section describes the procedure for real Zarr datasets at the DKRZ.

Path to the Zarr dataset used in the examples:

SAP0006PATH=/work/mh1570/k203123/prj/dolpung_coupled/icon-mpim/experiments/sap0006/outdata

Display the content of a Zarr dataset using a file as storage:

cdo xsinfon file://${SAP0006PATH}/sap0006.zarr/P1D_mean_z7#mode=zarr

Display the content of a Zarr dataset using S3 as storage:

cdo xsinfon s3://s3.waterpark.dkrz.de/misc/wcrp-hackathon/ICON/d3hp003.zarr/P1D_mean_z7_atm#mode=zarr,consolidated

Select data from a Zarr dataset:

cdo -f nc4 select,name=ssh,startdate=1960-01-01,enddate=1969-12-31 file://${SAP0006PATH}/sap0006.zarr/P1D_mean_z7#mode=zarr outfile.nc

Select the same data using the CDO query feature for better performance:

cdo -f nc4 --query group=P1D_mean_z7,name=ssh,startdate=1960-01-01,enddate=1969-12-31 copy file://${SAP0006PATH}/sap0006.zarr#mode=zarr,consolidated outfile.nc