Project

General

Profile

Accessing Zarr3 datasets with CDO

Added by Uwe Schulzweida about 1 month ago

This page describes an experimental version of CDO for reading Zarr V3 datasets.

The NetCDF repository https://github.com/uweschulzweida/netcdf-c/tree/zarr3 was used to read Zarr V3.
This branch is an extension of https://github.com/mannreis/netcdf-c/tree/zarr-http.
Essentially, the branch has been extended to include a simple implementation of the missing Zarr V3 feature for decoding shards.
It is based on NetCDF version 4.9.3, incorporating many Zarr3 enhancements by Dennis Heimbigner and Manuel Reis.
The branch differs by more than 50K LOC from the current NetCDF main branch; a merge is out of the question at present.

On levante, this NetCDF version has been installed at /home/m/m214003/local/netcdf-c-dev-zarr3 and the CDO application is located at /home/m/m214003/local/bin/cdozarr3.

Limitations

  • New NetCDF features from release 4.10.0 are not available
  • S3 driver is not working, yet
  • The creation of Zarr3 datasets has not been implemented
  • So far, this feature has only been tested with the xspies dataset on Levante.

Examples

Accessing a Zarr3 dataset with CDO works in exactly the same way as with Zarr2.
Read also:

Environment variables used in the examples:

export HDF5_PLUGIN_PATH=/home/m/m214003/local/netcdf-c-dev-zarr3/plugins
XSPIESPATH=/work/mh1570/m301120/datasets/xspies
Display the content of the xspies P1D_mean_z7 Zarr3 dataset:

cdozarr3 --query group=P1D_mean_z7 xsinfon file://${XSPIESPATH}/xps005.zarr#mode=zarr

Select the variable ssh from the xspies P1D_mean_z7 dataset and write the result to NetCDF4:

cdozarr3 -f nc4 --query group=P1D_mean_z7,name=ssh copy file://${XSPIESPATH}/xps005.zarr#mode=zarr outfile.nc