Project

General

Profile

Forums » Support »

-selname increases file size from 1 GB to 72 GB

Added by Dana King 24 days ago

Hello,

I am very new to cdo, so if this is a stupid question or there is an obvious easy solution, I am sorry.

I have some sea-ice area fraction and thickness data from CARRA2 for 5 years. CARRA2 data is on a curvilinear polar stereographic grid. Therefore, I wanted to regridd the data to a lonlat grid.

The original datafile is 1.04 GB. When I use -selname to select only sea-ice area fraction the resulting file has a size of 72.83 GB. I don't understand how that happens ? And it doesn't make sense to me, as the filesize should reduce.

I used this: cdo -selname,siconc siconc_2000_carra2.nc sic_2000.nc
In addition, I tried to regrid with this : cdo remapbil,r2880x2880 sic_2000.nc sic_2000_lonlat.nc

Whic ran until there was no space left on my computer and at that moment had a size of 36 GB. Can someone help me understand what or why this is happening.


Replies (5)

RE: -selname increases file size from 1 GB to 72 GB - Added by Uwe Schulzweida 24 days ago

Hello Dana,

This usually means that your data is stored compressed. CDO does not automatically compress the output. You can easily compress any NetCDF4 output using the -z zstd_9 option:

cdo -f nc4 -z zstd_9 remapbil,r2880x2880 -selname,siconc siconc_2000_carra2.nc sic_2000_lonlat.nc
Cheers,
Uwe

RE: -selname increases file size from 1 GB to 72 GB - Added by Dana King 24 days ago

Hello Uwe,

Thanks for the clarification. Unfortunately, I receive the following error message:

cdi  cdf_def_var_filter: filterid=32015  numParams=1

cdi  error (cdf_def_var_filter): nc_def_var_filter failed: NetCDF: Filter error: undefined filter encountered

I then tried to run the above command with just nc and it worked however the output file size was still 72 GB. Can you help me understand what the error message means?

Dana

RE: -selname increases file size from 1 GB to 72 GB - Added by Dana King 24 days ago

I was wondering if this could also be connect to the data, as the grid of the data goes from 0 to 360 and form 42.11 to 89.99 N, but the actual data is only between 0 to 40 E and 79 to 83 N and all the other values are NaN ?

RE: -selname increases file size from 1 GB to 72 GB - Added by Uwe Schulzweida 24 days ago

zstd is an external compression filter. The environment variable HDF5_PLUGIN_PATH must point to the directory where the filters are installed. You can install the filter as follows:

pip install hdf5plugin
export HDF5_PLUGIN_PATH=$(python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)")

Alternatively, you can use zip instead of zstd:
cdo -f nc4 -z zip_6 remapbil,r2880x2880 -selname,siconc siconc_2000_carra2.nc sic_2000_lonlat.nc

RE: -selname increases file size from 1 GB to 72 GB - Added by Dana King 23 days ago

Thank you so much, that worked :)
Have a good day,
Dana

    (1-5/5)