huge file size with "-b 32" option in concatenation

Added by Venugopal Reddy about 5 years ago

I end up with the following error while using "cdo cat" command.

cdo cat: 0%cdf_put_vara_double: name=u type=NC_SHORT minval=-32906.000000 maxval=11533.000000

Error (cdf_put_vara_double): NetCDF: Numeric conversion not representable

But I could successfully append the NetCDF files using "-b 32" flags with cdo. I believe all my files are good.

However, the output file size is huge. I wonder about the issue and wish to know if there is any solution for the same.

Replies (6)

RE: huge file size with "-b 32" option in concatenation - Added by Karin Meier-Fleischer about 5 years ago

Hi Venugopal,

can you give us more information about the files which you want to cat. Is it possible to upload test files?

-Karin

RE: huge file size with "-b 32" option in concatenation - Added by Venugopal Reddy about 5 years ago

Hi,
I am using ERA5 data at different pressure levels obtained from CDS Copernicus in NetCDF format. Each file size is more than 1.5Giga bytes. Please go through the attached file on the metadata.

datainfo (6.47 KB) datainfo

RE: huge file size with "-b 32" option in concatenation - Added by Ralf Mueller about 5 years ago

could you upload a singe timestep? or upload the data-request.

the input is compressed, usually CDO works in double precision and this will blow up the input when processed with CDO. I can say more when have the original file

RE: huge file size with "-b 32" option in concatenation - Added by Venugopal Reddy about 5 years ago

Hi,
Please find the attachments.

Download all files

era5.py (884 Bytes) era5.py	script using to download the data
sample.nc (3.49 MB) sample.nc	sample data with one variable

RE: huge file size with "-b 32" option in concatenation - Added by Ralf Mueller about 5 years ago

your input is 16-bit integer, with

cdo -b F32

you convert it to single precision floating point numbers, which doubles the file size. you can reduce precision from F32 to the original one with

cdo -b I16 -copu <input file> <output file>

hth
ralf

RE: huge file size with "-b 32" option in concatenation - Added by Venugopal Reddy about 5 years ago

Hi Ralf,
Thanks for the help and it worked with a small change i.e -copu to -copy.

cdo -b I16 -copy <input file> <output file>

(1-6/6)

Project

General

Profile

CDO