integer data types in new netcdf file?
Added by Etienne Tourigny almost 14 years ago
Greetings,
I am importing integer data into a netcdf file using the "input" operator.
As I do not need the float precision, the NC_BYTE (8-bit integer) or NC_SHORT (16-bit signed integer) would be sufficient.
However, cdo does not seem to support these datatypes in netcdf files, even with the use of -b <> and -p <> options.
I am able to convert the variables from float to byte or short using nco's ncap2, and then manipulate the data using cdo.
I think it would be much better that cdo support these datatypes when creating new grids.
Is there any way to do this?
thanks,
Etienne
Replies (6)
RE: integer data types in new netcdf file? - Added by Uwe Schulzweida almost 14 years ago
Hi Etienne,
Thanks for the request. Integers are supported for netCDF but this is not documented, yet. I will add the missing docu in the next release. You can use the option '-b' with the letter I in front of the number of bytes. So valid parameter for this option are I8, I16 and I32. Please do not use the option '-p' anymore. This option is obsolete and will be reused in one of the next releases.
Regards,
Uwe
RE: integer data types in new netcdf file? - Added by Etienne Tourigny almost 14 years ago
Thanks Uwe, tried and works well. Yes please update the docs!
I haven't checked though, how are results of subsequent calculations converted from double to int, as all calculations are done in double precision? Are they rounded or set to floor value?
After some time away, I'm "back in business" and you'll probably hear from me some more, with suggestions and new operators...
Wrote a simple "setvalues,I1,O1,I2,O2,...,In,On" operator, which I used to map surface value codes from one table to another.
Cheers.
Etienne
RE: integer data types in new netcdf file? - Added by Uwe Schulzweida almost 14 years ago
Yes, the CDO internal data type is double precision. The internal data type will be rounded down to the nearest integer if the external data type is integer.
Regards,
Uwe
RE: integer data types in new netcdf file? - Added by Sagar Parajuli about 10 years ago
Hi, could you please tell me how to convert the integer data type to double? I am having problem using divc, because the values in my file are in mixed formats.
Thank you,
RE: integer data types in new netcdf file? - Added by Uwe Schulzweida about 10 years ago
Use the CDO option -b F32 or -b F64 to convert the data to single or double precision floats:
cdo -b F32 divc,c ifile ofile
RE: integer data types in new netcdf file? - Added by Miles Sowden almost 2 years ago
yep realize this is an old post but issues have not been resolved.
So is it speed or size that matters 🙂
Roughly ¼ file size for 3x slower
I think processing speed (uncompressed) should win.
The problem is the wasted precision of the data, reduced to float 32. No reason this should not be integer 8 (or 16 short). However, the scale factor is not automatically applied so many programs will not utilse the correct value Using short (int 16) the filesize is reduced by 50% but the scale factor is still taken into consideration in the analysis. Much better (in my opinion) to create a new variable (e.g. Rain_x100) than to get an incorrect result.
My conclusion, keep to float 32 unscaled and uncompressed data.
Timing tests below
for f in *.nc; do echo $f; time cdo infov -timmax $f > $f.txt; done
AWAP_2021.nc maximum compression, slowest 857MB 0m10.361s
float64.nc checking if there is a penalty for 64/32 bit 6.7GB 0m3.181s
scale.nc converted to 16 bit integer 1.7GB 0m4.300s
tmp.nc uncompressed multiplied by 100 3.4GB 0m3.064s
Un_AWAP_2021.nc raw uncompresed file 3.4GB 0m2.996s