cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable"
Added by H Pal over 7 years ago
Hi,
I have several compressed netcdfs and when I try to run
cdo ensmean ifiles*.nc ofile.nc
I get the following:
cdo ensmean: Started child process "setctomiss,-inf mrro_ens0432.nc (pipe1.1)".
cdf_put_vara_double : ncid = 196608 varid = 3 val0 = 0.000000
cdf_put_vara_double : varname = rivflow
Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140431247976832:
#000: ../../../src/H5D.c line 415 in H5Dclose(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
Error (cdf_close) : NetCDF: HDF error
Any suggestion?
Thanks
Replies (11)
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by Karin Meier-Fleischer over 7 years ago
Hi,
what returns cdo sinfon on one of your netCDF files?
cdo sinfon ifile.nc
Is ist possible to upload two or more files?
-Karin
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
Hi Karin,
Thanks - I cannot upload any sample file as they are very large - and even if I try to get a time subset I get the same error.
When I try cdo sinfon ifile.nc I get the following:
File format : netCDF4 ZIP
-1 : Institut Source Ttype Levels Num Points Num Dtype : Parameter name
1 : unknown unknown instant 1 1 259200 1 F32z : rivflow
Grid coordinates :
1 : lonlat : points=259200 (720x360)
lon : -179.75 to 179.75 by 0.5 degrees_east circular
lat : 89.75 to -89.75 by -0.5 degrees_north
Vertical coordinates :
1 : surface : levels=1
Time coordinate : 3650 steps
RefTime = 0001-01-01 00:00:00 Units = hours Calendar = standard
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
1981-01-02 00:00:00 1981-01-03 00:00:00 1981-01-04 00:00:00 1981-01-05 00:00:00
1981-01-06 00:00:00 1981-01-07 00:00:00 1981-01-08 00:00:00 1981-01-09 00:00:00
1981-01-10 00:00:00 1981-01-11 00:00:00 1981-01-12 00:00:00 1981-01-13 00:00:00
1981-01-14 00:00:00 1981-01-15 00:00:00 1981-01-16 00:00:00 1981-01-17 00:00:00
1981-01-18 00:00:00 1981-01-19 00:00:00 1981-01-20 00:00:00 1981-01-21 00:00:00
1981-01-22 00:00:00 1981-01-23 00:00:00 1981-01-24 00:00:00 1981-01-25 00:00:00
1981-01-26 00:00:00 1981-01-27 00:00:00 1981-01-28 00:00:00 1981-01-29 00:00:00
1981-01-30 00:00:00 1981-01-31 00:00:00 1981-02-01 00:00:00 1981-02-02 00:00:00
1981-02-03 00:00:00 1981-02-04 00:00:00 1981-02-05 00:00:00 1981-02-06 00:00:00
1981-02-07 00:00:00 1981-02-08 00:00:00 1981-02-09 00:00:00 1981-02-10 00:00:00
1981-02-11 00:00:00 1981-02-12 00:00:00 1981-02-13 00:00:00 1981-02-14 00:00:00
1981-02-15 00:00:00 1981-02-16 00:00:00 1981-02-17 00:00:00 1981-02-18 00:00:00
1981-02-19 00:00:00 1981-02-20 00:00:00 1981-02-21 00:00:00 1981-02-22 00:00:00
1981-02-23 00:00:00 1981-02-24 00:00:00 1981-02-25 00:00:00 1981-02-26 00:00:00
1981-02-27 00:00:00 1981-02-28 00:00:00 1981-03-01 00:00:00 1981-03-02 00:00:00
................................................................................
................................................................................
...........................
1990-11-03 00:00:00 1990-11-04 00:00:00 1990-11-05 00:00:00 1990-11-06 00:00:00
1990-11-07 00:00:00 1990-11-08 00:00:00 1990-11-09 00:00:00 1990-11-10 00:00:00
1990-11-11 00:00:00 1990-11-12 00:00:00 1990-11-13 00:00:00 1990-11-14 00:00:00
1990-11-15 00:00:00 1990-11-16 00:00:00 1990-11-17 00:00:00 1990-11-18 00:00:00
1990-11-19 00:00:00 1990-11-20 00:00:00 1990-11-21 00:00:00 1990-11-22 00:00:00
1990-11-23 00:00:00 1990-11-24 00:00:00 1990-11-25 00:00:00 1990-11-26 00:00:00
1990-11-27 00:00:00 1990-11-28 00:00:00 1990-11-29 00:00:00 1990-11-30 00:00:00
1990-12-01 00:00:00 1990-12-02 00:00:00 1990-12-03 00:00:00 1990-12-04 00:00:00
1990-12-05 00:00:00 1990-12-06 00:00:00 1990-12-07 00:00:00 1990-12-08 00:00:00
1990-12-09 00:00:00 1990-12-10 00:00:00 1990-12-11 00:00:00 1990-12-12 00:00:00
1990-12-13 00:00:00 1990-12-14 00:00:00 1990-12-15 00:00:00 1990-12-16 00:00:00
1990-12-17 00:00:00 1990-12-18 00:00:00 1990-12-19 00:00:00 1990-12-20 00:00:00
1990-12-21 00:00:00 1990-12-22 00:00:00 1990-12-23 00:00:00 1990-12-24 00:00:00
1990-12-25 00:00:00 1990-12-26 00:00:00 1990-12-27 00:00:00 1990-12-28 00:00:00
1990-12-29 00:00:00 1990-12-30 00:00:00
Thanks
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by Karin Meier-Fleischer over 7 years ago
It's hard to say what the problem might be. Is the missing value always the same for all files? Do they contain NaN or -inf? To see how many missing values your file contains you can use cdo info.
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
Thanks -
I have run "cdo info" for few ifiles and all of them show the following (first few rows):
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1981-01-02 00:00:00 0 259200 0 : 0.0000 0.0000 0.0000 : -1
2 : 1981-01-03 00:00:00 0 259200 0 : 0.0000 inf inf : -1
3 : 1981-01-04 00:00:00 0 259200 0 : 0.0000 inf inf : -1
4 : 1981-01-05 00:00:00 0 259200 0 : 0.0000 inf inf : -1
5 : 1981-01-06 00:00:00 0 259200 0 : 0.0000 inf inf : -1
6 : 1981-01-07 00:00:00 0 259200 0 : 0.0000 inf inf : -1
7 : 1981-01-08 00:00:00 0 259200 0 : 0.0000 inf inf : -1
I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.
Any idea how to fix this?
Cheers,
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
H Pal wrote:
Thanks -
I have run "cdo info" for few ifiles and all of them show the following (first few rows):
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1981-01-02 00:00:00 0 259200 0 : 0.0000 0.0000 0.0000 : -1
2 : 1981-01-03 00:00:00 0 259200 0 : 0.0000 inf inf : -1
3 : 1981-01-04 00:00:00 0 259200 0 : 0.0000 inf inf : -1
4 : 1981-01-05 00:00:00 0 259200 0 : 0.0000 inf inf : -1
5 : 1981-01-06 00:00:00 0 259200 0 : 0.0000 inf inf : -1
6 : 1981-01-07 00:00:00 0 259200 0 : 0.0000 inf inf : -1
7 : 1981-01-08 00:00:00 0 259200 0 : 0.0000 inf inf : -1I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.
Any idea how to fix this?
ps. when I plot the netcdf on Panoply everything seems to be ok
Cheers,
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
H Pal wrote:
Thanks -
I have run "cdo info" for few ifiles and all of them show the following (first few rows):
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1981-01-02 00:00:00 0 259200 0 : 0.0000 0.0000 0.0000 : -1
2 : 1981-01-03 00:00:00 0 259200 0 : 0.0000 inf inf : -1
3 : 1981-01-04 00:00:00 0 259200 0 : 0.0000 inf inf : -1
4 : 1981-01-05 00:00:00 0 259200 0 : 0.0000 inf inf : -1
5 : 1981-01-06 00:00:00 0 259200 0 : 0.0000 inf inf : -1
6 : 1981-01-07 00:00:00 0 259200 0 : 0.0000 inf inf : -1
7 : 1981-01-08 00:00:00 0 259200 0 : 0.0000 inf inf : -1
I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.
Any idea how to fix this?
ps. when I plot the netcdf on Panoply everything seems to be ok
Cheers,
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by Karin Meier-Fleischer over 7 years ago
The problem seems to be that the files contain values with inf. Try to set the inf values to missing values
cdo ensmean -setctomiss,inf ifiles*.nc ofile.nc
-Karin
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
Karin Meier-Fleischer wrote:
The problem seems to be that the files contain values with inf. Try to set the inf values to missing values
[...]
-Karin
Thanks for helping, Karin.
However, I have just got the following error when starting the child process setctomiss,inf:
cdf_put_vara_double : ncid = 196608 varid = 3 val0 = 0.000000
cdf_put_vara_double : varname = rivflow
Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140009995946368:
#000: ../../../src/H5D.c line 415 in H5Dclose(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
Error (cdf_close) : NetCDF: HDF error
Thanks
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by Karin Meier-Fleischer over 7 years ago
I am in the dark without the data. Try the "-b F64" option (higher precision):
cdo -b F64 -ensmean -setctomiss,inf ifiles*.nc ofile.nc
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by Uwe Schulzweida over 7 years ago
cdo -b F64 -ensmean -setctomiss,inf ifiles*.nc ofile.ncThis CDO command doesn't work as expected, because the operator setctomiss can process only one inputfile. That means only the first input file of 'infiles*.nc' will be changed. You can try to change the order of the operators but in this case the input files of ensmean muss be one argument enclosed in single quotes:
cdo -b F64 -setctomiss,inf -ensmean 'ifiles*.nc' ofile.ncBut first we should clarify where all the inf's came from and this is not possible without some example data. Here is an example on how to select only the first 10 timestep of one NetCDF file with the NCO tool ncks:
ncks -d time,0,9 infile outfile
RE: cdo ensmean returns "Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable" - Added by H Pal over 7 years ago
Uwe Schulzweida wrote:
[...]This CDO command doesn't work as expected, because the operator setctomiss can process only one inputfile. That means only the first input file of 'infiles*.nc' will be changed. You can try to change the order of the operators but in this case the input files of ensmean muss be one argument enclosed in single quotes:
[...]But first we should clarify where all the inf's came from and this is not possible without some example data. Here is an example on how to select only the first 10 timestep of one NetCDF file with the NCO tool ncks:
[...]
Great!!
many thanks for all your help! It now works
the line
cdo -b F64 -setctomiss,inf -ensmean 'ifiles*.nc' ofile.nc
was successful. However, it was creating a very large ofile.nc so I just changed F64 to F32 which reduced both computational times and ofile.nc size.
ps. following 'ncks -d time,0,9 infile outfile' I did finally create a subset of my original data. So please find it here attached in case anyone is interested to have a look
Cheers
sample2.nc (1.25 MB) sample2.nc | |||
sample1.nc (1.25 MB) sample1.nc |