CDO

Hi,

what returns cdo sinfon on one of your netCDF files?

cdo sinfon ifile.nc

Is ist possible to upload two or more files?

-Karin

Hi Karin,

Thanks - I cannot upload any sample file as they are very large - and even if I try to get a time subset I get the same error.

When I try cdo sinfon ifile.nc I get the following:

File format : netCDF4 ZIP
    -1 : Institut Source   Ttype    Levels Num    Points Num Dtype : Parameter name
     1 : unknown  unknown  instant       1   1    259200   1  F32z : rivflow       
   Grid coordinates :
     1 : lonlat                   : points=259200 (720x360)
                              lon : -179.75 to 179.75 by 0.5 degrees_east  circular
                              lat : 89.75 to -89.75 by -0.5 degrees_north
   Vertical coordinates :
     1 : surface                  : levels=1
   Time coordinate :  3650 steps
     RefTime =  0001-01-01 00:00:00  Units = hours  Calendar = standard
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  1981-01-02 00:00:00  1981-01-03 00:00:00  1981-01-04 00:00:00  1981-01-05 00:00:00
  1981-01-06 00:00:00  1981-01-07 00:00:00  1981-01-08 00:00:00  1981-01-09 00:00:00
  1981-01-10 00:00:00  1981-01-11 00:00:00  1981-01-12 00:00:00  1981-01-13 00:00:00
  1981-01-14 00:00:00  1981-01-15 00:00:00  1981-01-16 00:00:00  1981-01-17 00:00:00
  1981-01-18 00:00:00  1981-01-19 00:00:00  1981-01-20 00:00:00  1981-01-21 00:00:00
  1981-01-22 00:00:00  1981-01-23 00:00:00  1981-01-24 00:00:00  1981-01-25 00:00:00
  1981-01-26 00:00:00  1981-01-27 00:00:00  1981-01-28 00:00:00  1981-01-29 00:00:00
  1981-01-30 00:00:00  1981-01-31 00:00:00  1981-02-01 00:00:00  1981-02-02 00:00:00
  1981-02-03 00:00:00  1981-02-04 00:00:00  1981-02-05 00:00:00  1981-02-06 00:00:00
  1981-02-07 00:00:00  1981-02-08 00:00:00  1981-02-09 00:00:00  1981-02-10 00:00:00
  1981-02-11 00:00:00  1981-02-12 00:00:00  1981-02-13 00:00:00  1981-02-14 00:00:00
  1981-02-15 00:00:00  1981-02-16 00:00:00  1981-02-17 00:00:00  1981-02-18 00:00:00
  1981-02-19 00:00:00  1981-02-20 00:00:00  1981-02-21 00:00:00  1981-02-22 00:00:00
  1981-02-23 00:00:00  1981-02-24 00:00:00  1981-02-25 00:00:00  1981-02-26 00:00:00
  1981-02-27 00:00:00  1981-02-28 00:00:00  1981-03-01 00:00:00  1981-03-02 00:00:00
   ................................................................................
   ................................................................................
   ...........................
  1990-11-03 00:00:00  1990-11-04 00:00:00  1990-11-05 00:00:00  1990-11-06 00:00:00
  1990-11-07 00:00:00  1990-11-08 00:00:00  1990-11-09 00:00:00  1990-11-10 00:00:00
  1990-11-11 00:00:00  1990-11-12 00:00:00  1990-11-13 00:00:00  1990-11-14 00:00:00
  1990-11-15 00:00:00  1990-11-16 00:00:00  1990-11-17 00:00:00  1990-11-18 00:00:00
  1990-11-19 00:00:00  1990-11-20 00:00:00  1990-11-21 00:00:00  1990-11-22 00:00:00
  1990-11-23 00:00:00  1990-11-24 00:00:00  1990-11-25 00:00:00  1990-11-26 00:00:00
  1990-11-27 00:00:00  1990-11-28 00:00:00  1990-11-29 00:00:00  1990-11-30 00:00:00
  1990-12-01 00:00:00  1990-12-02 00:00:00  1990-12-03 00:00:00  1990-12-04 00:00:00
  1990-12-05 00:00:00  1990-12-06 00:00:00  1990-12-07 00:00:00  1990-12-08 00:00:00
  1990-12-09 00:00:00  1990-12-10 00:00:00  1990-12-11 00:00:00  1990-12-12 00:00:00
  1990-12-13 00:00:00  1990-12-14 00:00:00  1990-12-15 00:00:00  1990-12-16 00:00:00
  1990-12-17 00:00:00  1990-12-18 00:00:00  1990-12-19 00:00:00  1990-12-20 00:00:00
  1990-12-21 00:00:00  1990-12-22 00:00:00  1990-12-23 00:00:00  1990-12-24 00:00:00
  1990-12-25 00:00:00  1990-12-26 00:00:00  1990-12-27 00:00:00  1990-12-28 00:00:00
  1990-12-29 00:00:00  1990-12-30 00:00:00

Thanks

It's hard to say what the problem might be. Is the missing value always the same for all files? Do they contain NaN or -inf? To see how many missing values your file contains you can use cdo info.

Thanks -

I have run "cdo info" for few ifiles and all of them show the following (first few rows):

-1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID
     1 : 1981-01-02 00:00:00       0   259200       0 :      0.0000      0.0000      0.0000 : -1            
     2 : 1981-01-03 00:00:00       0   259200       0 :      0.0000         inf         inf : -1            
     3 : 1981-01-04 00:00:00       0   259200       0 :      0.0000         inf         inf : -1            
     4 : 1981-01-05 00:00:00       0   259200       0 :      0.0000         inf         inf : -1            
     5 : 1981-01-06 00:00:00       0   259200       0 :      0.0000         inf         inf : -1            
     6 : 1981-01-07 00:00:00       0   259200       0 :      0.0000         inf         inf : -1            
     7 : 1981-01-08 00:00:00       0   259200       0 :      0.0000         inf         inf : -1

I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.

Any idea how to fix this?

Cheers,

H Pal wrote:

Thanks -

I have run "cdo info" for few ifiles and all of them show the following (first few rows):

-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1981-01-02 00:00:00 0 259200 0 : 0.0000 0.0000 0.0000 : -1
2 : 1981-01-03 00:00:00 0 259200 0 : 0.0000 inf inf : -1
3 : 1981-01-04 00:00:00 0 259200 0 : 0.0000 inf inf : -1
4 : 1981-01-05 00:00:00 0 259200 0 : 0.0000 inf inf : -1
5 : 1981-01-06 00:00:00 0 259200 0 : 0.0000 inf inf : -1
6 : 1981-01-07 00:00:00 0 259200 0 : 0.0000 inf inf : -1
7 : 1981-01-08 00:00:00 0 259200 0 : 0.0000 inf inf : -1

I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.

Any idea how to fix this?

ps. when I plot the netcdf on Panoply everything seems to be ok

Cheers,

H Pal wrote:

Thanks -

I have run "cdo info" for few ifiles and all of them show the following (first few rows):

-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 1981-01-02 00:00:00 0 259200 0 : 0.0000 0.0000 0.0000 : -1
2 : 1981-01-03 00:00:00 0 259200 0 : 0.0000 inf inf : -1
3 : 1981-01-04 00:00:00 0 259200 0 : 0.0000 inf inf : -1
4 : 1981-01-05 00:00:00 0 259200 0 : 0.0000 inf inf : -1
5 : 1981-01-06 00:00:00 0 259200 0 : 0.0000 inf inf : -1
6 : 1981-01-07 00:00:00 0 259200 0 : 0.0000 inf inf : -1
7 : 1981-01-08 00:00:00 0 259200 0 : 0.0000 inf inf : -1

I have just tested other netcdfs that have also been compressed and an they show reasonable values for mean and maximum.

Any idea how to fix this?
ps. when I plot the netcdf on Panoply everything seems to be ok

Cheers,

The problem seems to be that the files contain values with inf. Try to set the inf values to missing values

cdo ensmean -setctomiss,inf ifiles*.nc ofile.nc

-Karin

Karin Meier-Fleischer wrote:

The problem seems to be that the files contain values with inf. Try to set the inf values to missing values

[...]

-Karin

Thanks for helping, Karin.

However, I have just got the following error when starting the child process setctomiss,inf:

cdf_put_vara_double : ncid = 196608 varid = 3 val0 = 0.000000
cdf_put_vara_double : varname = rivflow

Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140009995946368:
#000: ../../../src/H5D.c line 415 in H5Dclose(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type

Error (cdf_close) : NetCDF: HDF error

Thanks

I am in the dark without the data. Try the "-b F64" option (higher precision):

cdo -b F64 -ensmean -setctomiss,inf ifiles*.nc ofile.nc

cdo -b F64 -ensmean -setctomiss,inf ifiles*.nc ofile.nc

cdo -b F64 -setctomiss,inf -ensmean 'ifiles*.nc' ofile.nc

ncks -d time,0,9 infile outfile

Uwe Schulzweida wrote:

[...]This CDO command doesn't work as expected, because the operator setctomiss can process only one inputfile. That means only the first input file of 'infiles*.nc' will be changed. You can try to change the order of the operators but in this case the input files of ensmean muss be one argument enclosed in single quotes:
[...]But first we should clarify where all the inf's came from and this is not possible without some example data. Here is an example on how to select only the first 10 timestep of one NetCDF file with the NCO tool ncks:
[...]

Great!!

many thanks for all your help! It now works

the line

cdo -b F64 -setctomiss,inf -ensmean 'ifiles*.nc' ofile.nc

was successful. However, it was creating a very large ofile.nc so I just changed F64 to F32 which reduced both computational times and ofile.nc size.

ps. following 'ncks -d time,0,9 infile outfile' I did finally create a subset of my original data. So please find it here attached in case anyone is interested to have a look

Cheers