CDO

Hi,

I think the problem is related to the missing "missing_value/Fill_value" attribute in your input NetCDF file; you may get the correct value if you set the missing value, say "setmissval,-9e33" before the timmmean/timavg operation

Cheers,
J

Hi J
I set the missing value with the command " cdo setmissval,-9e3 aux1.nc aux1_with_setmissval.nc " and then done " cdo timavg aux1_with_setmissval.nc aux2_with_setmissval.nc ".
In order to see the difference between this and the previous result (aux2.nc) I have put both results into text files:
ncdump aux2.nc > file1.txt
ncdump aux2_with_setmissval.nc > file2.txt
and
" diff --suppress-common-lines file1.txt file2.txt " shows that the values of the variables are not changed, that is, aux2_with_setmissval.nc is as wrong as aux2.nc.
Please note that aux1.nc is CF compliant (I tested here: http://titania.badc.rl.ac.uk/cgi-bin/cf-checker.pl?cfversion=1.4), so, in principle, cdo timavg would be able to treat this kind of file.
Thanks for your time on this.
Cheers,
E.

Hi,

Unfortunately I couldn't identify the problem. To me it works fine (may be I'm doing something wrong); here is the command sequence that I used to create my J_aux2.nc (attached)

cdo -b F64 -timmean -setmissval,-9e33 aux1.nc J2_aux2.nc

I'm using cdo-1.5.8

Cheers,
J

Your command " cdo -b F64 -timmean -setmissval,-9e33 aux1.nc J2_aux2.nc " works for me too !
but " cdo -timmean -setmissval,-9e33 aux1.nc J2_aux2.nc " does not work.
How did you get the hint on -b F64 ?
Cheers,
E.

The result depends of course also on the used data type. The data in the input file has the data type short integer. Therefor only the value 0 and 1 can be stored. If you need also the factional result than you have to change the data type to floating point. Use the CDO option "-b F32" for 32 bit and "-b F64" for 64 bit floats.

So, if I understand well, " cdo timavg " imposes on the output variable the type of the input variable. In this case, as the input variable was integer it wrote the output as integer. Am I correct?

If so, I see this as a cdo error.

Nothing in netcdf or CF standards, or more generally in mathematics, implies that the output of an average should be of the same type of the input. This is a unexpected cdo assumption that can lead to errors. In this case I was able to detect it but in more complex computations involving lots of steps and operations it could pass unnoticed. Please, ignore these statements if I am not correct in saying that " cdo timavg " imposes on the output variable the type of the input variable.

Any way, thanks a lot again for helping me debugging my programs and get the right results.
Cheers,
E.

e ddc wrote:

So, if I understand well, " cdo timavg " imposes on the output variable the type of the input variable. In this case, as the input variable was integer it wrote the output as integer. Am I correct?

If so, I see this as a cdo error.

Nothing in netcdf or CF standards, or more generally in mathematics, implies that the output of an average should be of the same type of the input. This is a unexpected cdo assumption that can lead to errors. In this case I was able to detect it but in more complex computations involving lots of steps and operations it could pass unnoticed. Please, ignore these statements if I am not correct in saying that " cdo timavg " imposes on the output variable the type of the input variable.

The point is: The only programming languages I know with automatic type conversion in the case of possible precision loss are Matlab, its clones like octave and R. All the others like C, Fortran, Python, Ruby and not even the command-line calculator bc return 1 if you compute 3/2. They leave precision handling to the user. And I do not regard this as an error of C, Fortran and all the others. They just keep the types unless the user wants something else. On many occasions this is not user-firendly, but you can rely on it ;-)

Yes, you are right. I have made that mistake so many times (forgot that type conversion and precision is a user business) that I cannot understand how could I do it again here. Perhaps I am too young and used to the more intuitive Matlab and alike. The best solution for people like me would be a note in the CDO manual explicitly saying what is the policy of CDO concerning type conversion. In the same style it warnings about the difference between averages and means Thanks. E.

Hi,

I have encountered a very similar problem with NOAA's PREC data set (I am attaching it here). The raw file contains precipitation anomalies and for some reason cdo treats all negative data as missing values - one can test this by using the ymonavg transformation and finding that most data in the result is missing data.

I have not been able to find any way to instruct cdo to recognize these negative values as data, rather than missing data. The only fix I have found was to open the data set in Ferret and save the precipitation anomaly variable in a new netcdf file. The new file is correctly read by cdo. Does anyone have a "clean" cdo solution to this problem?

Thank you very much in advance,
Alina

I think the missing values are due to "ymonavg" - try with ymonmean instead of ymonavg (CDO treats mean and average differently .. check the docs sec 1.7.1)

Hi,

Thank you for the quick reply. I use ymonavg only to highlight the fact that negative data is interpreted by cdo to be bad (missing) data. ymonmean results in "climatological monthly averages" that are strictly positive - which is just not right considering the input data (this is actually what made me realise that there was a problem with how cdo treated the negative values in the file).

Just in case this is not clear: If I open the raw data file in Ferret (my tool of predilection when it comes to creating maps) and map any given time step, I see a distribution of positive an negative values (precipitation anomalies), with no missing data.
If I then use cdo ymonmean on the raw data file and open the resulting file in Ferret and map any of the 12 time steps, I get conspicuously positive data.
If I use cdo ymonavg to actually highlight any hidden missing values and then plot that in Ferret I indeed see nearly entirely blank figures, as most data will show up as missing, because of course at some point in my time series, most grid points will have some negative anomalies, which are interpreted as missing by cdo and thus result in a missing value when ymonavg is applied.

I hope this makes it clear.

Thank you again,
Alina

from ncdump -h

       float precip(time, lat, lon) ;
                precip:long_name = "Average Monthly Rate of Precipitation" ;
                precip:valid_range = 0.f, 100.f ;
                precip:units = "mm/day" ;
                precip:actual_range = -898.69f, 15965.14f ;
                precip:missing_value = -9.96921e+36f ;
                precip:Fill_Value = -9.96921e+36f ;
...

The valid_range is 0-100, so values outside this range are treated as missing values, that includes negative values [though I'm not sure if CDO recognize "actual_range" attribute]. If you remove this valid_range attribute, I believe CDO will include negative values as well

Thank you very much, Jaison!

Indeed, by using the workaround you described on another forum thread a while ago (https://code.mpimet.mpg.de/boards/1/topics/1211?r=1215) I was able to fix the metadata issue by removing the valid_range attribute.
It sure is a pity that there is no straightforward way of fixing it directly with cdo, but at least your workaround is scriptable and works.

Many thanks,
Alina

you could use the operator setvrange [instead of removing the valid_range], to change the valid range to your desired range before applying other stats operators

Sadly, cdo setvrange does not work as intended in this case (I just checked it). According to Uwe Schulzweida's answer on the other forum thread I referenced before, "The setvrange operator can't be used because the data is set to missing value during the read process."