Project

General

Profile

Are infinties considered missing by CDO?

Added by Matt Thompson over 2 years ago

We recently found that some files we have been producing have stray infinities in them. Obviously a bad thing and we're looking to correct that, but I first thought "No we aren't" because:

$ cdo --version
Climate Data Operators version 2.0.5 (https://mpimet.mpg.de/cdo)
...
$ cdo infon -selname,T2MDEW badfile.nc4
cdo(1) selname: Process started
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 1984-08-18 13:30:00       0   259920      14 :      186.19      275.74      301.42 : T2MDEW
cdo(1) selname: Processed 259920 values from 147 variables over 1 timestep.
cdo    infon: Processed 259920 values from 1 variable over 1 timestep [0.06s 27MB].

So sure we have some missing values, but the Maximum is 301, not Infinity.

And yet:

$ ncdump -v T2MDEW badfile.nc4 | rg infinity
    191.1258, 188.2285, Infinityf, Infinityf, Infinityf, Infinityf,
    Infinityf, Infinityf, 198.5257, 202.6711, 204.7231, 206.4343, 208.0729,
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf,
    Infinityf, Infinityf, 199.5454, 203.4404, 206.7863, 209.5758, 210.0919,

There are 14 Infinityf there and CDO says there are 14 missing values. The odd thing is, we've set the missing value to 1e15:

                T2MDEW:_FillValue = 1.e+15f ;
                T2MDEW:missing_value = 1.e+15f ;
                T2MDEW:fmissing_value = 1.e+15f ;
                T2MDEW:vmin = -1.e+15f ;
                T2MDEW:vmax = 1.e+15f ;
                T2MDEW:valid_range = -1.e+15f, 1.e+15f ;

Is this the expected behavior of CDO in re Infinity? I suppose from the CF Conventions it says:

Since version 2.4 the NUG defines missing data as all values outside of the valid_range, and specifies how the valid_range should be defined from the _FillValue (which has library specified default values) if it hasn’t been explicitly specified.

And I guess Infinity is definitely outside that range!

Thanks,
Matt


Replies (1)

RE: Are infinties considered missing by CDO? - Added by Karin Meier-Fleischer over 2 years ago

Hi Matthew,

I would say ncdump is able to identyfy infinityf in the netCDF file and CDO automatically sets Infinityf values to the missing value. One problem can be that the maximum value of valid_range and the vmax attributes are set to the missing value too.

    (1-1/1)