Project

General

Profile

strange xvals and xsize for gaussian reduced grids

Added by Brian Højen-Sørensen about 3 years ago

We have recently updated one of our system for data processing and believe we have identified a minor bug.

We use cdo griddes as part of our automated checking of GRIB files by comparing to a known grid description. One of the datasets we receive from an external partner (ECMWF) suddenly couldn’t validate on the new system, but still works on our old system with CDO 1.9.3. The main reason seems to be some error in the GRIB file with a reduced Gaussian grid, where part of the grid metadata is sometimes wrong, but the reason it fails on newer versions of CDO (at least 1.9.8 and above), is due to added information to the griddes output in the newer versions, which doesn’t really make any sense.

In the old CDO versions there is no xvals and xsize in the ouput only yvals and ysize (see full griddes in griddes_1.9.3.txt).

cdo-1.9.3/bin/cdo griddes ex1.grb
gridtype  = gaussian_reduced
gridsize  = 6599680
ysize     = 2560
xname     = lon
xlongname = "longitude" 
xunits    = "degrees_east" 
yname     = lat
ylongname = "latitude" 
yunits    = "degrees_north" 
np        = 1280
yvals     = 89.94619 89.87648 ... ALL LATITUDES... -89.87648 -89.94619 
rowlon    = 20 24 ... ALL ROWS ... 24 20

For the new versions xvals and xsize is present but somewhat less useful and in our case sometimes wrong (see full griddes in griddes_1.9.10.txt)

cdo-1.9.10/bin/cdo griddes ex1.grb
gridtype  = gaussian_reduced
gridsize  = 6599680
xsize     = 2
ysize     = 2560
xname     = lon
xlongname = "longitude" 
xunits    = "degrees_east" 
yname     = lat
ylongname = "latitude" 
yunits    = "degrees_north" 
numLPE    = 1280
xvals     = 180.070093 180 
yvals     = 89.94619 89.87648 ... ALL LATITUDES... -89.87648 -89.94619 
reducedPoints = 20 24 ... ALL ROWS ... 24 20

The second example illustrates our problem, namely that xvals is just 180.070093 180 (first and last longitude), which doesn’t make a lot of sense. The grid is -179.93 to 180 degrees but for some reason it’s sometimes not provided correctly. For the second example file ex2.grb the result is as it should be.

cdo-1.9.10/bin/cdo griddes ex2.grb
gridtype  = gaussian_reduced
gridsize  = 6599680
xsize     = 2
ysize     = 2560
xname     = lon
xlongname = "longitude" 
xunits    = "degrees_east" 
yname     = lat
ylongname = "latitude" 
yunits    = "degrees_north" 
numLPE    = 1280
xvals     = -179.93 180 
yvals     = 89.94619 89.87648 ... ALL LATITUDES... -89.87648 -89.94619 
reducedPoints = 20 24 ... ALL ROWS ... 24 20

I would like to stress that we believe this part is an issue at ECMWF, but it could also be some issue with the underlying API, which doesn’t return the values consistently (somewhat less likely). For an actual forecast (~100 GRIB files) there seems to be a random number of files with one or the other result. I will also stress that the actual grid is the same in both files it is only the grid metadata that is different.

As a note, if we use CDO to create a new file (copy or selecting a single parameter using selvar) CDO changes the values to 0 and 359.929688 or -179.93 180, depending on which input file it is. So some sort of internal wrapping of coordinates is taking place (in different parts of the CDO code presumably). Also it could point to some difference in the file metadata that I haven't been able to locate, as it is the erronious file that gets 0 - 360, indicating that negative values should be wrapped by adding 360 which would fit the xvals output.

Our conclusion is that there is most likely something strange in the API or at ECMWF when they generate the files since the result is inconsistent. Just to do some extra checking I used Python and pygrib to view the data as well. It gave the same results as CDO.

But we do also think that the inclusion of xvals and xsize in the griddes output is flawed, as an xsize = 2 and the xvals just being the first and last longitude doesn’t really makes any sense (even when correct). We believe that the right solution is either to provide the full set of xvals (which is the same as the total gridsize on the reduced gaussian grid) or just not provide it at all and stick to yvals and rowlon/reducedPoints as in older versions. We would likely prefer the latter as the output would become quite large if all xvals were provided.

I unfortunately cannot provide the source files as they are too big (113 MB each) and if i try to reduce them, CDO will modify the values in question. I have, however, attached the examples with just a single parameter for testing on.


Replies (4)

RE: strange xvals and xsize for gaussian reduced grids - Added by Uwe Schulzweida about 3 years ago

Hi Brian,

I get the correct result with CDO for both files. And the coordinates in the files are also the same:

grib_dump ex_10u_ok.grb2 | grep longitudeOfFirstGridPointInDegrees
  longitudeOfFirstGridPointInDegrees = -179.93;
grib_dump ex_10u_err.grb2 | grep longitudeOfFirstGridPointInDegrees
  longitudeOfFirstGridPointInDegrees = -179.93;
Please check if the correct files have been attached.
I have introduced the two xvals to determine whether it is a global or regional grid.
Unfortunately I have not found any other way to implement this.

Cheers,
Uwe

RE: strange xvals and xsize for gaussian reduced grids - Added by Brian Højen-Sørensen about 3 years ago

Hi Uwe,

First of all sorry for the very late reply.

I attached the wrong files, but I have now located the error in the ECMWF files using grib_dump.

It turns out that all parameters are GRIB1 data with

longitudeOfFirstGridPointInDegrees = -179.93
longitudeOfLastGridPointInDegrees = 180

except one (precipitation type), which is a GRIB2 parameter with

longitudeOfFirstGridPointInDegrees = 180.07;
longitudeOfLastGridPointInDegrees = 180;

This is an obvious error and should be handled at ECMWF. The files that are reported as xvals = 180.070093 180 using cdo griddes are the ones where precipitation type is the first GRIB message, which seems to be the one CDO is using for reporting the grid.
I have attached a file with just the ptype parameter which has the issue as well as a grib_dump of the full files (one of them have ptype as the first message).

To conclude, I'm writing a bug report to ECMWF as the values of longitudeOfFirstGridPointInDegrees is clearly wrong. If they sort it out all is then fine.
We have also found a work around for our present case, which is to use griddes in combination with selvar, to make sure we are not checking the message from ptype.

I don't know if there could be any changes made to CDO which would somehow handle this, and in the end I believe that the issue is at ECMWF not CDO.

Cheers,
Brian

dump_ok.txt (56.7 KB) dump_ok.txt grib_dump with ptype as message 12
dump_err.txt (55.8 KB) dump_err.txt
ptype.grb2 (12.6 MB) ptype.grb2 GRIB2 ptype with wrong longitudeOfFirstGridPointInDegrees

RE: strange xvals and xsize for gaussian reduced grids - Added by Uwe Schulzweida about 3 years ago

Hi Brian,

Thanks for the files! It looks like only positive longitudes are stored in ecCodes for GRIB2.
I have updated the CDO code for Gaussian reduced grids. Now the longitudes are corrected if necessary.
This feature will be available in the next CDO release 2.0.0.

Cheers,
Uwe

RE: strange xvals and xsize for gaussian reduced grids - Added by Brian Højen-Sørensen almost 3 years ago

Hi Uwe,

That is correct, I actually wasn't aware before, but during the bug/solution-hunt I found out that GRIB2 only accepts 0-360, where GRIB1 was more flexible.
But It's really good to hear that you have implemented a workaround in your end. We look forward to trying out 2.0.0, as our current solution isn't the prettiest.

Cheers,
Brian

    (1-4/4)