Project

General

Profile

Missing grib metadata when using cdo tool?

Added by Bang AA over 3 years ago

I have a grib file (it is quite big) and I tried to use cdo to copy it to another grib file.

cdo -R copy ws10m_mean24h.grib test.grib

However, in this diff: https://www.diffchecker.com/w25sayVz it shows that the output of CDO on the right hand side misses the important metadata (Ensemble) and dataDate and startStep are altered.

Could someone explain to me what happened and how can I preserve the change from the source grib file: ws10m_mean24h.grib?

What I wanted to do is to invert the Lat axis (from South-North order to North-South order) of ws10m_mean24h.grib to test.grib but all the medata must be kept from ws10m_mean24h.grib.

cdo invertlat ws10m_mean24h.grib test.grib

Thanks,


Replies (9)

RE: Missing grib metadata when using cdo tool? - Added by Uwe Schulzweida over 3 years ago

CDO was not designed to preserve all GRIB keys during processing. We try our best to get as many GRIB keys as possible, but there will never be all of them. Maybe you have more success with GRIB specific tools like ecCodes from ECMWF.
Without the GRIB file and your CDO version number we can only speculate why some keys might not be correct.
Your File contains e.g. a local GRIB1 extension for COSMO data. This is not part of the GRIB1 standard and therefor CDO can't process this local extension, sorry.

RE: Missing grib metadata when using cdo tool? - Added by Klaus Wyser almost 3 years ago

I am running into a similar problem with metadata when processing GRIB files, so I just append to this issue. The problem: I have model output on a reduced Gauss grid that I need to remap on a regular lonlat grid. This works fine, but for soil variables the metadata are incomplete.

The command is

cdo remapbil,r512x256 -setgridtype,regular ...

In the original file I have

   Vertical coordinates :
     1 : depth_below_land         : levels=1
                            depth : 3.5 cm
                           bounds : 0-7 cm

but in the output I get:

   Vertical coordinates :
     1 : depth_below_land         : levels=1
                            depth : 7 cm

You may say this is no big deal, but actually it messes up our workflow because we then want use the remapped file as input for our regional model and it complains. Everything else (upper-air, surface fields) works fine, it's just the soil variables that cause problems.

An interesting fact is that the metadata were written correctly with older cdo versions. We still have cdo 1.6.9 on our system and I tried with

cdo -R remapbil,r512x256 ...

(there is no setgridtype command in 1.6.9)

Would be nice to get the soil stuff correct even with the current (and future) versions of cdo.

3 files are attached:
  • x.grb: model output (reduced Gauss grid)
  • x_cdo_1_6_9.grb: processed with cdo 1.6.9 (with grib_api lib)
  • x_cdo_1_9_8.grb: processed with cdo 1.9.8 (with eccodes lib)

RE: Missing grib metadata when using cdo tool? - Added by Klaus Wyser almost 3 years ago

To complement the post from yesterday: I have now made teh same test with cdo 1.9.7 and found that the metadata for soil variables are conserved with that version, too. And 1.9.7 also uses eccodes. Something must have happened between versions 1.9.7 and 1.9.8.

Example with cdo 1.9.7 is attached.

RE: Missing grib metadata when using cdo tool? - Added by Ralf Mueller almost 3 years ago

hi Klaus!

I checked 1.9.7.1 , 1.9.8, 1.9.9 and 1.9.10 with your model output and the command given (cdo remapbil,r512x256 -setgridtype,regular ). None of them removed the bounds from the vertical coordinate. Could you show the output of

cdo -V
for the version, that removes the bounds?

Another thing that came to my mind is the fact, that your GRIB1 input is not handles by nether eccodes nor grib_api by default. CDO has its own C-library called cgribex for that. In more recent versions of CDO it is possible to choose with an --eccodes option for all operators.

cheers
ralf

RE: Missing grib metadata when using cdo tool? - Added by Klaus Wyser almost 3 years ago

Thanks for your patience. Here is the output from cdo -V for the various versions:

     CDI library version : 1.6.9 of Dec 21 2020 13:59:41
 CGRIBEX library version : 1.7.2 of Apr 22 2015 13:44:04
GRIB_API library version : 1.24.0
  netCDF library version : 4.3.2 of Jun 20 2018 15:06:09 $
    HDF5 library version : 1.8.12
 SERVICE library version : 1.3.2 of Dec 21 2020 13:59:26
   EXTRA library version : 1.3.2 of Dec 21 2020 13:59:14
     IEG library version : 1.3.3 of Dec 21 2020 13:59:20
    FILE library version : 1.8.2 of Dec 21 2020 13:59:15
     CDI library version : 1.9.7.1
 cgribex library version : 1.9.3
 ecCodes library version : 2.8.2
  NetCDF library version : 4.4.1.1 of Jun 24 2020 16:43:34 $
    hdf5 library version : library undefined
    exse library version : 1.4.1
    FILE library version : 1.8.3
     CDI library version : 1.9.8
 ecCodes library version : 2.8.2
  NetCDF library version : 4.4.1.1 of Jun 24 2020 16:43:34 $
    hdf5 library version : 1.8.19
    exse library version : 1.4.1
    FILE library version : 1.8.3

It seems that our installaton of 1.9.8 has been built without gribex, could that explain the difference?

RE: Missing grib metadata when using cdo tool? - Added by Klaus Wyser almost 3 years ago

I tested the same commands with cdo 1.9.7 and flag --eccodes, and with that setting the metadata turn bad!

RE: Missing grib metadata when using cdo tool? - Added by Ralf Mueller almost 3 years ago

cgribex is enabled by default, so someone must have switched it off during the installation of 1.9.8. In this case, the difference in using grib_api and eccodes in-fact make a difference when processing grib1. I cannot easily test this right away, but the fact that I cannot reproduce the error when using cgribex seems to be enough prove IMO.

So the questions are

  • can you reinstall 1.9.8 with the defaults?
  • can you update to the latest release of CDO
  • what kind of OS do you use? maybe the OS package manager already have the right binary for you

RE: Missing grib metadata when using cdo tool? - Added by Klaus Wyser almost 3 years ago

Thanks Ralf for the help. The different versions of cdo have been installed as modules on our machine. I have to contact the sysadmins to learn more about why gribex support has been disabled in teh latest version. Hopefully they are willing to make a test.

RE: Missing grib metadata when using cdo tool? - Added by Ralf Mueller almost 3 years ago

ah ok - you can also use conda. the conda-forge channel has cdo-1.9.9.

sometimes it's good to get independent. at least to have workaround while waiting for the real solution. spack is another option worth knowing

cheers
ralf

    (1-9/9)