Project

General

Profile

Problems reading GRIB1 data after using CDO

Added by Jan Harlaß almost 8 years ago

I am using ERA-Interim data (ECMWF) to force an atmospheric model. The data is in GRIB1 format. Everything works fine. But since the files are quite large and I do have a lot of them, I compressed them with CDO as szip (cdo -z szip -copy .grb .szip). After uncompressing them back to GRIB1 (cdo -f grb -copy .szip .grb), the model can not read the data correctly. This even occurs when I just copy the GRIB1 data (cdo -copy .grb .grb). So my question is, what does CDO change in these files? Does it affect accuracy, missing values or the axis? Any help is appreciated!


Replies (9)

RE: Problems reading GRIB1 data after using CDO - Added by Jaison-Thomas Ambadan almost 8 years ago

Hi,

a wild guess: may be it is in the reduced grid; did you try the "-R" option, which convert GRIB1 data from reduced to regular grid, i.e, cdo -R -copy infile.grb outfile.grb

Cheers,
J.

RE: Problems reading GRIB1 data after using CDO - Added by Jaison-Thomas Ambadan almost 8 years ago

Hi again,

it seems you have already used the data in your model, so just ignore my previous post. Is your file accessible from tornado/blizzard/squall?

Cheers,
J.

RE: Problems reading GRIB1 data after using CDO - Added by Jan Harlaß almost 8 years ago

Yes I already used the data. It is a global dataset on a gaussian grid (N 128).
Unfortunately, I do not have access to the mentioned servers. Do you have access to HLRN or Kiel university (NESH)?

RE: Problems reading GRIB1 data after using CDO - Added by Jaison-Thomas Ambadan almost 8 years ago

Hi,

Do you have access to HLRN or Kiel university (NESH)?

I don't have access.

Anyway, I'll try your steps with one of the ERA interim data. Can you please post "cdo -V"?

Cheers,
J.

RE: Problems reading GRIB1 data after using CDO - Added by Jan Harlaß almost 8 years ago

Very kind of you!
CDO version:

Climate Data Operators version 1.5.0 (http://code.zmaw.de/projects/cdo)
Compiler: icc -std=c99 -O3 -Wl,-rpath,/sw/dataformats/hdf5/intel-11/1.8.6/lib -Wl,-rpath,/sw/dataformats/netcdf/4.1.2-intel/lib -Wl,-rpath,/sw/dataformats/szip/2.1/lib -Wl,-rpath,/sw/intel/Compiler/11.1/073/lib/intel64
 version: Intel(R) C Intel(R) 64 Compiler Professional for applications running on Intel(R) 64, Version 11.1    Build 20100806 Package ID: l_cproc_p_11.1.073
    with: NC4 PTHREADS SZ Z
Compiled: by bzfbmsch on hd3-ib1 (x86_64-unknown-linux-gnu) Apr 26 2011 16:20:40
     CDI library version : 1.5.0 of Apr 26 2011 16:20:01
 CGRIBEX library version : 1.5.0 of Feb 15 2011 10:23:39
  netCDF library version : 4.1.2 of Apr 19 2011 15:17:29 $
    HDF5 library version : 1.8.5
 SERVICE library version : 1.3.0 of Apr 26 2011 16:19:03
   EXTRA library version : 1.3.0 of Apr 26 2011 16:18:38
     IEG library version : 1.3.0 of Apr 26 2011 16:18:54
    FILE library version : 1.7.1 of Apr 26 2011 16:18:41

I am downloading the ECMWF data with a script:
#!/usr/bin/python
from ecmwf import ECMWFDataServer
import calendar

server = ECMWFDataServer(
       'http://data-portal.ecmwf.int/data/d/dataserver/',
       '################',
       '######email##########'
    )

for year in range(1979, 1980):
 print 'YEAR ',year
 for month in range(01,13):
  lastday1=calendar.monthrange(year,month)
  lastday=lastday1[1]
  bdate="%s%02d01"%(year,month)
  edate="%s%02d%s"%(year,month,lastday)
  print "######### ERA-interim  #########" 
  print 'get data from ', bdate,' to ',edate,' (YYYYMMDD)'
  print "################################" 
  server.retrieve({
    'dataset'  : "interim_full_daily",
    'date'     : "%s/to/%s"%(bdate,edate),
    'time'     : "00/06/12/18",
    'step'     : "0",
    'levtype'  : "sfc",
    'stream'   : "oper",
    'type'     : "an",
    'class'    : "ei",
    'grid'     : "128",
    'param'    : "165/166/167/168/134/151/235/31/34/33/141/139/170/183/236/39/40/41/42",
    'target'   : "ERA-Int_sfc_%s_%s.grb"%(bdate,edate),
    })

RE: Problems reading GRIB1 data after using CDO - Added by Jaison-Thomas Ambadan almost 8 years ago

Hi,

what does CDO change in these files? Does it affect accuracy,

As far as I understood, it won't affect the data accuracy. CDO only change some GRIB header info, and I think it is due to the difference in CGRIBEX library of CDO and GRIB_API library of the ECMWF. Attached are the ERA interim surface pressure data (era_interim_v1p2p1.grb) and the zipped-unzipped GRIB file (unzip.grb), just for testing. The only difference is the length of the GRIB meta info. The following meta-info are missing from the unzip.grb:

-----------------------------------------------
ECMWF local usage identifier. 1
(Mars labelling or ensemble forecast)
Class. 14
Type. 9
Stream. 1025
Version number or Experiment identifier. 0001
Forecast number. 0
Total number of forecasts. 0
-----------------------------------------------

rest of the data/values are same, and I suppose these differences shouldn't affect the model run? Unfortunately, I don't have any solution for your problem. But may be the CDO gurus (Uwe/Ralf) can provide more help/info.

Cheers,
J.

RE: Problems reading GRIB1 data after using CDO - Added by Jan Harlaß almost 8 years ago

I think this issue is heading in the right direction. Thank you Jaison!
I am using WRF and WPS. In WPS the program ungrib.exe reads the grib file according to a variable table, and for this it is reading the file header information. Since something is missing, this could be the reason for corrupted files after running ungrib.exe.
Now the question remains how to deal with that or to avoid the cut of of header information?

RE: Problems reading GRIB1 data after using CDO - Added by Uwe Schulzweida almost 8 years ago

The CDO copy operator will decode and encode the GRIB message. The CDO grib library doesn't support the hole GRIB standard. Therefor it could be possible that some header information are lost after the copy command. You have to use one of the select operator (e.g. selcode) to prevent the decoding/encoding. I have implemented the undocumented operator selall for this purpose:

cdo -z szip  selall  .grb .szip
Unfortunately there is no similar unzip method in CDO.
We use directly our GRIB tool for szip compression/decompression of grib files. This tool is available from: https://code.zmaw.de/projects/cgribex
You have to compile this tool with szip support. Here is an example:
./configure --with-szlib=PATH_TO_SZIP
make

app/grib -szip .grb .szip
app/grib -sunzip .szip .grb

Best regards,
Uwe

RE: Problems reading GRIB1 data after using CDO - Added by Jan Harlaß almost 8 years ago

Yes, cgribex works fine and correct. WRF can now read gribfiles which have been compressed and decompressed with cgribex. Thank you very much!

    (1-9/9)