Supress grb compression in cdo output

Added by Bjoern Hendrik Fock over 11 years ago

Hi,

I noticed that the output of the cdo operations

cdo  sellonlatbox,-5,75,73,84 -remapbil,t955grid ecmwf_2011-04-01_ciwc.grb test-01.grb
cdo  sellonlatbox,-5,75,73,84 -remapbil,t955grid ecmwf_2011-04-02_ciwc.grb test-02.grb

provide some different sizes of the output files, even though the input files have the same file size. I guess this is caused by different cloud cover and some data packing. I like to embed this cdo command in a remote data request job which transfers test-0*.grb to my destination server. As I like to poll on the transferred data, a predictable file size would be helpful.

Is there a possibility to force the output files to have a predictable file size?

Bjoern

Replies (4)

RE: Supress grb compression in cdo output - Added by Uwe Schulzweida over 11 years ago

Hi Bjoern,

The output file size differ, if the input files have missing values and the number of missing values between the input files differ. But in this case the input file size should also differ. Could you please upload both input files? That will help us to find out where the problem come from.

Cheers,
Uwe

RE: Supress grb compression in cdo output - Added by Bjoern Hendrik Fock over 11 years ago

Hi Uwe,

I added the two files. They are result of a MARS request including area selection:

retrieve,
  type     = fc,
  stream   = oper,
  expver   = 1,
  class    = od,
  date     = $MY_DATE,
  time     = 00:00:00,
  step     = $STEP_START/to/$STEP_END/by/$STEP_INC,
  levtype  = model level,
  param    = ciwc,
  levelist = 47/to/91,
  grid     = 0.125/0.125,
  area     = 84/-5/73/75,
  target   = $MY_FILE

Only cloud fields like ciwc or clwc are affected. The cdo command itself is most likely not relevant as the same size change effect is observed for cdo copy.

Bjoern

Download all files

ecmwf_2011-04-02_fc00_3_to_9_by_3_ciwc.grb (14.8 MB) ecmwf_2011-04-02_fc00_3_to_9_by_3_ciwc.grb
ecmwf_2011-04-01_fc00_3_to_9_by_3_ciwc.grb (14.8 MB) ecmwf_2011-04-01_fc00_3_to_9_by_3_ciwc.grb

RE: Supress grb compression in cdo output - Added by Uwe Schulzweida over 11 years ago

Hi Bjoern,

Thanks for the data, that helps really a lot to solve the problem!
Some of the GRIB records are constant (all values are zero).
Constant fields are packed high efficiently by default. You can switch off this
feature by setting the following environment variable:

export GRIB_GRIBEX_MODE_ON=1

Cheers,
Uwe

RE: Supress grb compression in cdo output - Added by Bjoern Hendrik Fock over 11 years ago

Great that works.

Vielen Dank und schönes Wochenende, Björn

(1-4/4)

Project

General

Profile

CDO