ecmwf ensemble grib to Netcdf issue

I am trying to convert ecmwf ensemble forecast data to Netcdf. It can be done by using grib_to_necdf. When I merge the files using cdo mergetime, the output is shuffled. The order of the ensemble is different in the input files. how to reorder and do the mergetime? sample grib files are attached.

Download all files

A1E03270000032712001 (645 KB) A1E03270000032712001
A1E03270000032706001 (645 KB) A1E03270000032706001

Replies (5)

RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller over 1 year ago

hi!

the problem is that all records in each file have the exact same timestamp. merging both cannot work because the ensible dimension is missing (from the CDO perspective)

IMO CDO works best with keeping ensemble member data in separate files. you can use

cdo splitrec A1E03270000032706001 A1E03270000032706001_ens 
cdo splitrec A1E03270000032712001 A1E03270000032712001_ens

to split the grib files. not sure, if the ensemble member have the same order in each input though.

RE: ecmwf ensemble grib to Netcdf issue - Added by AJILESH P over 1 year ago

Hi Ralf, thank you for the response, I am able to merge files using mergetime after converting to nc like this:

for i in `ls A1E032700*`; do grib_to_netcdf -D NC_FLOAT -o ${i}.nc $i; done
cdo mergetime *nc output.nc

cdo sinfov gives (This is with all the 61 files):

  File format : NetCDF2
    -1 : Institut Source   T Steptype Levels Num    Points Num Dtype : Parameter name
     1 : unknown  unknown  v instant      50   1      6391   1  F32  : tp
   Grid coordinates :
     1 : lonlat                   : points=6391 (77x83)
                        longitude : 72.4 to 87.6 by 0.2 degrees_east
                         latitude : 24.4 to 8 by -0.2 degrees_north
   Vertical coordinates :
     1 : generic                  : levels=50
                           number : 10 to 11
   Time coordinate :
                             time : 61 steps
     RefTime =  1900-01-01 00:00:00  Units = hours  Calendar = gregorian
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  2023-03-27 00:00:00  2023-03-27 06:00:00  2023-03-27 12:00:00  2023-03-27 18:00:00
  2023-03-28 00:00:00  2023-03-28 06:00:00  2023-03-28 12:00:00  2023-03-28 18:00:00
  2023-03-29 00:00:00  2023-03-29 06:00:00  2023-03-29 12:00:00  2023-03-29 18:00:00
  2023-03-30 00:00:00  2023-03-30 06:00:00  2023-03-30 12:00:00  2023-03-30 18:00:00
  2023-03-31 00:00:00  2023-03-31 06:00:00  2023-03-31 12:00:00  2023-03-31 18:00:00
  2023-04-01 00:00:00  2023-04-01 06:00:00  2023-04-01 12:00:00  2023-04-01 18:00:00
  2023-04-02 00:00:00  2023-04-02 06:00:00  2023-04-02 12:00:00  2023-04-02 18:00:00
  2023-04-03 00:00:00  2023-04-03 06:00:00  2023-04-03 12:00:00  2023-04-03 18:00:00
  2023-04-04 00:00:00  2023-04-04 06:00:00  2023-04-04 12:00:00  2023-04-04 18:00:00
  2023-04-05 00:00:00  2023-04-05 06:00:00  2023-04-05 12:00:00  2023-04-05 18:00:00
  2023-04-06 00:00:00  2023-04-06 06:00:00  2023-04-06 12:00:00  2023-04-06 18:00:00
  2023-04-07 00:00:00  2023-04-07 06:00:00  2023-04-07 12:00:00  2023-04-07 18:00:00
  2023-04-08 00:00:00  2023-04-08 06:00:00  2023-04-08 12:00:00  2023-04-08 18:00:00
  2023-04-09 00:00:00  2023-04-09 06:00:00  2023-04-09 12:00:00  2023-04-09 18:00:00
  2023-04-10 00:00:00  2023-04-10 06:00:00  2023-04-10 12:00:00  2023-04-10 18:00:00
  2023-04-11 00:00:00
cdo    sinfon: Processed 1 variable over 61 timesteps [0.00s 29MB]

But, the problem is, if I select one ensemble member and check, the variable is shuffled among the ensemble members, since in each file ensemble order is different. Is there any way I can re-order the ensemble in the input files?

RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller over 1 year ago

I think the ensemble member ID is part of each record. let me see ...

RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller over 1 year ago

ok, you seem to get the ensemble number for each file (after calling cdo splitrec ...) with this:

grib_dump ens000051.grb | grep perturbationNumber | rev | cut -d ' ' -f 1 | rev  | cut -d ';' -f 1

If you put this number as part of the filename and sort them accordingly, you should get a consistent ensemble order, I hope

RE: ecmwf ensemble grib to Netcdf issue - Added by AJILESH P over 1 year ago

Hi, thank you for the reply. I tried the suggested way and it is working. Also, I found another way of getting the ensemble number in the correct order using cdo sellevel.
After grib_to_netcdf use cdo to select each ensemble member using sellevel and then mergetime. Finally merge all ensembles. A sample script is here:

 for i in `ls ls A1E032700*`
 do
     grib_set -s type=pf -w type=cf "$i" "${i}".grb
     grib_to_netcdf -D NC_FLOAT -o ${i}.nc "${i}".grb
 done

 for lev in $( seq 0 50 )
 do
     for file in $( ls A1E*nc )
     do
         cdo sellevel,"$lev" "$file" "${lev}"_"${file}" 
     done
     cdo mergetime [0-9]*A1E*.nc "${lev}".nc
     rm [0-9]*A1E*.nc
 done

 cdo merge [0-9]*.nc cdo.nc

(1-5/5)

Project

General

Profile

CDO