ecmwf ensemble grib to Netcdf issue
Added by AJILESH P about 1 year ago
I am trying to convert ecmwf ensemble forecast data to Netcdf. It can be done by using grib_to_necdf. When I merge the files using cdo mergetime, the output is shuffled. The order of the ensemble is different in the input files. how to reorder and do the mergetime? sample grib files are attached.
Replies (5)
RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller about 1 year ago
hi!
the problem is that all records in each file have the exact same timestamp. merging both cannot work because the ensible dimension is missing (from the CDO perspective)
IMO CDO works best with keeping ensemble member data in separate files. you can use
cdo splitrec A1E03270000032706001 A1E03270000032706001_ens cdo splitrec A1E03270000032712001 A1E03270000032712001_ensto split the grib files. not sure, if the ensemble member have the same order in each input though.
RE: ecmwf ensemble grib to Netcdf issue - Added by AJILESH P about 1 year ago
Hi Ralf, thank you for the response, I am able to merge files using mergetime after converting to nc like this:
for i in `ls A1E032700*`; do grib_to_netcdf -D NC_FLOAT -o ${i}.nc $i; done
cdo mergetime *nc output.nc
cdo sinfov gives (This is with all the 61 files):
File format : NetCDF2
-1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name
1 : unknown unknown v instant 50 1 6391 1 F32 : tp
Grid coordinates :
1 : lonlat : points=6391 (77x83)
longitude : 72.4 to 87.6 by 0.2 degrees_east
latitude : 24.4 to 8 by -0.2 degrees_north
Vertical coordinates :
1 : generic : levels=50
number : 10 to 11
Time coordinate :
time : 61 steps
RefTime = 1900-01-01 00:00:00 Units = hours Calendar = gregorian
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
2023-03-27 00:00:00 2023-03-27 06:00:00 2023-03-27 12:00:00 2023-03-27 18:00:00
2023-03-28 00:00:00 2023-03-28 06:00:00 2023-03-28 12:00:00 2023-03-28 18:00:00
2023-03-29 00:00:00 2023-03-29 06:00:00 2023-03-29 12:00:00 2023-03-29 18:00:00
2023-03-30 00:00:00 2023-03-30 06:00:00 2023-03-30 12:00:00 2023-03-30 18:00:00
2023-03-31 00:00:00 2023-03-31 06:00:00 2023-03-31 12:00:00 2023-03-31 18:00:00
2023-04-01 00:00:00 2023-04-01 06:00:00 2023-04-01 12:00:00 2023-04-01 18:00:00
2023-04-02 00:00:00 2023-04-02 06:00:00 2023-04-02 12:00:00 2023-04-02 18:00:00
2023-04-03 00:00:00 2023-04-03 06:00:00 2023-04-03 12:00:00 2023-04-03 18:00:00
2023-04-04 00:00:00 2023-04-04 06:00:00 2023-04-04 12:00:00 2023-04-04 18:00:00
2023-04-05 00:00:00 2023-04-05 06:00:00 2023-04-05 12:00:00 2023-04-05 18:00:00
2023-04-06 00:00:00 2023-04-06 06:00:00 2023-04-06 12:00:00 2023-04-06 18:00:00
2023-04-07 00:00:00 2023-04-07 06:00:00 2023-04-07 12:00:00 2023-04-07 18:00:00
2023-04-08 00:00:00 2023-04-08 06:00:00 2023-04-08 12:00:00 2023-04-08 18:00:00
2023-04-09 00:00:00 2023-04-09 06:00:00 2023-04-09 12:00:00 2023-04-09 18:00:00
2023-04-10 00:00:00 2023-04-10 06:00:00 2023-04-10 12:00:00 2023-04-10 18:00:00
2023-04-11 00:00:00
cdo sinfon: Processed 1 variable over 61 timesteps [0.00s 29MB]
But, the problem is, if I select one ensemble member and check, the variable is shuffled among the ensemble members, since in each file ensemble order is different. Is there any way I can re-order the ensemble in the input files?
RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller about 1 year ago
I think the ensemble member ID is part of each record. let me see ...
RE: ecmwf ensemble grib to Netcdf issue - Added by Ralf Mueller about 1 year ago
ok, you seem to get the ensemble number for each file (after calling cdo splitrec ...) with this:
grib_dump ens000051.grb | grep perturbationNumber | rev | cut -d ' ' -f 1 | rev | cut -d ';' -f 1
If you put this number as part of the filename and sort them accordingly, you should get a consistent ensemble order, I hope
RE: ecmwf ensemble grib to Netcdf issue - Added by AJILESH P about 1 year ago
Hi, thank you for the reply. I tried the suggested way and it is working. Also, I found another way of getting the ensemble number in the correct order using cdo sellevel.
After grib_to_netcdf use cdo to select each ensemble member using sellevel and then mergetime. Finally merge all ensembles. A sample script is here:
for i in `ls ls A1E032700*`
do
grib_set -s type=pf -w type=cf "$i" "${i}".grb
grib_to_netcdf -D NC_FLOAT -o ${i}.nc "${i}".grb
done
for lev in $( seq 0 50 )
do
for file in $( ls A1E*nc )
do
cdo sellevel,"$lev" "$file" "${lev}"_"${file}"
done
cdo mergetime [0-9]*A1E*.nc "${lev}".nc
rm [0-9]*A1E*.nc
done
cdo merge [0-9]*.nc cdo.nc