Selecting multiple variables, at multiple levels and add to a single file

Hello,

I'm currently working with a dataset of multiple grib files, that contain various parameters, of which some are multi-level, and some are single level. Every file contains information for one timestep. My goal is to merge all those files into a single nc file, I need all the parameters, but not all the levels.

How can I do this in the fastest way possible?

I've tried multiple solutions with varying results:

1- This one liner solution gives me the error below (var165 is a single level parameter):

cdo select,name=var165 input1.grib input2.grib output.nc

cdo    select (Warning): Input streams have different parameter names!
     cdo    select (Abort): Number of levels of the input parameters do not match!

2- Trying the same command with a multiple level parameter yields the same error:

cdo select,name=var131,level=30000,40000 input1.grib input2.grib output.nc

3- I then tried to loop over all the single grib files, convert to a nc file with the parameters I want, and then concatenate everything. Unfortunately, I couldn't find a way to  select both single level parameters and required levels from the multilevel parameters at the same time

Consider var165 and var165 are single level, and var131 and var132 have 11 pressure levels

3.1 - This solution will do what I want, but I don't need all the levels and I don't know how to do the selection
     files=()
     for FILE in ${GRIBS[@]};
     do
        cdo -f nc4 select,name=var165,var166,var131,var132 $FILE "$FILE.nc" 
        files+=($FILE.nc)
     done

ncrcat -A ${files[@]} output.nc

3.2 - I also tried to divide into single level parameters and multilevel parameters files, but at the end I can't combine them into a single file, because not all the files have the same parameters inside.

files=()
     for FILE in ${GRIBS[@]};
     do
        cdo -f nc4 select,name=var165,var166 $FILE "$FILE_SINGLE.nc" 
        cdo -f nc4 select,name=var131,var132,level=30000,40000 $FILE "$FILE_MULTI.nc" 
     done

a=(*.nc)
     files=()
     for FILE in ${a[@]}
     do
     files+=($FILE)
     done

ncrcat -A ${files[@]} output.nc

I feel like I am close to the solution, but its the first time I use CDO, and I need a little help to get past this.

I have a big dataset that I will have to transform on a daily basis, so I'm looking for the fastest solution too.

I'm using CDO version 1.9.9 on a CENTOS7.0 and I've attached 2 of the files.

Cheers,
Paul

Download all files

input2.grib (40.1 MB) input2.grib
input1.grib (40.1 MB) input1.grib

Replies (6)

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Ralf Mueller about 4 years ago

hi Paul!

for your two input files I have a simple solution, but it creates a lot of temporal output:

cdo splitparam input1.grib input1_
cdo splitparam input2.grib input2_
cdo cat input1_166.128.grib input2_166.128.grib input_166.128.grib
cdo cat input1_165.128.grib input2_165.128.grib input_165.128.grib
cdo cat input1_132.128.grib input2_132.128.grib input_132.128.grib
cdo cat input1_131.128.grib input2_131.128.grib input_131.128.grib
cdo merge input_166.128.grib input_165.128.grib input_132.128.grib input_131.128.grib inputAll.grib

the problem is, that your inputs have the same fields but in different order.

hm, are the variables 166 165 132 131 the only variables your are interested in in all your inputs?

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Paul Mota about 4 years ago

Hi Ralf,

Thank you for taking the time to answer my question.

Unfortunately I have a total of 28 parameters, and 41 files, so I'm not sure if your solution will be viable.

Plus the problem of retaining only a few of the pressure levels remain. I could, of course, select only the levels I'm interested in with something like:

cdo select,name=var165,level=30000,40000 input_165.128.grib output_165.grib

Due to the large number of parameters and files, programming all this instructions seems a very complex task.

My solution 3.1 does almost everything I want. Its somewhat fast, and allows me to join all the parameters with the correct timeaxis in a single file. The only hiccup, is that I end up with all the pressure levels in the file. Is there a way I can remove some of the levels a posteriori ? Or any other workaround I am not seeing?

I will try to implement and test your solution anyway.

Cheers

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Ralf Mueller about 4 years ago

for getting rid of levels, you can use the delete operator. It used the same syntax as select

hth
ralf

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Ralf Mueller about 4 years ago

BTW: on grib you can use the Unix command -1 : Date Time Level Gridsize 1 : 2021-02-19 03:00:00 100000 1166961 2 : 2021-02-19 03:00:00 95000 1166961 3 : 2021-02-19 03:00:00 92500 1166961 4 : 2021-02-19 03:00:00 90000 1166961 5 : 2021-02-19 03:00:00 85000 1166961 6 : 2021-02-19 03:00:00 80000 1166961 7 : 2021-02-19 03:00:00 70000 1166961 8 : 2021-02-19 03:00:00 60000 1166961 9 : 2021-02-19 03:00:00 50000 1166961 10 : 2021-02-19 03:00:00 40000 1166961 11 : 2021-02-19 03:00:00 30000 1166961 12 : 2021-02-19 03:00:00 100000 1166961 13 : 2021-02-19 03:00:00 95000 1166961 14 : 2021-02-19 03:00:00 92500 1166961 15 : 2021-02-19 03:00:00 90000 1166961 16 : 2021-02-19 03:00:00 85000 1166961 17 : 2021-02-19 03:00:00 80000 1166961 18 : 2021-02-19 03:00:00 70000 1166961 19 : 2021-02-19 03:00:00 60000 1166961 20 : 2021-02-19 03:00:00 50000 1166961 21 : 2021-02-19 03:00:00 40000 1166961 22 : 2021-02-19 03:00:00 30000 1166961 23 : 2021-02-19 03:00:00 0 1166961 24 : 2021-02-19 03:00:00 0 1166961 25 : 2021-02-19 06:00:00 100000 1166961 26 : 2021-02-19 06:00:00 95000 1166961 27 : 2021-02-19 06:00:00 92500 1166961 28 : 2021-02-19 06:00:00 90000 1166961 29 : 2021-02-19 06:00:00 85000 1166961 30 : 2021-02-19 06:00:00 80000 1166961 31 : 2021-02-19 06:00:00 70000 1166961 32 : 2021-02-19 06:00:00 60000 1166961 33 : 2021-02-19 06:00:00 50000 1166961 34 : 2021-02-19 06:00:00 40000 1166961 35 : 2021-02-19 06:00:00 30000 1166961 36 : 2021-02-19 06:00:00 100000 1166961 37 : 2021-02-19 06:00:00 95000 1166961 38 : 2021-02-19 06:00:00 92500 1166961 39 : 2021-02-19 06:00:00 90000 1166961 40 : 2021-02-19 06:00:00 85000 1166961 41 : 2021-02-19 06:00:00 80000 1166961 42 : 2021-02-19 06:00:00 70000 1166961 43 : 2021-02-19 06:00:00 60000 1166961 44 : 2021-02-19 06:00:00 50000 1166961 45 : 2021-02-19 06:00:00 40000 1166961 46 : 2021-02-19 06:00:00 30000 1166961 47 : 2021-02-19 06:00:00 0 1166961 48 : 2021-02-19 06:00:00 0 1166961 : Date Time Level Gridsize
In order to keep the temporal 'cat'. So another options is, to call

cat input1.grib input2.grib > OUT.grib

This would then be your inventory of everything

cdo infov OOO.nc Miss : Minimum Mean Maximum : Parameter name 0 : -23.079 -0.55080 19.437 : var132 0 : -29.561 -0.62973 30.064 : var132 0 : -29.582 -0.61683 31.824 : var132 0 : -30.495 -0.59842 33.490 : var132 0 : -30.315 -0.48918 34.498 : var132 0 : -29.112 -0.37282 33.951 : var132 0 : -29.567 -0.034833 30.855 : var132 0 : -31.682 0.16312 34.911 : var132 0 : -37.659 -0.027699 43.841 : var132 0 : -42.032 -0.34253 50.343 : var132 0 : -46.946 -0.29278 66.116 : var132 0 : -18.833 -0.64328 20.307 : var131 0 : -28.300 -0.39190 26.060 : var131 0 : -27.701 -0.20517 27.674 : var131 0 : -27.489 0.083479 28.839 : var131 0 : -27.329 0.80785 31.281 : var131 0 : -25.378 1.5390 32.622 : var131 0 : -23.312 2.9813 40.625 : var131 0 : -29.217 4.5683 53.845 : var131 0 : -27.574 6.5425 65.676 : var131 0 : -27.338 9.0860 77.662 : var131 0 : -24.100 12.144 93.807 : var131 0 : -18.901 -0.60735 20.318 : var165 0 : -19.837 -0.45204 19.445 : var166 0 : -21.734 -0.55599 21.547 : var132 0 : -29.461 -0.61262 31.820 : var132 0 : -30.414 -0.60025 32.570 : var132 0 : -29.575 -0.58349 33.472 : var132 0 : -29.663 -0.47422 34.806 : var132 0 : -30.626 -0.40788 33.499 : var132 0 : -33.368 -0.084603 32.820 : var132 0 : -32.069 0.094793 35.587 : var132 0 : -37.365 -0.032287 43.323 : var132 0 : -39.875 -0.25279 52.406 : var132 0 : -49.612 -0.27595 65.919 : var132 0 : -19.942 -0.65450 21.808 : var131 0 : -27.407 -0.38803 27.171 : var131 0 : -29.232 -0.21561 29.424 : var131 0 : -30.128 0.050246 31.888 : var131 0 : -28.632 0.80091 30.587 : var131 0 : -25.760 1.5778 31.756 : var131 0 : -24.876 3.0202 39.311 : var131 0 : -27.688 4.5700 54.124 : var131 0 : -28.683 6.5805 67.973 : var131 0 : -29.984 9.0326 78.016 : var131 0 : -21.773 12.118 88.477 : var131 0 : -18.797 -0.63196 21.640 : var165 0 : -19.344 -0.46929 22.063 : var166 Miss : Minimum Mean Maximum : Parameter name order, you need to make sure the order of input file in the 'cat' call follows this.

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Paul Mota about 4 years ago

Thank you so much Ralf,

This last solution work very well. Unfortunately after I join all the grib files, I fail to convert to netcdf because not all the grib files have the same parameters. But I can easely deal with this problem by adding them to the original files before concatenating everything.

I will report back with my final solution.

Cheers

RE: Selecting multiple variables, at multiple levels and add to a single file - Added by Ralf Mueller about 4 years ago

just in case you are not aware of it: For converting grib to netcdf there are infact several options

cdo -f grb -copy ...
ncl_convert2nc from the NCL package
grib_to_netcdf from the eccodes package

I usually check them in this order.

cheers
ralf

(1-6/6)

Project

General

Profile

CDO