CDO

Dear Etienne,

I don't know if this grouping is supposed to work like this, I'll ask our grouping expert.
What works is:

cdo -O merge [ -seltimestep,1 E5ml00_1H_2000-01-01_129-1 -seltimestep,1 E5ml00_1H_2000-01-01_152-1 ] tmp1

cdo -O merge [ -seltimestep,1: E5ml00_1H_2000-01-01_129-1 E5ml00_1H_2000-01-01_152-1 ] tmp1

It should work. Sorry for the inconvenience. Fixed it.
The problem was quite easy to find with your report :)
I will recheck other nested scenarios on Monday and with the next version this issue should be fixed.

Kind regards
Oliver Heidmann

Thanks for such as quick response Uwe and finding a fix Oliver!

Uwe's suggestion works for this simple case:

cdo -O merge [ -seltimestep,1 E5ml00_1H_2000-01-01_129-1 -seltimestep,1 E5ml00_1H_2000-01-01_152-1 ] tmp1

however, the full commandline requires `apply` with inner grouping and this does not work

#
eval ${CDO} O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_129 ] mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_152 ] mergetime -apply,-seltimestep,1/24/\
6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_130 ] mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_138 ] mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_155 ] ${NDGPATH}/${\
NDGTAG}${YEAR}${MONTH}.sp

eval ${CDO} -O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_129 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_152 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_130 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_138 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_155 ] ] ${NDGPATH}/${NDGTAG}${YEAR}${MONTH}.sp                                                                                                                                                                                                

using the proposed workaround:

eval ${CDO} -O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_129 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_152 ] -mergetime -apply,-seltimestep,1/24/\
6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_130 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_138 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_155 ] ${NDGPATH}/${\
NDGTAG}${YEAR}${MONTH}.sp

is executed as such:

++ cdo -O -L -merge '[' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_129 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_129 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_152 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_152 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_130 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_130 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_138 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_138 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_155 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_155 ']' /perm/c3et/nudging/tmp/input_files/era5_/era5_200001//era5_200001.sp

cdo (Abort): -merge [ -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_129 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_129 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_152 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_152 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_130 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_130 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_138 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_138 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_155 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_155 ] /perm/c3et/nudging/tmp/input_files/era5_/era5_200001//era5_200001.sp  
                                                                                                                                                                                                                         ^ Using two operators with variable input without using sub groups [] is not allowed

This one is hard to parse with my brain.
For this I have to know where each operator needs to be attached (aka its parent).
I guess that every mergetimestep is a child for the first merge operator?
In that case every thing should work when you add the missing brackets for -merge.
Currently the parser cannot know if the -mergetimestep is meant to be added to the preceding -mergetimestep or to any of the further preceding operators with variable number of inputs.

I rewrote the whole call into a shell script for more visibility:
First argument: cdo path
Result: prints the resulting call

Only thing added to it are the mentioned brackets around all the input of merge.
Since I dont have the files, testing this my self would take quite a while.

CDO=$1
SELECT="-mergetime -apply,seltimestep,1,24/6" 
POOL="" 
YEAR="" 
MONTH="" 
DATA_PATH="${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}" 
NDGPATH="" 
NDTAG="" 

OFILE=${NDGPATH}/${NDGTAG}${YEAR}${MONTH}.sp

echo "${CDO} -O -L -merge [ [ $SELECT [ $DATA_PATH-*_129 ] ] [ $SELECT [ $DATA_PATH-*_152 ] ] [ $SELECT [ $DATA_PATH-*_130 ] ] [ $SELECT [ $DATA_PATH-*_138 ] ] [ $SELECT [ $DATA_PATH-*_155 ] ] ] $OFILE" 

Different question:
I just realized what a monster I created with the brackets and the apply operator. While I think this is fascinating, in terms of usability this seems to be a nightmare.
So my question is: How often does it happen that cdo calls with this complexity are required? Once a week? Once a Month? Daily?

Kind regards
Oliver Heidmann

Hi Oliver, thanks for your time and patience.

the way you re-wrote the command is exactly what is required. The merging should happen on the results of the 5 different mergetime operations.

I don't have all the files myself as I am currently testing this. But it should work as-is with the few sample files I provided.

These scripts will be used to process decades of data (ERA5 data used for atmospheric nudging) like this, then we will update it very few months/years when we need updated data.

I just realized that my answer applies to my branch only.
This will not work with the current CDO version.
When do you need this?

Hi Olivier,
it's not very urgent as there is an easy (although less efficient) workaround, which is to generate temporary files for each of the children mergetime nodes. I mainly wanted to check if the problem I have can be fixed in future versions.

I have misread some of the information:
The error message about the missing usage of [] seems to be actually wrong.
I am currently checking that.