merge does not work well with Argument Groups
Added by Etienne Tourigny almost 2 years ago
Dear all,
I am trying to use merge on some results done using argument groups, but it seems that only the first argument group is processed. The script processes many files with several timesteps, but I have limited the number of files and timesteps to the bare minimum.
This works on a collaborator's system using cdo-2.0.6 but it does not work on one of my system (with both cdo-2.0.6 and cdo-2.1.1) and another independent system using cdo-2.1.1.
If I do the sub-commands for each file separately and then merge the results it works, but this implies extra I/O as intermediate files are generated. Could someone shed some light on this please?
Attaching the 2 reduced example files.
c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo -O merge [ [ -seltimestep,1 E5ml00_1H_2000-01-01_129-1 ] [ -seltimestep,1 E5ml00_1H_2000-01-01_152-1 ] ] tmp1 cdo(1) seltimestep: Process started cdo(1) seltimestep: Processed 410240 values from 1 variable over 1 timestep cdo merge: Processed 410240 values from 1 variable over 1 timestep [0.40s 75MB] c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo showvar tmp1 lnsp cdo showname: Processed 1 variable [0.33s 56MB] c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo -O -seltimestep,1 E5ml00_1H_2000-01-01_129-1 tmp1 cdo seltimestep: Processed 1 variable over 1 timestep [0.34s 68MB] c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo -O -seltimestep,1 E5ml00_1H_2000-01-01_152-1 tmp2 cdo seltimestep: Processed 1 variable over 1 timestep [0.35s 68MB] c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo -O merge [ tmp1 tmp2 ] tmp3 cdo merge: Processed 2 variables over 2 timesteps [0.36s 70MB] c3et@ac6-200: /perm/c3et/nudging/tmp/tmp > cdo showvar tmp3 z lnsp cdo showname: Processed 2 variables [0.34s 62MB]
Replies (8)
RE: merge does not work well with Argument Groups - Added by Uwe Schulzweida almost 2 years ago
Dear Etienne,
I don't know if this grouping is supposed to work like this, I'll ask our grouping expert.
What works is:
cdo -O merge [ -seltimestep,1 E5ml00_1H_2000-01-01_129-1 -seltimestep,1 E5ml00_1H_2000-01-01_152-1 ] tmp1or this more compact version (unfortunately not documented yet):
cdo -O merge [ -seltimestep,1: E5ml00_1H_2000-01-01_129-1 E5ml00_1H_2000-01-01_152-1 ] tmp1Cheers,
Uwe
RE: merge does not work well with Argument Groups - Added by Oliver Heidmann almost 2 years ago
It should work. Sorry for the inconvenience. Fixed it.
The problem was quite easy to find with your report :)
I will recheck other nested scenarios on Monday and with the next version this issue should be fixed.
Kind regards
Oliver Heidmann
RE: merge does not work well with Argument Groups - Added by Etienne Tourigny almost 2 years ago
Thanks for such as quick response Uwe and finding a fix Oliver!
Uwe's suggestion works for this simple case:
cdo -O merge [ -seltimestep,1 E5ml00_1H_2000-01-01_129-1 -seltimestep,1 E5ml00_1H_2000-01-01_152-1 ] tmp1
however, the full commandline requires `apply` with inner grouping and this does not work
#
eval ${CDO} O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_129 ] mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_152 ] mergetime -apply,-seltimestep,1/24/\${MONTH}-*_130 ]
6 [ ${POOL}/${YEAR}/*_1H_${YEAR}mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_138 ] mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}${MONTH}-*_155 ] ${NDGPATH}/${\
NDGTAG}${YEAR}${MONTH}.sp
eval ${CDO} -O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_129 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_152 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_130 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_138 ] ] [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_155 ] ] ${NDGPATH}/${NDGTAG}${YEAR}${MONTH}.sp
using the proposed workaround:
eval ${CDO} -O -L -merge [ -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_129 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_152 ] -mergetime -apply,-seltimestep,1/24/\
6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_130 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_138 ] -mergetime -apply,-seltimestep,1/24/6 [ ${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}-*_155 ] ${NDGPATH}/${\
NDGTAG}${YEAR}${MONTH}.sp
is executed as such:
++ cdo -O -L -merge '[' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_129 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_129 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_152 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_152 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_130 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_130 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_138 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_138 ']' -mergetime -apply,-seltimestep,1/24/6 '[' /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_155 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_155 ']' /perm/c3et/nudging/tmp/input_files/era5_/era5_200001//era5_200001.sp
cdo (Abort): -merge [ -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_129 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_129 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_152 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_152 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_130 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_130 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_138 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_138 ] -mergetime -apply,-seltimestep,1/24/6 [ /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-01_155 /perm/c3et/nudging/tmp/input_files/era5_raw/2000/E5ml00_1H_2000-01-02_155 ] /perm/c3et/nudging/tmp/input_files/era5_/era5_200001//era5_200001.sp
^ Using two operators with variable input without using sub groups [] is not allowed
RE: merge does not work well with Argument Groups - Added by Oliver Heidmann almost 2 years ago
This one is hard to parse with my brain.
For this I have to know where each operator needs to be attached (aka its parent).
I guess that every mergetimestep is a child for the first merge operator?
In that case every thing should work when you add the missing brackets for -merge.
Currently the parser cannot know if the -mergetimestep is meant to be added to the preceding -mergetimestep or to any of the further preceding operators with variable number of inputs.
I rewrote the whole call into a shell script for more visibility:
First argument: cdo path
Result: prints the resulting call
Only thing added to it are the mentioned brackets around all the input of merge.
Since I dont have the files, testing this my self would take quite a while.
CDO=$1
SELECT="-mergetime -apply,seltimestep,1,24/6"
POOL=""
YEAR=""
MONTH=""
DATA_PATH="${POOL}/${YEAR}/*_1H_${YEAR}-${MONTH}"
NDGPATH=""
NDTAG=""
OFILE=${NDGPATH}/${NDGTAG}${YEAR}${MONTH}.sp
echo "${CDO} -O -L -merge [ [ $SELECT [ $DATA_PATH-*_129 ] ] [ $SELECT [ $DATA_PATH-*_152 ] ] [ $SELECT [ $DATA_PATH-*_130 ] ] [ $SELECT [ $DATA_PATH-*_138 ] ] [ $SELECT [ $DATA_PATH-*_155 ] ] ] $OFILE"
edit : fixed SELECT not containing the seltimestep operator
edit2: fixed missing $PATH
edit3: fixed usage of name PATH -> changed to DATA_PATH
Different question:
I just realized what a monster I created with the brackets and the apply operator. While I think this is fascinating, in terms of usability this seems to be a nightmare.
So my question is: How often does it happen that cdo calls with this complexity are required? Once a week? Once a Month? Daily?
Kind regards
Oliver Heidmann
RE: merge does not work well with Argument Groups - Added by Etienne Tourigny almost 2 years ago
Hi Oliver, thanks for your time and patience.
the way you re-wrote the command is exactly what is required. The merging should happen on the results of the 5 different mergetime operations.
I don't have all the files myself as I am currently testing this. But it should work as-is with the few sample files I provided.
These scripts will be used to process decades of data (ERA5 data used for atmospheric nudging) like this, then we will update it very few months/years when we need updated data.
RE: merge does not work well with Argument Groups - Added by Oliver Heidmann almost 2 years ago
I just realized that my answer applies to my branch only.
This will not work with the current CDO version.
When do you need this?
RE: merge does not work well with Argument Groups - Added by Etienne Tourigny almost 2 years ago
Hi Olivier,
it's not very urgent as there is an easy (although less efficient) workaround, which is to generate temporary files for each of the children mergetime nodes. I mainly wanted to check if the problem I have can be fixed in future versions.
RE: merge does not work well with Argument Groups - Added by Oliver Heidmann almost 2 years ago
I have misread some of the information:
The error message about the missing usage of [] seems to be actually wrong.
I am currently checking that.