Project

General

Profile

Piping mergetime into replace sets all timesteps to the first merged timestep

Added by Lukas Schefczyk 3 days ago

Hello,
not sure if usererror with brackets or bug :
I am trying to replace the accumulated total_preciption over time with hourly total_precipitation with the replace operator.
When replacing with a intermediate file it works fine, but when chaining the replace with the rest,
the fields for that var for each timestep gets set to the selected "first" timestep, but only the first 2 , the 3th works half as intended as the second timestep has much higher values then the rest.

Here is a reproducible example script with icon nwp data.

#number of cores
np=16
#this downloads 20 little hourly nwp files dwd opendata
wget  -r -nH -R "index.html*" --no-parent --cut-dirs=6 -A '*_00?_*' https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/{tot_prec,pmsl}/
find . -maxdepth 1 -name '*.bz2' -print0 | xargs -0 -I [] -P ${np} bunzip2 []

#i also tested it without -f nc and staying within grib2, but same results.
cdo -f nc -merge -mergetime [ ./*PMSL.grib2 ] -mergetime [ ./*TOT_PREC.grib2 ] all.nc

cdo info all.nc | head -n 20

#doing the replace with tempfile works
cdo mergetime [ -seltimestep,1 -selname,tp all.nc -deltat -selname,tp all.nc ] tp_temp_ok.nc
cdo replace all.nc tp_temp_ok.nc replaced_ok.nc

cdo info replaced_ok.nc | head -n 20

#doing replace in chain outputs 0 on all fields of replaced var with following chain, because first timestep is all 0.
cdo replace all.nc -mergetime [ -seltimestep,1 -selname,tp all.nc -deltat -selname,tp all.nc ]  all0.nc

cdo info all0.nc | head -n 20

#testing if other chaining works , it does.
cdo replace -selname,tp all.nc -selname,tp tp_temp_ok.nc chaintest.nc

cdo info chaintest.nc | head -n 20 

#Testing selstimestep 2
cdo replace all.nc -mergetime [ -seltimestep,2 -selname,tp all.nc -deltat -selname,tp all.nc ]  all0_step2.nc
# Here it takes timestep 2 and puts it everywhere
cdo info all0_step2.nc | head -n 20

#Testing selstimestep 3
cdo replace all.nc -mergetime [ -seltimestep,3 -selname,tp all.nc -deltat -selname,tp all.nc ]  all0_step3.nc
#This works like intended, but timestep2 is weird
cdo info all0_step3.nc | head -n 20

#Or look at fields with ncview 

ncview all.nc & ncview replaced_ok.nc & ncview all0_step2.nc & ncview all0_step3.nc

So something weird happening with the piping of mergetime into replace or did i used the brackets wrong?!

It Works with the tempfile and then replace, so this is a workaround but with unnecessary i/o :)

tested on 2 different machines with 2.4.0 and 2.4.2, sorry if this is fixed with a newer version, can't update....

First: Ubuntu 24.04.2 LTS within WSL2 Win11
cdo -V
Climate Data Operators version 2.4.0 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -std=gnu++20 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/cdo-E16453/cdo-2.4.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/cdo-E16453/cdo-2.4.0=/usr/src/cdo-2.4.0-1build3 -fopenmp -pthread
CXX version : g++ (Ubuntu 13.2.0-23ubuntu3) 13.2.0
C Compiler: gcc -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/cdo-E16453/cdo-2.4.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/cdo-E16453/cdo-2.4.0=/usr/src/cdo-2.4.0-1build3 -Wall -pedantic -fPIC -fopenmp -pthread -pthread
C version : gcc (Ubuntu 13.2.0-23ubuntu3) 13.2.0
F77 Compiler: f77 -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffile-prefix-map=/build/cdo-E16453/cdo-2.4.0=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -fcf-protection -fdebug-prefix-map=/build/cdo-E16453/cdo-2.4.0=/usr/src/cdo-2.4.0-1build3
F77 version : GNU Fortran (Ubuntu 13.2.0-23ubuntu3) 13.2.0
Features: 62GB 32threads c++20 OpenMP45 Fortran pthreads HDF5 NC4/HDF5/threadsafe OPeNDAP sz udunits2 proj xml2 magics curl fftw3 hirlam_extensions sse2
Libraries: yac/3.1.0 NetCDF/4.9.2 HDF5/1.10.10 proj/9.4.0 xml2/2.9.14 curl/8.5.0 magics/4.15.2
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr
CDI library version : 2.4.0
ecCodes library version : 2.34.1
NetCDF library version : 4.9.2 of Mar 31 2024 08:09:36 $
exse library version : 1.5.0
FILE library version : 1.9.1

Second: Windows Server with cygwin64
Climate Data Operators version 2.4.2 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-cygwin
CXX Compiler: g++ -std=gnu++20 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=3 -fstack
-protector-strong --param=ssp-buffer-size=4 -Werror=return-type -ffile-prefix-map=/cygdrive/d/a/scallywag/cd
o/cdo-2.4.2-1.x86_64/build=/usr/src/debug/cdo-2.4.2-1 -ffile-prefix-map=/cygdrive/d/a/scallywag/cdo/cdo-2.4.
2-1.x86_64/src/cdo-2.4.2=/usr/src/debug/cdo-2.4.2-1 -D_GNU_SOURCE -fopenmp -pthread
CXX version : g++ (GCC) 11.4.0
CXX library : ranges
C Compiler: gcc -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=3 -fstack-protector-stro
ng --param=ssp-buffer-size=4 -ffile-prefix-map=/cygdrive/d/a/scallywag/cdo/cdo-2.4.2-1.x86_64/build=/usr/src
/debug/cdo-2.4.2-1 -ffile-prefix-map=/cygdrive/d/a/scallywag/cdo/cdo-2.4.2-1.x86_64/src/cdo-2.4.2=/usr/src/d
ebug/cdo-2.4.2-1 -fopenmp -pthread -pthread
C version : gcc (GCC) 11.4.0
F77 Compiler: gfortran -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=3 -fstack-protect
or-strong --param=ssp-buffer-size=4 -ffile-prefix-map=/cygdrive/d/a/scallywag/cdo/cdo-2.4.2-1.x86_64/build=/
usr/src/debug/cdo-2.4.2-1 -ffile-prefix-map=/cygdrive/d/a/scallywag/cdo/cdo-2.4.2-1.x86_64/src/cdo-2.4.2=/us
r/src/debug/cdo-2.4.2-1
F77 version : GNU Fortran (GCC) 11.4.0
Features: 766GB 48threads c++20 OpenMP45 Fortran pthreads HDF5 NC4/HDF5 OPeNDAP sz udunits2 proj xml2 curl f
ftw3 hirlam_extensions sse2
Libraries: yac/3.1.0 NetCDF/4.9.2 HDF5/1.12.3 proj/9.4.0 xml2/2.10.4 curl/8.11.0(h8.8.0)
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr
CDI library version : 2.4.2
cgribex library version : 2.3.0
ecCodes library version : 2.33.0
NetCDF library version : 4.9.2 of Jan 27 2024 20:17:03 $
exse library version : 2.0.0
FILE library version : 1.9.1


Replies (1)

RE: Piping mergetime into replace sets all timesteps to the first merged timestep - Added by Uwe Schulzweida 3 days ago

Hello Lukas,

This is definitely a bug in CDO! We will fix this problem in the next CDO release 2.5.3.
Thank you very much for this well prepared report!

Cheers,
Uwe

    (1-1/1)