operator chain
Added by Ivana Cvijanovic 13 days ago
Hi,
I have a .csh script that loops through the years and calculates the result of an expression. My input variables are in separate files. I have lots of data and it is impractical to write out the result for -merge first and then do -expr as a separate step. I would like to perform merge and expr without writing out the intermediate file.
Here is what I tried to do:
set u10 = "${work}/u10/u10m_global_${year}.nc"
set v10 = "${work}/v10/v10m_global_${year}.nc"
set ws = "${work}/ws_global_${year}.nc"
cdo -expr,'ws=sqrt(u10^2+v10^2)*0.725727;' -merge "$u10" "$v10" "$ws"
And it doesnt work as it tries to open "$ws" file instead of creating it:
cdo(1) merge: Process started
HDF5-DIAG: Error detected in HDF5 (2.2.0) thread 2:
#000: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5F.c line 490 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
major: File accessibility
minor: Not an HDF5 file
#001: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLcallback.c line 4109 in H5VL_file_specific(): file specific failed
major: Virtual Object Layer
minor: Can't operate on object
#002: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLcallback.c line 4045 in H5VL__file_specific(): file specific failed
major: Virtual Object Layer
minor: Can't operate on object
#003: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLnative_file.c line 344 in H5VL__native_file_specific(): error in HDF5 file check
major: File accessibility
minor: Can't get value
#004: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5Fint.c line 1084 in H5F__is_hdf5(): unable to open file
major: File accessibility
minor: Unable to initialize object
#005: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5FD.c line 967 in H5FD_open(): can't open file
major: Virtual File Layer
minor: Unable to open file
#006: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5FDsec2.c line 288 in H5FD__sec2_open(): unable to open file: name = '/Volumes/BigData/ws_global_1976.nc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessibility
minor: Unable to open file
Is there a way to correctly connect these two commands ? Thank you.
Replies (5)
RE: operator chain - Added by Uwe Schulzweida 9 days ago
Hello Ivana,
I can't reproduce the problem. I suspect you're using an older version of CDO. There was an issue with concurrent access to multiple netCDF4 files. In that case, you need to use the CDO option -L:
cdo -L -expr,'ws=sqrt(u10^2+v10^2)*0.725727;' -merge "$u10" "$v10" "$ws"If that doesn't help, we'll need the output of “cdo -V”.
Cheers,
Uwe
RE: operator chain - Added by Ivana Cvijanovic 7 days ago
Hi Uwe,
Thank you for looking into this.
Here is cdo-V:
Climate Data Operators version 2.5.0 (https://mpimet.mpg.de/cdo)
System: aarch64-apple-darwin20.0.0
CXX Compiler: arm64-apple-darwin20.0.0-clang++ -std=gnu++20 -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm/include -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/cdo_1776950124603/work=/usr/local/src/conda/cdo-2.5.0 -fdebug-prefix-map=/opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY -fopenmp -pthread
CXX version : clang version 19.1.7
CXX library :
C Compiler: arm64-apple-darwin20.0.0-clang -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm/include -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/cdo_1776950124603/work=/usr/local/src/conda/cdo-2.5.0 -fdebug-prefix-map=/opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm=/usr/local/src/conda-prefix -fopenmp -pthread -pthread
C version : clang version 19.1.7
F77 Compiler: /Users/runner/miniforge3/conda-bld/cdo_1776950124603/_build_env/bin/arm64-apple-darwin20.0.0-gfortran -march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm/include -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/cdo_1776950124603/work=/usr/local/src/conda/cdo-2.5.0 -fdebug-prefix-map=/opt/homebrew/Caskroom/miniconda/base/envs/cdo-arm=/usr/local/src/conda-prefix
F77 version : GNU Fortran (GCC) 14.3.0
Features: 48GB 14threads c++20 OpenMP51 Fortran pthreads HDF5 NC4/HDF5 OPeNDAP udunits2 proj xml2 magics curl fftw3
Libraries: yac/3.1.0 NetCDF/4.10.0 HDF5/2.2.0(h2.1.0) proj/9.7.1 xml2/2.14.6 curl/8.22.0(h8.19.0) magics/4.16.0
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr
CDI library version : 2.5.0
cgribex library version : 2.3.1
ecCodes library version : 2.47.0
NetCDF library version : 4.10.0 of Jul 4 2026 19:09:54 $
exse library version : 2.0.0
FILE library version : 1.9.1
I tried using -L with different data and I get the same problem:
cdo -L -expr,'rh = 100*exp(17.625 * ((d2m-273.15)/ (243.04 + d2m-273.15) - (t2m-273.15)/ (243.04 + t2m-273.15)));' -merge /Volumes/BigData/ERA5_global/ta/ta_global_2000.nc /Volumes/BigData/ERA5_global/td/td_global_2000.nc /Volumes/BigData/ERA5_global/rh_global_2000.nc
cdo(1) merge: Process started
HDF5-DIAG: Error detected in HDF5 (2.2.0) thread 2:
#000: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5F.c line 490 in H5Fis_accessible(): unable to determine if file is accessible as HDF5
major: File accessibility
minor: Not an HDF5 file
#001: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLcallback.c line 4109 in H5VL_file_specific(): file specific failed
major: Virtual Object Layer
minor: Can't operate on object
#002: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLcallback.c line 4045 in H5VL__file_specific(): file specific failed
major: Virtual Object Layer
minor: Can't operate on object
#003: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5VLnative_file.c line 344 in H5VL__native_file_specific(): error in HDF5 file check
major: File accessibility
minor: Can't get value
#004: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5Fint.c line 1084 in H5F__is_hdf5(): unable to open file
major: File accessibility
minor: Unable to initialize object
#005: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5FD.c line 967 in H5FD_open(): can't open file
major: Virtual File Layer
minor: Unable to open file
#006: /Users/runner/miniforge3/conda-bld/hdf5_1786232861201/work/src/H5FDsec2.c line 288 in H5FD__sec2_open(): unable to open file: name = '/Volumes/BigData/ERA5_global/rh_global_2000.nc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessibility
minor: Unable to open file
RE: operator chain - Added by Uwe Schulzweida 6 days ago
It looks as though the output file cannot be created. Do you have write access to the /Volumes/BigData directory?
RE: operator chain - Added by Ivana Cvijanovic 6 days ago
I confirm I have write access to this directory.
My impression is that CDO "thinks" that /Volumes/BigData/ERA5_global/rh_global_2000.nc is one of the files to merge. If I place .nc file of such name in the folder, than it works, but I fear it is creating nonsense.
RE: operator chain - Added by Uwe Schulzweida 6 days ago
Would it be possible to upload the two files? Although I don’t have HDF5 version 2.2.0 installed, I’d still be interested in testing it.