difference between python implementation and pure cdo
Added by Mario Mech over 2 years ago
Hej all,
I have a short question on numerical differences when extracting ERA5 data with cdo.
Using the "plain" cdo operator like:
cdo -f nc -P 32 -seltimestep,13 -selday,1 -sellonlatbox,-30,50,65,89 -setgridtype,regular /pool/data/ERA5/sf00_1H/2019/E5sf00_1H_2019-04_034 /tmp/reduced_sf_20190401_1200_034.nc
and the the cdo python module like:
from cdo import *
cdo = Cdo()
cdo.sellonlatbox('-30,50,65,89',input='-seltimestep,13 -selday,01 -setgridtype,regular /pool/data/ERA5/sf00_1H/2019/E5sf00_1H_2019-04_034', output='/tmp/reduced_sf_20190401_1200_034_py.nc', options='-f nc -P 32')
results in slightly different output for the extracted SST (or other variables) in the order of 3*10^-5 for some grid points (see attached figure).
I'm not really concerned about it. It is just more out of curiosity whether it is the order of the commands or anything else I should do differently.
All best
Mario
| Figure_1.png (14.4 KB) Figure_1.png | difference between both sst outputs |
Replies (4)
RE: difference between python implementation and pure cdo - Added by Karin Meier-Fleischer over 2 years ago
Hi Mario,
I'm sorry but I can't reproduce the problem on levante. I used jupyterhub with the Python3 kernel.
from cdo import *
cdo = Cdo()
#-- the following has to run in a single cell
!cdo -f nc -P 32 -seltimestep,13 -selday,1 -sellonlatbox,-30,50,65,89 -setgridtype,regular \
/pool/data/ERA5/sf00_1H/2019/E5sf00_1H_2019-04_034 \
/tmp/reduced_sf_20190401_1200_034_kmf.nc
#-- next cell
cdo.sellonlatbox('-30,50,65,89',
input='-seltimestep,13 -selday,01 -setgridtype,regular /pool/data/ERA5/sf00_1H/2019/E5sf00_1H_2019-04_034',
output='/tmp/reduced_sf_20190401_1200_034_py_kmf.nc',
options='-f nc -P 32')
cdo.diff(input='/tmp/reduced_sf_20190401_1200_034_kmf.nc /tmp/reduced_sf_20190401_1200_034_py_kmf.nc')
The diff command result is an empty list so the data sets are equal.
RE: difference between python implementation and pure cdo - Added by Mario Mech over 2 years ago
Hi Karin,
this is interesting. Im as well on levante python kernel 3 and for me it does show differences:
from cdo import *
cdo = Cdo()
cdo.diff(input='/scratch/b/b380702/reduced_sf_20190401_12_034.nc /scratch/b/b380702/reduced_sf_20190401_12_034_py.nc')
[69]:
['Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter ID',
'1 : 2019-04-01 12:00:00 0 24424 3557 3 : F F 3.0518e-05 1.1112e-07 : 34.128',
'1 of 1 records differ',
'0 of 1 records differ more than 0.001']
RE: difference between python implementation and pure cdo - Added by Karin Meier-Fleischer over 2 years ago
Maybe you have loaded another cdo module under the hood. My python-cdo is version 1.5.4 from Python 3 kernel and the cdo versions under the hood is 2.0.3.
print(cdo.__version__()) print(cdo.version())
RE: difference between python implementation and pure cdo - Added by Mario Mech over 2 years ago
Ok, here we go. That might be the reason.
Since the differences are small, I don't or stop comparing ;-)
Thank you very much for you help.
from cdo import *
cdo = Cdo()
cdo.diff(input='/scratch/b/b380702/reduced_sf_20190401_12_034.nc /scratch/b/b380702/reduced_sf_20190401_12_034_new.nc')
print(cdo.__version__())
print(cdo.version())
1.5.4
2.0.3
[b380702@l20060 ~]$ module load cdo
[b380702@l20060 ~]$ cdo -V
Climate Data Operators version 2.0.5 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/g++ -std=gnu++14 -g -O2 -fopenmp -pthread
CXX version : g++ (Spack GCC) 11.2.0
C Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (Spack GCC) 11.2.0
F77 Compiler: /sw/spack-workplace/spack/lib/spack/env/gcc/gfortran -g -O2
F77 version : GNU Fortran (Spack GCC) 11.2.0
Features: 251GB 256threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5/threadsafe OPeNDAP SZ UDUNITS2 PROJ XML2 MAGICS CURL FFTW3 AVX2
Libraries: HDF5/1.12.1 proj/8.1.0 xml2/2.9.7 curl/7.61.1 magics/4.9.3
CDI data types: SizeType=size_t DateType=int64_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 2.0.5
cgribex library version : 2.0.1
ecCodes library version : 2.21.0
NetCDF library version : 4.8.1 of Feb 17 2022 15:36:58 $
hdf5 library version : 1.12.1 threadsafe
exse library version : 1.4.2
FILE library version : 1.9.1