Chain operation and no-chain operaion got slightly different result
Added by Jialin Wang about 6 years ago
Hi, I'm using using CDO to perform a ensemble mean and a selection with some grib2 data. I found some difference of similar operations.
Here is the code I used, both of them worked:- chain operation:
cdo sellonlatbox,100,140,14,56 -ensmean '*/latlon/*_t190.grib2' chain.grib2 - no-chain operation:
cdo ensmean '*/latlon/*_t190.grib2' tmp.grib2cdo sellonlatbox,100,140,14,56 tmp.grib2 no_chain.grib2
When I use cdo diff chain.grib2 no_chain.grib2 , I got the following result, notice the Max_Absdiff, the value is temperature with units of Kelvin and I think 0.03 is not a small number:
Date Time Level Gridsize Miss Diff : S Z Max_Absdiff Max_Reldiff : Parameter ID
1 : 1985-06-08 18:00:00 2 1763 0 1763 : F F 0.033234 0.00012109 : 0.0.0
2 : 1985-06-09 00:00:00 2 1763 0 1763 : F F 0.036652 0.00013041 : 0.0.0
3 : 1985-06-09 06:00:00 2 1763 0 1763 : F F 0.038086 0.00013561 : 0.0.0
4 : 1985-06-09 12:00:00 2 1763 0 1763 : F F 0.035217 0.00012705 : 0.0.0
5 : 1985-06-09 18:00:00 2 1763 0 1361 : F F 0.031250 0.00011316 : 0.0.0
6 : 1985-06-10 00:00:00 2 1763 0 1763 : F F 0.037750 0.00013578 : 0.0.0
7 : 1985-06-10 06:00:00 2 1763 0 1763 : F F 0.034546 0.00012430 : 0.0.0
8 : 1985-06-10 12:00:00 2 1763 0 1763 : F F 0.033264 0.00011866 : 0.0.0
9 : 1985-06-10 18:00:00 2 1763 0 1763 : F F 0.036926 0.00013200 : 0.0.0
10 : 1985-06-11 00:00:00 2 1763 0 1763 : F F 0.035675 0.00012792 : 0.0.0
11 : 1985-06-11 06:00:00 2 1763 0 1763 : F F 0.046021 0.00016223 : 0.0.0
12 : 1985-06-11 12:00:00 2 1763 0 1763 : F F 0.036346 0.00013068 : 0.0.0
13 : 1985-06-11 18:00:00 2 1763 0 1763 : F F 0.034943 0.00012659 : 0.0.0
14 : 1985-06-12 00:00:00 2 1763 0 1763 : F F 0.035522 0.00012727 : 0.0.0
15 : 1985-06-12 06:00:00 2 1763 0 1763 : F F 0.036926 0.00013293 : 0.0.0
16 : 1985-06-12 12:00:00 2 1763 0 1763 : F F 0.037781 0.00013635 : 0.0.0
17 : 1985-06-12 18:00:00 2 1763 0 1763 : F F 0.033234 0.00012012 : 0.0.0
18 : 1985-06-13 00:00:00 2 1763 0 1763 : F F 0.032959 0.00011863 : 0.0.0
19 : 1985-06-13 06:00:00 2 1763 0 1763 : F F 0.034668 0.00012417 : 0.0.0
20 : 1985-06-13 12:00:00 2 1763 0 1763 : F F 0.036652 0.00013049 : 0.0.0
21 : 1985-06-13 18:00:00 2 1763 0 1763 : F F 0.034393 0.00012375 : 0.0.0
22 : 1985-06-14 00:00:00 2 1763 0 1763 : F F 0.036652 0.00013173 : 0.0.0
23 : 1985-06-14 06:00:00 2 1763 0 1763 : F F 0.039764 0.00014118 : 0.0.0
24 : 1985-06-14 12:00:00 2 1763 0 1763 : F F 0.031830 0.00011359 : 0.0.0
25 : 1985-06-14 18:00:00 2 1763 0 1763 : F F 0.032379 0.00011692 : 0.0.0
26 : 1985-06-15 00:00:00 2 1763 0 1763 : F F 0.038055 0.00013729 : 0.0.0
27 : 1985-06-15 06:00:00 2 1763 0 1324 : F F 0.031250 0.00011179 : 0.0.0
28 : 1985-06-15 12:00:00 2 1763 0 1763 : F F 0.037231 0.00013248 : 0.0.0
29 : 1985-06-15 18:00:00 2 1763 0 1763 : F F 0.023865 8.6447e-05 : 0.0.0
30 : 1985-06-16 00:00:00 2 1763 0 1763 : F F 0.031555 0.00011362 : 0.0.0
31 : 1985-06-16 06:00:00 2 1763 0 1763 : F F 0.015930 5.7535e-05 : 0.0.0
32 : 1985-06-16 12:00:00 2 1763 0 1763 : F F 0.036102 0.00012777 : 0.0.0
33 : 1985-06-16 18:00:00 2 1763 0 1763 : F F 0.032654 0.00011732 : 0.0.0
34 : 1985-06-17 00:00:00 2 1763 0 1763 : F F 0.034088 0.00012237 : 0.0.0
34 of 34 records differ
cdo diff: Processed 119884 values from 2 variables over 68 timesteps [0.21s 23MB]
Is this error caused by the bug of CDO or is there any problem in my command? Does anyone got similar problem as this?
Glad to receive any reply about this. Thanks in advance.
Here is my cdo --version
Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)
Compiled: by unknown on 63b2d8ab9116 (x86_64-unknown-linux-gnu) Feb 14 2018 17:31:19
CXX Compiler: g++ -fPIC -DPIC -g -O2 -std=c++11 -fopenmp -fPIC -DPIC -fopenmp
CXX version : g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
C Compiler: gcc -std=gnu99 -fPIC -DPIC -fopenmp
C version : gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
F77 Compiler:
F77 version : ./configure: line 21772: -V: command not found
Features: 47GB C++11 DATA PTHREADS OpenMP3 HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ.4 XML2 CURL FFTW3 SSE2
Libraries: HDF5/1.10.1 proj/4.93 xml2/2.9.7 curl/7.58.0(h7.55.1)
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.3 of Feb 14 2018 17:29:20
CGRIBEX library version : 1.9.0 of Jan 22 2018 09:24:03
GRIB_API library version : 2.7.0
NetCDF library version : 4.4.1.1 of Nov 8 2017 13:32:10 $
HDF5 library version : 1.10.1 threadsafe
EXSE library version : 1.4.0 of Feb 14 2018 17:29:14
FILE library version : 1.8.3 of Feb 14 2018 17:29:12
Replies (3)
RE: Chain operation and no-chain operaion got slightly different result - Added by Uwe Schulzweida about 6 years ago
The compression in GRIB is lossy. Therefore it is not a good idea to store intermediate calculations in GRIB format.
You can store the intermediate result to NetCDF and use the last step of you calculations to convert the data to GRIB:
cdo -f nc ensmean '*/latlon/*_t190.grib2' tmp.nc cdo -f grb2 sellonlatbox,100,140,14,56 tmp.nc no_chain.grib2
RE: Chain operation and no-chain operaion got slightly different result - Added by Jialin Wang about 6 years ago
Thanks for your reply, kind of got what's wrong here.
I run your commands and the Max_Absdiff is 0.015 now. This is quite helpful, I think I'd better use NetCDF in later operations.
By the way, is there any official document about this subject?
Thanks a lot again.
RE: Chain operation and no-chain operaion got slightly different result - Added by Jialin Wang about 6 years ago
Uwe Schulzweida wrote:
The compression in GRIB is lossy. Therefore it is not a good idea to store intermediate calculations in GRIB format.
You can store the intermediate result to NetCDF and use the last step of you calculations to convert the data to GRIB:
[...]
I switch to netCDF like this and finally cdo diff got completely same result, thanks again.cdo -f nc sellonlatbox,100,140,14,56 -ensmean '*/latlon/*_t190.grib2' chain.nc
cdo -f nc ensmean '*/latlon/*_t190.grib2' tmp.nccdo sellonlatbox,100,140,14,56 tmp.nc no_chain.nc