Project

General

Profile

Forums » Open discussion »

cdo eof issue

Added by Christ Reflin over 3 years ago

Dear everyone,

I have been using cdo to perform eof analysis on different domain and I was able to obtain the result as I expected.
However, some of data (ex: anom_1.nc and anom_2.nc in zip file attached ) could not produce the result. It took so long to finish (>4 hours running). Comparing with the other data, it was just 30 minutes running.

Once it finish, warning message appears 

statistics-module (Warning): Eigenvalue computation with one-sided jacobi scheme
did not converge properly.Eigenvalue computation with one-sided jacobi scheme
did not converge properly. xxxxx  of xxxxxx pairs of columns did
not achieve requested orthogonality of 1e-12

As you can see, Not only I have tried to change different environment variable but also I have used the default

export CDO_SVD_MODE=jacobi
export CDO_WEIGHT_MODE=off
export MAX_JACOBI_ITER=100
NEOF=5

 $ cdo -v eof,$NEOF anom_1.nc eigen_value.nc eigen_vector.nc
 OpenMP:  num_procs=4  max_threads=1  num_devices=0
cdo eof: Using CDO_SVD_MODE 'jacobi' from Environment
cdo eof: Using CDO_WEIGHT_MODE 'off' from Environment
cdo eof: Counting timesteps in ifile
cdo eof: Found 11323 timeSteps
cdo eof: Calculating 5 eigenvectors and 2070 eigenvalues in grid_space
cdo eof: Allocated eigenvalue/eigenvector structures with nts=11323 gridsize=2070
cdo eof: Calculating covar matrices for 1 levels of var0 (Precipitation_Flux)
cdo eof: processing level 0
cdo eof: Using MAX_JACOBI_ITER 100 from Environment
cdo eof: Using FNORM_PRECISION 1e-12 from default

Here is my cdo version:

$ cdo -V
Climate Data Operators version 1.9.4rc1 (http://mpimet.mpg.de/cdo)
System: x86_64-unknown-cygwin
CXX Compiler: g++ -g -O2 -fopenmp
CXX version : g++ (GCC) 6.4.0
C Compiler: gcc -g -Wall -O2 -DPIC -pie -mwindows -fopenmp
C version : gcc (GCC) 6.4.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (GCC) 6.4.0
Features: 7GB C++14 Fortran DATA PTHREADS OpenMP45 HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ.4 SSE2
Libraries: HDF5/1.8.17 proj/5
Filetypes: srv ext ieg grb1 grb2 nc1 nc4 nc4c nc5
     CDI library version : 1.9.3 of Mar 13 2018 15:54:29
 CGRIBEX library version : 1.9.0 of Jan 22 2018 09:24:03
GRIB_API library version : 1.16.0
  NetCDF library version : 4.4.0 of Jun 21 2016 15:28:18 $
    HDF5 library version : 1.8.17 threadsafe
    EXSE library version : 1.4.0 of Mar 13 2018 15:54:15
    FILE library version : 1.8.3 of Mar 13 2018 15:54:45

To give you the background of the data,

anom_1: the actual data does not contain missing value
anom_2: the actual data contains missing value. Undocumented CDO function "fillmiss" fill the missing value.

Could you help me to find out where my mistake is?
does the issue come from the data itself or my script or my cdo version?

Since I have been dealing with this for the last couple days, I could not understand what to do next. Please help.

Thank you
Christian

anom_1 and anom_2.zip (29.1 MB) anom_1 and anom_2.zip

Replies (2)

RE: cdo eof issue - Added by Ralf Mueller about 3 years ago

hi!

thx for the precise input. I will run a comparison on my laptop with your data. At a first speedup option, you can use OpenMP parallelization by adding the '-P <numThreads>' option. your data is somewhat special because it uses a relatively large number of timesteps. Timesteps are handled differently by CDO compared to horizontal grids because of the operators chaining feature of CDO. that's why some inputs lead to comparably long processing times. An extreme example would be a timeseries for a single grid point with 1M values. The reading of the data takes a lot longer than XArray.

RE: cdo eof issue - Added by Ralf Mueller about 3 years ago

I ran

cdo -P 4 -v eof,$NEOF anom_1.nc eigen_value.nc eigen_vector.nc
with your variable settings from above
cdo -P 4 -v eof,$NEOF anom_1.nc eigen_value.nc eigen_vector.nc
 OpenMP:  num_procs=8  max_threads=4
cdo    eof: Using CDO_SVD_MODE 'jacobi' from Environment
cdo    eof: Using CDO_WEIGHT_MODE 'off' from Environment
cdo    eof: Counting timesteps in ifile
cdo    eof: Found 11323 timeSteps
cdo    eof: Calculating 5 eigenvectors and 2451 eigenvalues in grid_space
cdo    eof: Allocated eigenvalue/eigenvector structures with nts=11323 gridsize=2451
cdo    eof: Calculating covar matrices for 1 levels of var0 (Precipitation_Flux)
cdo    eof: processing level 0
cdo    eof: Using MAX_JACOBI_ITER 100 from Environment
cdo    eof: Using FNORM_PRECISION 1e-12 from default
cdo    eof: Finished one-sided jacobi scheme for eigenvalue computation after 100 iterations
statistics-module (Warning): Eigenvalue computation with one-sided jacobi scheme
                             did not converge properly. 3002475 of 3002475 pairs of columns did
                             not achieve requested orthogonality of      1e-12
cdo    eof (Warning): Setting Matrix and Eigenvalues to 0 before return
cdo    eof: Processed 27752673 values from 1 variable over 11323 timesteps [1154.81s 160MB].

done after roughly 20mins

    (1-2/2)