CDO

Hi Suman,

missing values are not fully supported with eofs.

From CDO documentation chapter 2.11.1 EOFS - Empirical Orthogonal Functions:

Missing values are not fully supported. Support is only checked for 
non-changing masks of missing values in time. Although there still 
will be results, they are not trustworthy, and a warning will occur. 
In the latter case we suggest to replace missing values by 0 in infile.

-Karin

Dear Karin
Thank you very much for your reply.I agree with you. I have seen that using cdo 1.5.9 it is producing results (not sure correct or not). But using 1.8.2 on the same dataset it is not running and throwing the error message. Please provide some way out to get rid of the results. I mean is there any secondary way to find eofs for data with missing values? Please reply. Actually I am very much dependent on cdo and don't want to use other tools.
Any kind of help will be highly appreciated.

With Regards
Suman

Dear Karin
I have another query. As you have suggested that to replace missing values by 0, whether the results will be trustworthy after that? I am doing eof analysis only on land data points although my domain contains oceans too (having filled with missing values). Whether the results will be correct? Because otherwise all the efforts will go to vain.
Please reply.

With Regards
Suman

Which version are you using? Please, send the output of

cdo -V

Can you upload the data file and which cdo command you are using (cdo -v eof,3 rainfall.nc eigen.nc eof.nc)?

-Karin

Dear Karin
I am using cdo 1.8.2. The output of cdo -V is as follows:

Climate Data Operators version 1.8.2 (http://mpimet.mpg.de/cdo)
Compiled: by suman on localhost.localdomain (i686-pc-linux-gnu) Jun 7 2017 10:37:18
Compiler: gcc -std=gnu99 -g -O2 -fopenmp
version: gcc (GCC) 4.5.1 20100924 (Red Hat 4.5.1-4)
Features: DATA PTHREADS OpenMP3
Libraries:
Filetypes: srv ext ieg grb1 nc1 nc2
CDI library version : 1.8.2 of Jun 7 2017 10:36:55
CGRIBEX library version : 1.7.6 of Dec 20 2016 19:55:24
NetCDF library version : 4.1.1 of Nov 25 2012 07:40:13 $
SERVICE library version : 1.4.0 of Jun 7 2017 10:36:51
EXTRA library version : 1.4.0 of Jun 7 2017 10:36:47
IEG library version : 1.4.0 of Jun 7 2017 10:36:50
FILE library version : 1.8.3 of Jun 7 2017 10:36:48

I am getting error message like this:
*[suman@localhost temp]$ /home/suman/Cdo-1.8.2/bin/cdo -v eof,3 rainfall.nc eig.nc eof.nc
OpenMP: num_procs = 4 max_threads = 1
cdo eof: Using CDO_SVD_MODE 'jacobi' from default
cdo eof: Using CDO_WEIGHT_MODE 'on' from default
cdo eof: Total area = 0.302291 steradians
cdo eof: Total area = 0.302291
cdo eof: Counting timesteps in ifile
cdo eof: Found 32 timeSteps
cdo eof: Calculating 3 eigenvectors and 32 eigenvalues in time_space
cdo eof: Allocated eigenvalue/eigenvector structures with nts=32 gridsize=17415

cdo eof (Abort): Missing values unsupported!*

Please help me

With Regards
Suman

To set the missing values to 0 you can pipe the commands

cdo -v -eof,3 -setmisstoc,0 rainfall.nc eig.nc eof.nc

Output:

cdo eof: Using CDO_SVD_MODE 'jacobi' from  default
cdo eof: Using CDO_WEIGHT_MODE 'on' from  default
cdo eof: Started child process "setmisstoc,0 rainfall.nc (pipe1.1)".
cdo eof: Total area = 0.302291 steradians
cdo eof: Total area = 0.302291
cdo eof: Counting timesteps in ifile
cdo eof: Found 32 timeSteps
cdo(2) setmisstoc: Processed 557280 values from 1 variable over 32 timesteps ( 0.01s )
cdo eof: Started child process "setmisstoc,0 rainfall.nc (pipe1.2)".
cdo eof: Calculating 3 eigenvectors and 32 eigenvalues in time_space
cdo eof: Allocated eigenvalue/eigenvector structures with nts=32 gridsize=17415
cdo eof: Calculating covar matrices for 1 levels of var0 (rf)
cdo eof: processing level 0
cdo eof: allocating covar with 32 x 32 elements | npack=17415
cdo eof: finished calculation of covar-matrix for var rf
cdo eof: Using MAX_JACOBI_ITER 12 from default
cdo eof: Using FNORM_PRECISION 1e-12 from default
cdo eof: Finished one-sided jacobi scheme for eigenvalue computation after 9 iterations
cdo(3) setmisstoc: Processed 557280 values from 1 variable over 32 timesteps ( 0.03s )
cdo eof: Processed 557280 values from 2 variables over 32 timesteps ( 0.04s )

But I don't know if it is trustworthy.

-Karin

Dear forum,

I would like to revamp this thread and bring up to the attention of the CDO team one aspect about setting missing values to 0, prior EOF analysis.

I understand the philosophy behind the CDO documentation, that is to keep the scientific advices at the bare minimum and to let the user discover and make his own savvy choices. On the other hand, the suggestion "-setmisstoc,0" could be misleading for the unexperienced.

EOF si an eigen technique and a matrix with NaN is not clearly the best choice. But filling NaN with zeroes will impact the numerical decomposition, because those 0 are assumed to be valid matrix entries.
One way to tackle the problem is to preprocess the time series and interpolate with a function of choice (e.g. splines) or to fill up the matrix with other algorithms (e.g. expectation/maximization algorithm). The problem here is that one has to a-posterior "reconstruct" missing entries in the matrix, so that the SVD is still faithful but not biased by the zeroes.

I have read all threads in the forum about EOF and NaN support and it seems to me that this detail has not received the deserved attention.