CDO

Are your serious?

How can you expect a single word about performance without giving any information about

• your input data (except the number of timesteps)
• your CDO version
• YOUR HARDWARE
• your IO-subsystem speed (ramdisk, flashdisk, harddisk,...)

My answer is: buy a bigger machine.

cheers
ralf

Before you are buying a bigger machine try the following:

• if not done, configure cdo with fftw3: configure --with-fftw3 .......
• if available use more cores. Here an example with 4 cores: cdo -P 4 bandpass,... ifile ofile

These two steps should speed up the filter operators a lot, send us your feedback!

Cheers,
Uwe

Hi all.

First off, CDO is a fantastic tool so thanks for developing it.

I have experienced quite different speeds using the lowpass/highpass filters on different versions of CDO. It also seems I get different results using different versions.

As an example, with this file [1] using a command such as "cdo lowpass,0.9125 ifile.nc olfile.nc".

It takes ~10 min, 37 seconds using cdo/1.7.2 and only 6 seconds using 1) cdo/1.6. Now these are on different machines but the latter is on a desktop with minimal resources.

I have also tried this using 16,8,1 processes on this machine [4] on a bigger file (569 mb) and it ran out of time after an hour using cdo/1.7.2 but on the same machine using 1 processor it could do a 29 Gb file using cdo/1.5.4 in ~8-9 hours. So, in summary, totally different machine to the previous two but much much faster using the older version.

Now going back to my toy file [1] and comparing outputs from my first examples:
Using cdo/1.6 : cdo lowpass,0.9125 test_file_for_ollie.nc test_file_for_ollie_fast.nc
Using cdo/1.7.2 : cdo lowpass,0.9125 test_file_for_ollie.nc test_file_for_ollie_slow.nc

The above outputs are in [2]. When I do "cdo sub test_file_for_ollie_fast.nc test_file_for_ollie_slow.nc out.nc" I get 'non-zero' results, e.g. [3]. Clearly the algorithm has changed between versions...? Thus, should I be using the newer version? And any ideas on why it's so much slower?

When I tested the algorithm on smaller toy files and using cdo/1.7.2 I was fairly happy with the output (checking with power spectra).

I didn't compile/install CDO on these machines so I can't say what compile flags were used but can try and find out if that is the suspected issue.

Finally, if it turns out I should be using the new version of CDO, suggestions on how to speed up the process (I have a 48 hour walltime limit but many cores available!), perhaps spatial tiling? Would be appreciated.

Cheers,
Chris.

[1] https://cloudstor.aarnet.edu.au/plus/index.php/s/xrgr1WDn2SF8md0

[2] https://cloudstor.aarnet.edu.au/plus/index.php/s/ejAkAJGRNPtz8lI
https://cloudstor.aarnet.edu.au/plus/index.php/s/uMdhnFU6Uwt1kla

[3] https://cloudstor.aarnet.edu.au/plus/index.php/s/DMyuCLmX16gyO6O

[4] http://nci.org.au/systems-services/national-facility/peak-system/raijin/

@Chris Parap,

It was a mistake for me to post this question without the details.
Before, I am using a dual core unit (CentOS 7). I also tried to compare 1.7.2 with a lower version of cdo (e.g. 1.6.9) but the processing time did not change.
I also tried @Uwe's suggestion to reconfigure the installation with configure --with-fftw3 but the processing time is still the same.
I also compared this with gcc and intel compiler and the results are still the same.

I later used a new quad core unit and follow Uwe's suggestion and it became faster. I think the speed of the filter depends on the computer hardware (number of available processors) and size of the file that you are processing.
Maybe processing your files in chunks will reduce the processing time.

Here are my results with

% cdo -V                                                                                                                                                                                                            [Tue 2017-02-21|14:36:48]
Climate Data Operators version 1.8.0 (http://mpimet.mpg.de/cdo)
Compiled: by ram on luthien (x86_64-unknown-linux-gnu) Feb 20 2017 11:31:09
Compiler: gcc -g -O3 -std=gnu99 -Wall -fopenmp -march=native   
 version: gcc (GCC) 6.1.1 20160501
Features: DATA PTHREADS OpenMP45 HDF5 NC4/HDF5 OPeNDAP SZ UDUNITS2 PROJ.4 XML2 MAGICS CURL FFTW3 AVX2
Libraries: HDF5/1.10.0 proj/4.93 xml2/2.9.4 curl/7.52.1
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c 
     CDI library version : 1.8.0 of Feb 20 2017 11:30:49
 CGRIBEX library version : 1.7.6 of Dec 20 2016 19:55:24
GRIB_API library version : 1.19.0
  NetCDF library version : 4.4.1 of Oct 21 2016 04:58:10 $
    HDF5 library version : 1.10.0
 SERVICE library version : 1.4.0 of Feb 20 2017 11:30:46
   EXTRA library version : 1.4.0 of Feb 20 2017 11:30:45
     IEG library version : 1.4.0 of Feb 20 2017 11:30:45
    FILE library version : 1.8.2 of Feb 20 2017 11:30:45

% cdo  lowpass,0.9125 test_file_for_ollie.nc t.nc                                                                                                                                                                   [Tue 2017-02-21|14:37:19]
cdo lowpass (Warning): Filtering of multi-year times series doesn't works properly with a standard calendar.
cdo lowpass (Warning):   Please delete the day 1992-02-29 (cdo del29feb)
cdo lowpass (Warning): Filtering of multi-year times series doesn't works properly with a standard calendar.
cdo lowpass (Warning):   Please delete the day 1996-02-29 (cdo del29feb)
cdo lowpass (Warning): Filtering of multi-year times series doesn't works properly with a standard calendar.
cdo lowpass (Warning):   Please delete the day 2000-02-29 (cdo del29feb)
cdo lowpass (Warning): Filtering of multi-year times series doesn't works properly with a standard calendar.
cdo lowpass (Warning):   Please delete the day 2004-02-29 (cdo del29feb)
cdo lowpass (Warning): Filtering of multi-year times series doesn't works properly with a standard calendar.
cdo lowpass (Warning):   Please delete the day 2008-02-29 (cdo del29feb)
cdo lowpass: Processed 1227200 values from 1 variable over 61360 timesteps ( 3.37s )
<ram@luthien:~/Downloads>

using openMP does not give any benefit. CPU is

 facter processors                                                                                                                                                                                                 [Tue 2017-02-21|14:40:27]
{
  count => 8,
  isa => "unknown",
  models => [
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz",
    "Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz" 
  ],
  physicalcount => 1,
  speed => "3.80 GHz" 
}

As you can see in Ralf's cdo -V output, his CDO version was compiled with FFTW3:

Features: DATA PTHREADS OpenMP45 HDF5 NC4/HDF5 OPeNDAP SZ UDUNITS2 PROJ.4 XML2 MAGICS CURL FFTW3 AVX2

A CDO installation without FFTW3 could slow down the filtering if the number of time steps isn't a power of 2. In this case you can cut or extend your time series down/up to the nearest power of two to speed up the filtering.
The following example is using only the first 32697 time steps of the time series. This gives nearly the same performance as with FFTW3.

cdo lowpass,0.9125  -seltimestep,1/32768 test_file_for_ollie.nc result

The result is the same with and without FFTW3.

Older CDO versions < 1.6.4 do not support FFTW3. To speed up the filtering in these version the time series it filled up with zeros to the next time-step-number that is a power of 2. The result differ but the signal has a similar pattern. To avoid effects of zero-padding cut or extend your time series down/up to the nearest power of two.

I have not written the code and documentation but this feature is well documented.

Cheers,
Uwe

For the future wary traveller...

In the end, counter to Lyndon:

I also tried @Uwe's suggestion to reconfigure the installation with configure --with-fftw3 but the processing time is still the same.

I found CDO version 1.8.0 with FFTW3 produced dramatic speedups similar to what I had experienced with older versions of CDO (pre 1.6.4).

Thus, Uwe's comments are consistent with my experience.

The alluded to doc changes (and behaviour) are also shown, attached.

Thanks again for developing a great tool!

Cheers,
Chris

[1]
[raijin6] $ cdo -V
Climate Data Operators version 1.8.0 (http://mpimet.mpg.de/cdo)
Compiled: by aab900 on raijin7 (x86_64-unknown-linux-gnu) Feb 22 2017 09:26:52
Compiler: icc -std=gnu99 -g -O2 -fopenmp
version: icc (ICC) 16.0.3 20160415
Features: DATA PTHREADS OpenMP4 HDF5 NC4/HDF5 OPeNDAP FFTW3 SSE2
Libraries: HDF5/1.8.14
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c
CDI library version : 1.8.0 of Feb 22 2017 09:22:48
CGRIBEX library version : 1.7.6 of Dec 20 2016 19:55:24
GRIB_API library version : 1.12.3
NetCDF library version : 4.3.3.1 of Mar 19 2015 11:24:25 $
HDF5 library version : 1.8.14
SERVICE library version : 1.4.0 of Feb 22 2017 09:22:05
EXTRA library version : 1.4.0 of Feb 22 2017 09:21:44
IEG library version : 1.4.0 of Feb 22 2017 09:21:53
FILE library version : 1.8.2 of Feb 22 2017 09:21:45