CDO

Fixed, formatting:

Hello,
I'm trying to use cdo bandpass filter on files with a lot of timesteps on juwels@fz.juelich.de. It was very slow and I found a topic (https://code.mpimet.mpg.de/boards/2/topics/4413) describing that cdo should be installed with fftw3 support to speed up this command. I followed the instructions at http://www.studytrails.com/blog/install-climate-data-operator-cdo-with-netcdf-grib2-and-hdf5-support/ and installed zlib, hdf5, netcdf, jasper, grib and also fftw3 of course. All the libraries were build successfully. I then tried to install cdo as follows:

./configure --prefix=/p/project/chhb19/jstreffi/software/cdo/install CFLAGS=-fPIC --with-fftw3=/p/project/chhb19/jstreffi/software/cdo/install --with-netcdf=/p/project/chhb19/jstreffi/software/cdo/install --with-jasper=/p/project/chhb19/jstreffi/software/cdo/install --with-hdf5=/p/project/chhb19/jstreffi/software/cdo/install --with-grib_api=/p/project/chhb19/jstreffi/software/cdo/install

Every single make check test fails with the same error message:
@ ../src/cdo: symbol lookup error: ../src/cdo: undefined symbol: __libm_exp_table_128

> Currently Loaded Modules:
>   1) Stages/2019a                (S)    12) zlib/.1.2.11           (H)  23) freetype/.2.10.0   (H)  34) libGLU/.9.0.0        (H)  45) libreadline/.8.0       (H)  56) ANTLR/.2.7.7-Python-2.7.16 (H)
>   2) GCCcore/.8.3.0              (H)    13) Szip/.2.1.1            (H)  24) gperf/.3.1         (H)  35) freeglut/.3.0.0      (H)  46) Tcl/8.6.9                   57) libdap/.3.20.3             (H)
>   3) binutils/.2.32              (H)    14) cURL/7.64.1                 25) ncurses/.6.1       (H)  36) NASM/.2.14.02        (H)  47) SQLite/.3.27.2         (H)  58) NCO/4.7.9
>   4) StdEnv                      (H)    15) netCDF/4.6.3                26) util-linux/.2.33.1 (H)  37) libjpeg-turbo/.2.0.2 (H)  48) Tk/.8.6.9              (H)  59) YAXT/0.6.0
>   5) jsctools/.0.1               (H,S)  16) netCDF-Fortran/4.4.5        27) fontconfig/.2.13.1 (H)  38) JasPer/.2.0.16       (H)  49) libxslt/.1.1.33        (H)  60) CDO/1.9.5
>   6) .juwels-env                 (H)    17) parallel-netcdf/1.11.0      28) X11/20190311            39) ecCodes/2.12.0            50) libffi/.3.2.1          (H)  61) libxml2/.2.9.9             (H)
>   7) icc/.2019.3.199-GCC-8.3.0   (H)    18) GMP/6.1.2                   29) libdrm/.2.4.97     (H)  40) HDF5/1.10.5               51) libyaml/.0.2.2         (H)  62) Perl/5.28.1
>   8) ifort/.2019.3.199-GCC-8.3.0 (H)    19) nettle/.3.4            (H)  30) LLVM/8.0.0              41) CMake/3.14.0              52) PostgreSQL/11.2             63) Doxygen/1.8.15
>   9) Intel/2019.3.199-GCC-8.3.0         20) bzip2/.1.0.6           (H)  31) XZ/.5.2.4          (H)  42) UDUNITS/.2.2.26      (H)  53) protobuf/.3.7.1        (H)
>  10) IntelMPI/2018.5.288                21) expat/.2.2.6           (H)  32) libunwind/.1.3.1   (H)  43) GSL/2.5                   54) gflags/.2.2.2          (H)
>  11) imkl/2019.3.199                    22) libpng/.1.6.36         (H)  33) Mesa/19.0.1             44) Java/1.8                  55) libspatialindex/.1.9.0 (H)
> 
>   Where:
>    S:  Module is Sticky, requires --force to unload or purge
>    H:             Hidden Module

Dr. Google did not provide much with much to go off of for this error. Do you know what happened and how I can fix this? Full logfile in attachment.

Thank you for your support, best regards,
Jan Streffing

hi Jan!

I would do the following

1. check if you can use conda: the conda-forge package for cdo is build with fftw3 support
2. ask the admins to install cdo-1.9.8 as a module (meanwhile)
3. ask why the existing CDO module (CDO/1.9.5) is not compiled with fftw3

This should provide you with a reasonable workaround as long as the admins fix the real problem.

don't get me wrong, but debugging errors like this in an environment full of modules (which possibly change dynamic linking behavior) usually doen't end well. For a basic analysis, I would remove all modules with

module purge

and run ldd on the cdo binary to check, what's missing.

hth
ralf

Hi Ralf,
Thank you for the conda tip. 5 minutes of work and it functions. Much better than installing everything with make, messing it up and waiting for the admins

/p/project/chhb19/jstreffi/software/miniconda3/envs/pyn_env_py2/bin/cdo --verbose timmean -bandpass,182,730 ../00001/6h_VO_00001.nc VO_bpf
 OpenMP:  num_procs=78  max_threads=1  num_devices=0
cdo(2) bandpass: Process started
cdo(2) bandpass: Using fftw3 lib
cdo(2) bandpass: Time step 6 hour
cdo(2) bandpass: fmin=182  fmax=730
cdo timmean:  2001-05-31 18:00:00  vfrac = 1, nsets = 1460
cdo(2) bandpass: Processed 1420288000 values from 1 variable over 1460 timesteps
cdo timmean: Processed 1420288000 values from 1 variable [167.45s 10GB]

I just tested this cdo version with fftw3 and is took about 167s to process 1 year of 6hourly 19x320x160 data. That's quite usable speed for me.

Cheers, Jan

hi Jan!

thx for the feedback + happy hacking ;-)
-ralf