CDO

I'm running a simple GCM on the cheyenne HPC compute cluster, and have been trying to process the results in parallel as the model runs in blocks of 100 days. On 40 cores, the output is 160 lats by 320 lons, 60 levels, and 200 timesteps (the model runs in 100 day blocks, saves every 12 hours).

For some reason, it appears that even with a sufficient number of cores, the more parallel CDO processes, the slower each process runs. Using other tools like NCO and NCL, more parallel processes were no slower than running a single process. Note that I ensure my code runs on at least as many cores as the number of background processes. Does anyone have any idea how this could have come about?

Here's the script I wrote to test this (results in the comments at the bottom) -- it runs -zonmean on the NetCDF files (the model was run on 40 cores; produced 40 files in parallel), first processing 1 of these files, then 2 files concurrently, then 4, 10, 20, and 40. I ran this for 40 400MB model files, but have provided a smaller 60MB you can test with. May be hard to reproduce if you don't have HPC access.

#!/usr/bin/env bash
# Test the strange bottleneck issue observed with concurrent CDO processes
shopt -s nullglob
dir=~/scratch/test10x4/d0000-d0100 # any will do, just make sure it has contents
cd $dir

# Get file(s)
i=0
np=0
nprocs=(1 2 4 10 20 40)
file=sample.nc # simple sample file
# files=(*_interp.????.nc) # can't use glob for number because tmp files show up
# nf=${#files[@]}
# [ $nf -eq 0 ] && echo "Error: No files found." && exit 1
echo Files: ${files[@]}

# Get zonal mean using increasingly more concurrent processes; since 40 cores
# are available, for most programs, this should be fine
# while [ $np -le $nf ] && [ $i -lt ${#nprocs[@]} ]; do
while [ $i -lt ${#nprocs[@]} ]; do
  # First CDO
  np=${nprocs[i]}
  echo $np concurrent processes
  # Option A) Loop over model files
  # for file in ${files[@]::np}; do
  #   time cdo -s -O -zonmean $file ${file%.nc}_tmp.nc 2>/dev/null &
  # done
  # Option B) Read from single file "sample.nc" 
  for j in $(seq 1 $np); do
    time cdo -s -O -zonmean $file ${file%.nc}_${j}.nc 2>/dev/null &
  done
  wait # will get all the error messages
  echo

  # Then compare to NCO
  echo $np concurrent NCO processes
  # Option A) Loop over model files
  # for file in ${files[@]::np}; do
  #   time cdo -s -O -zonmean $file ${file%.nc}_tmp.nc 2>/dev/null &
  #   time ncwa -a lon $file ${file%.nc}_$tmp.nc &
  # done
  # Option B) Read from single file "sample.nc" 
  for j in $(seq 1 $np); do
    time ncwa -O -a lon $file ${file%.nc}_${j}.nc &
  done
  wait
  echo

  # Next
  let i+=1
done

# Times for 400MB files on *40* cores
# | N   | NCO walltime | CDO walltime
# | --- | ---          | ---                  |
# | 1   | 3s           | 15s                  |
# | 2   | 3s           | 16s                  |
# | 4   | 3s           | 20s                  |
# | 10  | 4s           | 25-35s               |
# | 20  | 4-5s         | 45-55s               |
# | 40  | 6-7s         | 100-200s, most >150s |
#
# Times for 400MB files on *4* cores -- *no change*
# | N   | NCO walltime | CDO walltime |
# | --- | ---          | ---          |
# | 1   | 3s           | 15s          |
# | 2   | 3s           | 17s          |
# | 4   | 3s           | 20s          |

I am using CDO version 1.9.5, compiled with pthread support and MPI support (which as I understand is only for parallel computations for some particular commands?). It was downloaded with *anaconda, but I had the same issues using the version 1.9.4 provided by the supercomputer. See below.

$ cdo -V
Climate Data Operators version 1.9.5 (http://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -fPIC -DPIC -g -O2 -std=c++11 -fopenmp -fPIC -DPIC  -m64 -fPIC -fopenmp
CXX version : g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
C Compiler: gcc -std=gnu99 -fPIC -DPIC  -m64 -fPIC -fopenmp
C version : gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Features: 252GB 72threads C++11 Fortran DATA PTHREADS OpenMP3 HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ.4 XML2 CURL FFTW3 SSE2
Libraries: HDF5/1.10.3 proj/4.93 xml2/2.9.8 curl/7.62.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
     CDI library version : 1.9.5
 CGRIBEX library version : 1.9.1
GRIB_API library version : 2.9.2
  NetCDF library version : 4.6.1 of Oct 20 2018 02:37:36 $
    HDF5 library version : 1.10.3 threadsafe
    EXSE library version : 1.4.0
    FILE library version : 1.8.3