CDO

Hello,

I am relatively new to CDO and NCO and have learned a lot from these forums.

I am currently working with netCDF4 files that contain five years of daily data and am using cdo timsum to calculate the number of days above a specified temperature threshold.

When I tested the timsum command (cdo -f nc -timsum -gec,105 infile.nc outfile.nc) on a netcdf4 file containing just one year of data it took about 5 minutes. When I tested the same command on the file containing five years of data, it took a full hour. Thinking that maybe I should break the five year data into one year chunks, I tried two things:

1. Use cdo splityear to create 5 one year files (this took about an hour)
2. Use ncks to hyperslab along the time dimension (this took on the order of one minute)

My best guess is that the difference in processing time between the cdo splityear and ncks is that the output format is different. When I do an ncdump -k on the different output files, cdo splityear is generating 64-bit offset files while ncks is generating netCDF4.

This has major time implications for the next step of using cdo timsum. When running timsum on the cdo splityear output (64-bit offset), the time is trivial (~ 3 seconds for each one year file). When running the same command on the ncks output (netCDF4), it takes 12 minutes.

In other words, if I use the fast option (ncks) for breaking into one year chunks, cdo timsum is very slow in processing the output. If, on the other hand, I use the slow method to split the years (cdo splityear), cdo timsum is fast in processing the output.

Why the time difference in these different processes? And is there any way to avoid having one of these steps be so slow?

For reference, here is the output of cdo -V:
Climate Data Operators version 1.9.2 (http://mpimet.mpg.de/cdo)
Compiled: by kristinadahl on kristinas-imac.lan (x86_64-apple-darwin14.5.0) Nov 19 2018 11:57:11
CXX Compiler: clang++ -std=gnu++11 -g -O2 -D_THREAD_SAFE -pthread
CXX version : Apple LLVM version 7.0.2 (clang-700.1.81)
C Compiler: clang -g -O2 -D_THREAD_SAFE -pthread
C version : Apple LLVM version 7.0.2 (clang-700.1.81)
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Homebrew GCC 8.2.0) 8.2.0
Features: 32GB Fortran DATA PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ AVX2
Libraries: HDF5/1.10.4
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.2 of Nov 19 2018 11:56:47
CGRIBEX library version : 1.9.0 of Sep 29 2017 10:16:02
GRIB_API library version : 1.27.0
NetCDF library version : 4.6.1 of Nov 19 2018 11:53:52 $
HDF5 library version : 1.10.4
SERVICE library version : 1.4.0 of Nov 19 2018 11:56:44
EXTRA library version : 1.4.0 of Nov 19 2018 11:56:43
IEG library version : 1.4.0 of Nov 19 2018 11:56:44
FILE library version : 1.8.3 of Nov 19 2018 11:56:43

I am running on a Mac OSX.

Thanks in advance for any insights!