CDO

All,

I'm wondering if you could help me out and make sure I'm using CDO in the correct way. In my use, I've tended to see CDO as a faster tool than or as fast a tool as, say, NCO and the like (cdo diffn vs. h5diff a prime example) with the benefit of some nice usability (commands/operators are nice and self-explanatory). But, I've avoided its use with compressed files as that seems to slow it down (esp. relative to NCO). But today I realized it might actually more to do with chunking or both?

To wit, using some slightly old tools, NCO 4.6.1 and CDO 1.7.2, I was doing some tests on merging some NC4 files. A model I work on has seen some good speed benefit if we write out our diagnostic output in many small files rather than a few large ones (probably due to load balancing, etc.) But, in the end, we want the files to be "large" again so that many, many existing tools and processes don't need to be rewritten. So, being lazy, rather than use netCDF directly, I turn to tools, namely nkcs --append and cdo merge. But, the timings are interesting. First, if our output is not chunked or compressed:

 ncks: 11.25 sec
  CDO:  8.79 sec

Cool, the CDO advantage I'm used to. Now, if the files are compressed by our model (deflate level 1):

 ncks: 66.27 sec
  CDO: 28.94 sec
CDO-z: 73.71 sec

The two different CDO lines there are because if you do a 'cdo merge' on compressed files, the output is written uncompressed so to get the compressed final output, you have to run with '-z zip'. Those numbers are roughly what I've seen.

But, it turns out, what I meant to do was to have our model compress and chunk our files internally and output them, but I forgot. If I use those outputs, things get interesting:

 ncks:  72.29 sec
  CDO: 150.99 sec
CDO-z: 508.39 sec

FYI, the commands used above are:

 ncks: /usr/bin/time -p sh -c "find . -iname 'f516_fp.inst3_3d_asm_Np-___*20150415*nc4' -print0 | xargs -0 -I file ncks -h -A file test_nco.nc4" 
  CDO: /usr/bin/time -p sh -c 'cdo merge f516_fp.inst3_3d_asm_Np-___*20150415*nc4 test_cdo.nc4'
CDO-z: /usr/bin/time -p sh -c 'cdo -z zip merge f516_fp.inst3_3d_asm_Np-___*20150415*nc4 test_cdo.zip.nc4'

One reason CDO appears to be much slower with chunking is that it seems to be re-chunking as well:

$ ncdump -hsc f516_fp.inst3_3d_asm_Np-___O3.20150415_0000z.nc4 | grep -e 'O3:_Storage' -A3
        O3:_Storage = "chunked" ;
        O3:_ChunkSizes = 1, 1, 91, 192 ;
        O3:_DeflateLevel = 1 ;
        O3:_Shuffle = "true" ;
$ ncdump -hsc test_nco.nc4 | grep -e 'O3:_Storage' -A3
        O3:_Storage = "chunked" ;
        O3:_ChunkSizes = 1, 1, 91, 192 ;
        O3:_DeflateLevel = 1 ;
        O3:_Shuffle = "true" ;
$ ncdump -hsc test_cdo.nc4 | grep -e 'O3:_Storage' -A3
        O3:_Storage = "chunked" ;
        O3:_ChunkSizes = 1, 11, 181, 288 ;
        O3:_Endianness = "little" ;
    float OMEGA(time, lev, lat, lon) ;
$ ncdump -hsc test_cdo.zip.nc4 | grep -e 'O3:_Storage' -A3
        O3:_Storage = "chunked" ;
        O3:_ChunkSizes = 1, 11, 181, 288 ;
        O3:_DeflateLevel = 1 ;
        O3:_Shuffle = "true" ;

So, I suppose my first question is: is there a way to have CDO preserve the chunking during a merge on compressed and chunked files? Perhaps our current chunking isn't ideal, but it is the chunk pattern we've used for this resolution, 1152x721.

Second: Is there a better way to call CDO to do these operations? That is, options to pass along with 'merge' to get performance?

Thanks,
Matt