Large memory usage in CDO collgrid command
I have 78 netcdf files each around 17MB, with shape (time=1, h=2048, w=2048) to be merged spatially. The single timestep is shared for all 78 files. The collgrid merge command below was able to produce the output netcdf of size 1.3GB, but the memory usage during the merge was 5GB.
INPUT_DIR="/home/muye/Merge-tiles/nctiles"
OUTPUT_FILE="/home/muye/Merge-tiles/merged.nc"
mkdir -p "$(dirname "$OUTPUT_FILE")"
/usr/bin/time -v cdo --single -r collgrid "$INPUT_DIR/Crop17_??_??.nc" "$OUTPUT_FILE"
Logs from the command: cdo collgrid: Processed 327155712 values from 78 variables over 78 timesteps [12.87s 5046MB].
An individual netcdf has the following structure:
<xarray.Dataset> Size: 17MB
Dimensions: (x: 2048, y: 2048, time: 1)
Coordinates:
* x 
float64 16kB 143.3 143.3 143.3 143.3 ... 143.5 143.5 143.5
* y (y) float64 16kB -34.8 -34.8 -34.8 -34.8 ... -35.0 -35.0 -35.0
* time (time) datetime64[ns] 8B 2017-01-01
Data variables:
var0 (time, y, x) float32 17MB ...
I wonder whether the 5GB memory usage was making sense (really required when the output grid is created, or something not efficient), and if it is possible to reduce the memory usage in this operation. Thanks in advance for any idea!
Replies (8)
RE: Large memory usage in CDO collgrid command - Added by Dale Chen 3 days ago
Uwe Schulzweida wrote in :
By regular I mean normal lon/lat grids, like your data.
I see. So my netcdfs do have the potential to reduce memory consumption in collgrid by a factor of 2, if I avoid storing all grid cell indices. Can I achieve that by deleting e.g. all lon indices in each netcdf?
A brief example would be greatly appreciated 
RE: Large memory usage in CDO collgrid command - Added by Dale Chen 3 days ago
Dale Chen wrote in :
Uwe Schulzweida wrote in :
By regular I mean normal lon/lat grids, like your data.
I see. So my netcdfs do have the potential to reduce memory consumption in
collgridby a factor of 2, if I avoid storing all grid cell indices. Can I achieve that by deleting e.g. all lon indices in each netcdf?A brief example would be greatly appreciated
I actually tried ds = ds.drop_vars('x') for each input netcdf before collgrid. But the memory consumption is still 5GB during the operation. I wonder if operator collgrid or another could help achieve 'not storing all grid cell indices'?