Project

General

Profile

Memory footprint when resampling ...

Added by Niklas Roeber about 2 months ago

Hi, why does the memory footprint of CDO increase when resampling 3D data from ICON to lon/lat? I tried a number of alternatives, directly and with precomputing weights, using remapnn and remapcon, yet the memory consumption increases with every height level. Granted, the data is big, R2B11 to a similarly sized lon/lat grid, but 1TB of main memory is not enough to do this. The data is F32, and one variable with grid is 77G. I did interpolate the height before to accommodate the correct cell height using gh2hl. Are there any suggestions to do the interpolation more memory efficient? It looks like that CDO is also using a different mask for each level:

cdo -P 60 remapnn,r2b11_ll2.txt atmo_3d_cloud5.nc atmo_3d_cloud_ll.nc
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335544318)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335544306)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335544285)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335544204)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335543921)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335542718)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335536710)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335500568)
cdo remapnn: Nearest neighbor weights from unstructured (335544320) to lonlat (32768x16384) grid, with source mask (335368939)

Is this because of the gh2hl operation and missing values at some cells?

Thanks and Cheers, Niklas


Replies (2)

RE: Memory footprint when resampling ... - Added by Uwe Schulzweida about 2 months ago

Hi Niklas,

Yes, the different mask for each level cause this memory requirement. All interpolation weights must be saved for all the different masks.
I think you can ignore the missing values if you use remapnn:

cdo -P 60 setctomiss,-9999 -remapnn,r2b11_ll2.txt -setmisstoc,-9999 atmo_3d_cloud5.nc atmo_3d_cloud_ll.nc 

Cheers,
Uwe

RE: Memory footprint when resampling ... - Added by Niklas Roeber about 2 months ago

Awesome, thanks Uwe!!!

Cheers, Niklas

    (1-2/2)