Project

General

Profile

Forums » Support »

cdo remapbil on large (?) files

Added by martin lindahl over 4 years ago

I try to do cdo -L remapbil on a file with dimensions 43199*20879 (1.7G on disk). It works for smaller windows as a result but for result files close to the original dimensions cdo segfaults ("bus error"). By monitoring w top, I get the impression that the process just runs out of memory. I get similar results on different machines with up to 60 G memory.

Any clarification welcome!

/Martin

$ cdo -V
Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fopenmp
CXX version : g++ (Ubuntu 9.2.1-28ubuntu1) 9.2.1 20200203
C Compiler: gcc -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -pedantic -fPIC -fopenmp
C version : gcc (Ubuntu 9.2.1-28ubuntu1) 9.2.1 20200203
F77 Compiler: f77 -g -O2 -fdebug-prefix-map=/build/cdo-51aH2m/cdo-1.9.9~rc1=. -fstack-protector-strong
F77 version : unknown
Features: 62GB 12threads C++14 Fortran DATA PTHREADS OpenMP HDF5 NC4/HDF5/threadsafe OPeNDAP SZ UDUNITS2 PROJ MAGICS CURL FFTW3 SSE2
Libraries: HDF5/1.10.4 proj/6.3.0 curl/7.68.0(h7.66.0)
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.9rc1
ecCodes library version : 2.16.0
NetCDF library version : 4.7.3 of Jan 23 2020 04:18:16 $
hdf5 library version : library undefined
exse library version : 1.4.1
FILE library version : 1.8.3

$ cdo griddes XXX

#
  1. gridID 1 #
    gridtype = lonlat
    gridsize = 901951921
    xsize = 43199
    ysize = 20879
    xname = lon
    xlongname = "longitude"
    xunits = "degrees_east"
    yname = lat
    ylongname = "latitude"
    yunits = "degrees_north"
    xfirst = -179.9876388889
    xinc = 0.0083333333
    yfirst = -89.9876388889
    yinc = 0.0083333333
    scanningMode = 64
    cdo griddes: Processed 1 variable [0.01s 47MB]

Replies (7)

RE: cdo remapbil on large (?) files - Added by Ralf Mueller over 4 years ago

hi martin!

what was the exact CDO call? which source grid did you use for remapbil?

cheers
ralf

RE: cdo remapbil on large (?) files - Added by martin lindahl over 4 years ago

Hi!

Here is the requested info:

$ cdo -L remapbil,custom_grid infile.nc outfile.nc #I included the -L as I understand it could enforce thread synchronization
Killed

$ cat custom_grid

gridtype = lonlat
gridsize = 901931042
xname = lon
xlongname = Longitude
xunits = degrees_east
yname = lat
ylongname = Latitude
yunits = degrees_north
xsize = 43198
ysize = 20879
xfirst = -179.987638905
xinc = 0.00833339
yfirst = -89.98763895
yinc = 0.0083333

Regards, Martin

RE: cdo remapbil on large (?) files - Added by Ralf Mueller over 4 years ago

do you also have the grid of the infile? This is also important, esp. for memory consumption.

'-L' is only for serializing special parts of the IO - basically in case you have a non-thread-safe hdf5 installation.

I had no problems with creating data with that grid

cdo -f nc -topo,griddes tHuge.nc

RE: cdo remapbil on large (?) files - Added by martin lindahl over 4 years ago

The infile grid:

$cdo griddes infile.nc

cdo griddes infile.nc #
  1. gridID 1 #
    gridtype = lonlat
    gridsize = 901951921
    xsize = 43199
    ysize = 20879
    xname = lon
    xlongname = "longitude"
    xunits = "degrees_east"
    yname = lat
    ylongname = "latitude"
    yunits = "degrees_north"
    xfirst = -179.9876388889
    xinc = 0.0083333333
    yfirst = -89.9876388889
    yinc = 0.0083333333
    scanningMode = 64
    cdo griddes: Processed 1 variable [0.01s 51MB].

Creating a file works fine:

$ cdo -f nc -topo,custom_grid tHuge.nc

cdo topo: [7.51s 6935MB] (and the file looks ok w cdo -griddes)

/Martin

RE: cdo remapbil on large (?) files - Added by Ralf Mueller over 4 years ago

hi!

I also run out of memory doing this on a machine with 256GB memory, but with 1024GB it works.

cheers
ralf

RE: cdo remapbil on large (?) files - Added by Uwe Schulzweida over 4 years ago

You can try to run the remapping with the CDO option --no_remap_weights. In this case no weights are holt in memory. That reduces the amount of memory significantly but therefor the weights needs to be recalculated for every input field.

cdo --no_remap_weights remapbil,custom_grid infile.nc outfile.nc

RE: cdo remapbil on large (?) files - Added by martin lindahl over 4 years ago

Thank you, that resolved the issue!

    (1-7/7)