cdo sp2gpl fails
Added by Christian Zeman about 5 years ago
Hi all,
I get the following error when I try to transform a file from spectral space to grid point space:
#! cdo sp2gpl file_spectral.nc flie_gp.nc Error (sptrans_new) : Allocation of 67135078400 bytes failed. [ line 147 file specspace.cc ] System error message : Cannot allocate memory
Somehow it tries to allocate a huge amount of memory even though the file is only 26MB. Does anybody have an idea what could be the cause for this issue? Any help would be appreciated.
Thank you and kind regards,
Christian
File:¶
You can find the file attached. Below you can see the most important information.
cdo info:
#! cdo info file_spectral.nc
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 2018-05-29 00:00:00 50000 6556160 0 : -0.0016366 1.2016e-09 0.0013878 : 120.128
cdo info: Processed 6556160 values from 1 variable over 1 timestep [0.06s 93MB]
cdo griddes:
#! cdo griddes file_spectral.nc # # gridID 1 # gridtype = spectral gridsize = 6556160 truncation = 2559 complexpacking = 1 cdo griddes: Processed 1 variable [0.00s 18MB]
CDO:¶
I'm using the following version of CDO:
#! cdo --version
Climate Data Operators version 1.9.6 (http://mpimet.mpg.de/cdo)
System: x86_64-conda_cos6-linux-gnu
CXX Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1554658343410/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ -fPIC -DPIC -g -O2 -std=c++11 -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/usr/local/Miniconda3-envs/envs/2019/envs/iacpy3_2019/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix -fopenmp
CXX version : unknown
C Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1554658343410/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/usr/local/Miniconda3-envs/envs/2019/envs/iacpy3_2019/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix -fopenmp
C version : unknown
F77 Compiler: /home/conda/feedstock_root/build_artifacts/cdo_1554658343410/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/usr/local/Miniconda3-envs/envs/2019/envs/iacpy3_2019/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
F77 version : unknown
Features: 15GB 8threads C++11 Fortran DATA PTHREADS OpenMP45 HDF5 NC4/HDF5/threadsafe OPeNDAP UDUNITS2 PROJ.4 XML2 CURL FFTW3 SSE3
Libraries: HDF5/1.10.4 proj/5.2 xml2/2.9.8 curl/7.64.0(h7.64.1)
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.6
cgribex library version : 1.9.2
ecCodes library version : 2.12.3
NetCDF library version : 4.6.2 of Dec 17 2018 19:36:03 $
hdf5 library version : 1.10.4 threadsafe
exse library version : 1.4.1
FILE library version : 1.8.3
But I get the same error with an older version of cdo (1.9.5) on a different computer.
| file_spectral.nc (25 MB) file_spectral.nc |
Replies (7)
RE: cdo sp2gpl fails - Added by Uwe Schulzweida about 5 years ago
CDO precalculates the legendre polynomials to speed up the spectral transformation. To store this legendre polynomials two fields with the following size are needed:
(ntr + 1) * (ntr + 2) * nlat / 2 with ntr = 2559 and nlat = 2560: 2560 * 2561 * 2560 / 2 = 8391884800 values = 67135078400 bytes
RE: cdo sp2gpl fails - Added by Christian Zeman almost 5 years ago
Thank you for the response. Is there a way to suppress this precalculation? Speed is not an issue for me, but 67GB will not fit into shared memory on any node I have access to.
RE: cdo sp2gpl fails - Added by Ralf Mueller almost 5 years ago
in case u have an account at CSCS, there are nodes with 120GM ram (https://user.cscs.ch/access/running/#synoptic-table)
RE: cdo sp2gpl fails - Added by Uwe Schulzweida almost 5 years ago
There is not way to suppress this precalculation, sorry.
Unfortunately, we found that the spectral transformation does not work properly with more than 4 million spectral coefficients. We will disable this transformation in the next CDO release, if there are more than 4 million spectral coefficients, since we can not solve this problem.
RE: cdo sp2gpl fails - Added by Christian Zeman almost 5 years ago
Ok. Just to make sure that I understood it correctly:
With spectral coefficients you mean the time-dependent coefficients of which I have (ntr+1)*(ntr+2) values and in my case it's over 6 million, so I shouldn't use CDO anyway, regardless whether the Legendre polynomials fit in memory or not?
RE: cdo sp2gpl fails - Added by Christian Zeman almost 5 years ago
Ok, thank you for your help.
Kind regards, Christian