ydrunpctl memory issues
Added by Ed Maurer about 13 years ago
I have been running into a memory allocation issue when using the ydrunpctl function. the command I run is this:
cdo ydrunpctl,90,5 junk_1961-1990.all_tx.SE.nc junk_ydrunmin_tn.SE.nc junk_ydrunmax_tx.SE.nc junk_ydrunpctl90_SE.nc
which returns the following:
Error (hsetCreateVarLevels) : Allocation of 404 bytes failed. [ line 227 file percentiles.c ]
System error message : Cannot allocate memory
The 3 input files can be found at http://hydro.engr.scu.edu/files/junk/
I know the input files are large, but I have successfully used similar functions (ydrunmean, ydrunmin/max) on files 4x as large on my system (6GB RAM). I keep trying smaller and smaller versions of these files, but receive the same error message. I'd appreciate any clues as to what might solve this.
Thanks,
Ed
Replies (1)
RE: ydrunpctl memory issues - Added by Uwe Schulzweida about 13 years ago
Hi Ed,
The percentile algorithm uses histograms. The default number of histogram bins is 101. Therefor the percentile function needs ~100x more memory than the other statistic functions. With your example data 10GB of memory is needed. You can reduce the amount of memory by reducing the number of histogram bins with the environment variable CDO_PCTL_NBINS. But this will also reduce the accuracy of the percentile. CDO_PCTL_NBINS=51 (6GB memory) is a good choice for your data set.
Best regards,
Uwe