Error (ensureBufferSize)
Added by selam haftu over 3 years ago
Good day
i recently observed this error when i run a loop in CDO. the operation performs well and prints the output but the error always shows as indicated in attached photo. can i ignore the error and proceed or the error can cause damage to my data. this is the loop is used: for i in $(ls); do cdo outputtab,data,value -fldmean -sellonlatbox,7.4,11.08,9.34,12.7 ${i} > output/${i}final.csv; done
CDO version 1.9.4rc1
attached are also some of my data.
thank you for your time and effort.
Replies (5)
RE: Error (ensureBufferSize) - Added by Karin Meier-Fleischer over 3 years ago
Hi Selam,
I'm not able to reproduce the error on my mac (16GB memory). There is a typo in the command line, it has to be date and not data in outputtab.
bash-3.2$ mkdir output
bash-3.2$ for i in $(ls *.nc); do cdo outputtab,date,value -fldmean -sellonlatbox,7.4,11.08,9.34,12.7 ${i} > output/${i}final.csv; done
cdo(1) fldmean: Process started cdo(2) sellonlatbox: Process started cdo(2) sellonlatbox: Processed 1 variable over 12 timesteps. cdo(1) fldmean: Processed 300 values from 1 variable over 12 timesteps. cdo outputtab: Processed 12 values from 1 variable over 12 timesteps [0.05s 21MB]. cdo(1) fldmean: Process started cdo(2) sellonlatbox: Process started cdo(2) sellonlatbox: Processed 1 variable over 12 timesteps. cdo(1) fldmean: Processed 300 values from 1 variable over 12 timesteps. cdo outputtab: Processed 12 values from 1 variable over 12 timesteps [0.05s 20MB]. cdo(1) fldmean: Process started cdo(2) sellonlatbox: Process started cdo(2) sellonlatbox: Processed 1 variable over 12 timesteps. cdo(1) fldmean: Processed 300 values from 1 variable over 12 timesteps. cdo outputtab: Processed 12 values from 1 variable over 12 timesteps [0.05s 21MB]. cdo(1) fldmean: Process started cdo(2) sellonlatbox: Process started cdo(2) sellonlatbox: Processed 1 variable over 12 timesteps. cdo(1) fldmean: Processed 300 values from 1 variable over 12 timesteps. cdo outputtab: Processed 12 values from 1 variable over 12 timesteps [0.05s 21MB].
bash-3.2$ cat output/pr_Amon_EC-Earth3-Veg_ssp245_r1i1p1f1_gr_201501-201512.ncfinal.csv
# date value 2015-01-16 -1.51791e-25 2015-02-15 -1.561836e-25 2015-03-16 -2.137217e-25 2015-04-16 1.429571e-06 2015-05-16 1.174679e-05 2015-06-16 3.390031e-05 2015-07-16 6.754132e-05 2015-08-16 0.0001040338 2015-09-16 2.984311e-05 2015-10-16 2.529349e-06 2015-11-16 3.457773e-08 2015-12-16 -1.44307e-25
-Karin
RE: Error (ensureBufferSize) - Added by Karin Meier-Fleischer over 3 years ago
The error message is telling you that the allocation of 790 GB memory failed.
You should split the files for your loop to a appropriate subset.
RE: Error (ensureBufferSize) - Added by selam haftu over 3 years ago
my computer is windows with (12GB) memory, it still gives the error even after splitting the individual operators. can i use the bash in windows and how?
RE: Error (ensureBufferSize) - Added by Karin Meier-Fleischer over 3 years ago
The problem are not the operators but the number of files or each file size itself. Test the cdo call without the loop with a single file.
RE: Error (ensureBufferSize) - Added by selam haftu over 3 years ago
Thank you for kind responses and effort, we really appreciate.
i however have more than 1000 files to work on, that was the reason i did the loop. however, i did individual operation for some of the files and compared the output with those that showed error on allocating memory and the output are the same. i will proceed with the loop.
thank you.