Cdo in Python too many files error
Added by Michael Debertshaeuser over 3 years ago
Hey,
I try to get a ensmean out of many input files.
input files = hourly_all_ml_f_zero.txt
I use for the terminal:
MYFILES=`cat /home/mdeberts/masterthesis/preprocessing/data/hourly_all_ml_f_zero.txt | cut -f2 -d" "`
Then
ulimit -s 15000000000000000
ulimit -n 15000
to increase the maximum of files I can use.
Then
cdo -b F32 ensmean $MYFILES ensmean_zero_2019.nc
This works great.
BUT
if I try the same within python
Here I have a pandas dataframe with the exact same pathes (df_csv_bin_one_zero['dir_ml'].tolist() == hourly_all_ml_f_zero.txt)
and do
cdo.ensmean(input=df_csv_bin_one_zero['dir_ml'].tolist(), output="ensmean_zero_2019.nc")
I now get an error
OSError: [Errno 7] Argument list too long: '/bin/sh'
I tried the resource modul for python but couldn't fix that problem. Maybe I used this lib wrong.
Can someone tell me how to use CDO in Python with many input files?