Project

General

Profile

Cdo in Python too many files error

Added by Michael Debertshaeuser almost 3 years ago

Hey,

I try to get a ensmean out of many input files.
input files = hourly_all_ml_f_zero.txt

I use for the terminal:
MYFILES=`cat /home/mdeberts/masterthesis/preprocessing/data/hourly_all_ml_f_zero.txt | cut -f2 -d" "`

Then
ulimit -s 15000000000000000
ulimit -n 15000

to increase the maximum of files I can use.

Then
cdo -b F32 ensmean $MYFILES ensmean_zero_2019.nc

This works great.

BUT
if I try the same within python

Here I have a pandas dataframe with the exact same pathes (df_csv_bin_one_zero['dir_ml'].tolist() == hourly_all_ml_f_zero.txt)
and do
cdo.ensmean(input=df_csv_bin_one_zero['dir_ml'].tolist(), output="ensmean_zero_2019.nc")

I now get an error
OSError: [Errno 7] Argument list too long: '/bin/sh'

I tried the resource modul for python but couldn't fix that problem. Maybe I used this lib wrong.

Can someone tell me how to use CDO in Python with many input files?