Project

General

Profile

Ensmean over 10 years

Added by Michael Debertshaeuser about 3 years ago

Hello,

I want to compute the mean temperature, u, v etc for hourly *.nc files for the last 10 years.
Apparently that are many files and I get an error if I run the ensmean.
I pass in a list of all the hourly filenames within the bash shell and I get an error

cdo ensmean $MYFILES ~/ml_full_hpc_ensmean.nc

--->

"/var/tmp/slurmd_spool/job8844938/slurm_script: line 42: /cluster/easybuild/broadwell/software/CDO/1.9.8-intel-2019b/bin/cdo: Argument list too long"

I already used ulimit to enhance the maximum length for the arguments.

Is there a way to pass those files on the fly or a batch of files, rather than needing to pass all files at once?

I found this connected topic but wonder if something changed within the last four years.
https://code.mpimet.mpg.de/boards/1/topics/4326


Replies (5)

RE: Ensmean over 10 years - Added by Michael Debertshaeuser about 3 years ago

The data for 1 month is around 5000 MB.
So for the entire dataset it's 5000 MB * 12 * 10 = 600000 MB

If I try it with cdo do I need 600000MB Ram?

RE: Ensmean over 10 years - Added by Karin Meier-Fleischer about 3 years ago

Hi Michael,

the error message tells you that your command line argument list is too long.

Without the files I can only guess that you have to play the wild card game - see https://code.mpimet.mpg.de/boards/53/topics/3662

cdo -ensmean ’*.nc*’ outfile.nc

-Karin

RE: Ensmean over 10 years - Added by Uwe Schulzweida about 3 years ago

Hi Michael,

Yes, ensmean needs to read one horizonzal field from all files.
Try timmean in combination with cat to reduce the memory consumption:

cdo timmean -cat ’*.nc*’ outfile.nc
-Uwe

RE: Ensmean over 10 years - Added by Michael Debertshaeuser about 3 years ago

Danke für die Hilfe,

ich konnte das Problem lösen indem ich die stündlichen files in ein file pro Jahr umgewandelt habe und damit den Fehler

Argument list too long

umgangen.

Allerdings habe ich auch kein ensmean benötigt sondern timmean danke für den Tip.

RE: Ensmean over 10 years - Added by Michael Debertshaeuser about 3 years ago

Michael Debertshaeuser wrote:

Danke für die Hilfe,

ich konnte das Problem lösen indem ich die stündlichen files in ein file pro Jahr umgewandelt habe und damit den Fehler

Argument list too long

umgangen.

Allerdings habe ich auch kein ensmean benötigt sondern timmean danke für den Tip.

Sorry I was answering in german out of a habit.

Thanks for the help,

I was able to solve the problem by converting the hourly files to one file per year and thus eliminating the error

Argument list too long

However, I also did not need ensmean but timmean thanks for the tip.

    (1-5/5)