Project

General

Profile

Forums » Open discussion »

Spliting and averaging large files

Added by Carlos Teixeira about 3 years ago

Hello; I have large (~100GB) 30 days outputs and I would like to split that in monthly output and do monthly averages.

I did that in the past for much smaller files merging it first in one file than splitting it (maybe that was dumb but worked).

Now I am trying to do:

cdo -monmean file1.nc file2.nc file3.nc out.nc

and

cdo -splitmon file1.nc file2.nc file3.nc out.nc

And got the following error:

cdo (Abort): Unprocessed Input, could not process all Operators/Files

also tried cdo -splitmon file[1-3].nc out.nc with the same errors.

And cdo mergetime -apply,splitmon file[1-3].nc out.nc with wrong results..

Any help and smarter way to do it?

Many thanks

Carlos


Replies (2)

RE: Spliting and averaging large files - Added by Karin Meier-Fleischer almost 3 years ago

Hi Carlos,

I don't understand what you want to do.

First, the operators monmean and splitmon take only one input file, not multiple. See documentation https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.8.22 and https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#subsection.2.2.10.

If you have a file that contains multiple months you don't have to split the file before doing the monmean.

cdo -monmean infile.nc outfile.nc

If you have multiple monthly files you can merge them with

cdo -O -mergetime 'file*.nc' outfile.nc

If you have multiple monthly files and want to calculate the monthly mean and store the result in one file

cdo -monmean -mergetime 'mon*.nc' o.nc

-Karin

RE: Spliting and averaging large files - Added by Carlos Teixeira almost 3 years ago

Hi Karin;

Thanks for your replay and sorry for the cross posting.

I have ocean model outputs that are not 30 days even (due to restarts) and the idea was to have monthly outputs.

Since the outputs are very large (~100 GB for 30 days), I can not merge 12 months. I was wondering that maybe splitmon could work for multiple files, but as you pointed out, it is not possible. I had not get this info from the manual, sorry.

Thanks for your help!

Carlos

    (1-2/2)