Project

General

Profile

calculating daily climatology from multi-file dataset

Added by Don Murray almost 12 years ago

Hi-

I have daily data which each year in a separate file, e.g.:

temp2_echam5_amip_vary_ghg_mean_1980.nc
temp2_echam5_amip_vary_ghg_mean_1981.nc
temp2_echam5_amip_vary_ghg_mean_1982.nc
temp2_echam5_amip_vary_ghg_mean_1983.nc
temp2_echam5_amip_vary_ghg_mean_1984.nc
temp2_echam5_amip_vary_ghg_mean_1985.nc
temp2_echam5_amip_vary_ghg_mean_1986.nc
temp2_echam5_amip_vary_ghg_mean_1987.nc
etc

I want to create a 30 year climatology from these files and was thinking of using ydaymean. However, ydaymean doesn't take multiple files (at least from the docs). What's the best way to going about this? The files are large so combining them into one 30year file might not be the best thing to do.

Thanks for any help you can give.

Don


Replies (10)

RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller almost 12 years ago

With 1.5.9 you can use wildcards with the cat operators. So you might try something like this:

cdo yeardaymean -cat "temp2_echam5_amip_vary_ghg_mean_19*.nc" climate.nc

hth
ralf

RE: calculating daily climatology from multi-file dataset - Added by Uwe Schulzweida almost 12 years ago

For this feature the wildcards have to be expanded in CDO and not by the Shell. Therefor single quotes must be used:

cdo yeardaymean -cat 'temp2_echam5_amip_vary_ghg_mean_19*.nc' climate.nc

RE: calculating daily climatology from multi-file dataset - Added by Don Murray almost 12 years ago

Ralf and Uwe-

Thanks for the suggestions. Are only simple wildcards allowed, or can I use more complex regular expressions. Since I'll be going over 30 years and want to exclude some of the files, a simple 19* won't work.

Don

RE: calculating daily climatology from multi-file dataset - Added by Uwe Schulzweida almost 12 years ago

Hi Don,

We implemented a simple pattern matching via glob [http://en.wikipedia.org/wiki/Glob_%28programming%29]. Allowed wildcards are `*', `?' or `[ characters ]'.

Uwe

RE: calculating daily climatology from multi-file dataset - Added by Argel Ramírez Reyes over 6 years ago

Hi! I'm just starting to use CDO.

In some posts I have seen the answer "As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.", but the first option appears to overcome that limitation. Is there something I am missing?
Thanks!

RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller over 6 years ago

if you use operators like 'cat' or merge together with a wildcard your can more operators on top of the results of therese operations. But operators output cannot be used for input of 'cat' or 'merge'

hth
ralf

RE: calculating daily climatology from multi-file dataset - Added by Athanasia Nikolaou 3 months ago

Hello,

I am using this command above and I want to combine files which are 18 Gb each for each year.
The above command with the "cat" operator concatenates the input files into one where the climatology is indeed correct but the dimension is huge/equal to the total of input files.
Is there any way that the output file is actually only one file
with 365 values for all the climatological period and does not include the
input files yearly content that increases its dimensions?

Thank you a lot!

RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller 3 months ago

hi!

which command exactly do you use? the yeardaymean will create one output file covering a single year as a climatology. so if your input is daily and is 18GB I expect the output to have the same size.

cheers

RE: calculating daily climatology from multi-file dataset - Added by Athanasia Nikolaou 3 months ago

Hi Ralf!

I am using exactly the "ydaymean" as follows:

cdo ydaymean -cat "filename1year.nc" "filenameAnotherYear.nc" output.nc

(the yeardaymean was not found in the latest documentation I searched https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5980002.8.32
so I assumed you changed name of the function to "ydaymean" which has the same description)

RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller 3 months ago

can you u make the inputs available for download? for testing a small selection would be fine. cdo -remapnn,global_10 will create a very coarse grid. selecting only a single variable will also be useful to reduce filesize. gzip compression works very good with netcdf

    (1-10/10)