calculating daily climatology from multi-file dataset
Added by Don Murray almost 12 years ago
Hi-
I have daily data which each year in a separate file, e.g.:
temp2_echam5_amip_vary_ghg_mean_1980.nc
temp2_echam5_amip_vary_ghg_mean_1981.nc
temp2_echam5_amip_vary_ghg_mean_1982.nc
temp2_echam5_amip_vary_ghg_mean_1983.nc
temp2_echam5_amip_vary_ghg_mean_1984.nc
temp2_echam5_amip_vary_ghg_mean_1985.nc
temp2_echam5_amip_vary_ghg_mean_1986.nc
temp2_echam5_amip_vary_ghg_mean_1987.nc
etc
I want to create a 30 year climatology from these files and was thinking of using ydaymean. However, ydaymean doesn't take multiple files (at least from the docs). What's the best way to going about this? The files are large so combining them into one 30year file might not be the best thing to do.
Thanks for any help you can give.
Don
Replies (10)
RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller almost 12 years ago
With 1.5.9 you can use wildcards with the cat operators. So you might try something like this:
cdo yeardaymean -cat "temp2_echam5_amip_vary_ghg_mean_19*.nc" climate.nc
hth
ralf
RE: calculating daily climatology from multi-file dataset - Added by Uwe Schulzweida almost 12 years ago
For this feature the wildcards have to be expanded in CDO and not by the Shell. Therefor single quotes must be used:
cdo yeardaymean -cat 'temp2_echam5_amip_vary_ghg_mean_19*.nc' climate.nc
RE: calculating daily climatology from multi-file dataset - Added by Don Murray almost 12 years ago
Ralf and Uwe-
Thanks for the suggestions. Are only simple wildcards allowed, or can I use more complex regular expressions. Since I'll be going over 30 years and want to exclude some of the files, a simple 19* won't work.
Don
RE: calculating daily climatology from multi-file dataset - Added by Uwe Schulzweida almost 12 years ago
Hi Don,
We implemented a simple pattern matching via glob [http://en.wikipedia.org/wiki/Glob_%28programming%29]. Allowed wildcards are `*', `?' or `[ characters ]'.
Uwe
RE: calculating daily climatology from multi-file dataset - Added by Argel Ramírez Reyes over 6 years ago
Hi! I'm just starting to use CDO.
In some posts I have seen the answer "As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.", but the first option appears to overcome that limitation. Is there something I am missing?
Thanks!
RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller over 6 years ago
if you use operators like 'cat' or merge together with a wildcard your can more operators on top of the results of therese operations. But operators output cannot be used for input of 'cat' or 'merge'
hth
ralf
RE: calculating daily climatology from multi-file dataset - Added by Athanasia Nikolaou 2 months ago
Hello,
I am using this command above and I want to combine files which are 18 Gb each for each year.
The above command with the "cat" operator concatenates the input files into one where the climatology is indeed correct but the dimension is huge/equal to the total of input files.
Is there any way that the output file is actually only one file
with 365 values for all the climatological period and does not include the
input files yearly content that increases its dimensions?
Thank you a lot!
RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller 2 months ago
hi!
which command exactly do you use? the yeardaymean
will create one output file covering a single year as a climatology. so if your input is daily and is 18GB I expect the output to have the same size.
cheers
RE: calculating daily climatology from multi-file dataset - Added by Athanasia Nikolaou 2 months ago
Hi Ralf!
I am using exactly the "ydaymean" as follows:
cdo ydaymean -cat "filename1year.nc" "filenameAnotherYear.nc" output.nc
(the yeardaymean was not found in the latest documentation I searched https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-5980002.8.32
so I assumed you changed name of the function to "ydaymean" which has the same description)
RE: calculating daily climatology from multi-file dataset - Added by Ralf Mueller 2 months ago
can you u make the inputs available for download? for testing a small selection would be fine. cdo -remapnn,global_10
will create a very coarse grid. selecting only a single variable will also be useful to reduce filesize. gzip compression works very good with netcdf