Project

General

Profile

merge scattered timesteps

Added by Jannek Squar over 5 years ago

Hey everybody,

to avoid problems with the system's argument limit, which occurs if you execute cdo with too many/long files, I wanted to subdivide my input data set (one file per timestep and vertical level) into several subsets, merge them into temporary files and merge those then together into the final file. I use cdo select,name=q file1 file2 ... to merge the specific humidity from ICON OpenData files (in this case I could also use cdo merge since only one variable is present in the timesteps)
If I do not pay attention to the composition of a subset (I can provide a minimal working example if you wish for), I receive a warning like "Warning (gribapiScanTimestep1): Record 4 (name=q id=0.1.0 lev1=1 lev2=2) timestep 1: Inconsistent verification time!" as well as an error ("cdo select (Abort): Input streams have different number of layers per timestep!").

Merging all files at once works, since all timesteps and vertical layers are available, the problem only strikes if I try to create "arbitrary" subsets. Is there a possibility to go via arbitrary subsets? It is not necessary that the intermediate merged files are readable or you can work with, I'm only interested in the final merged file.


Replies (2)

RE: merge scattered timesteps - Added by Karin Meier-Fleischer over 5 years ago

Hi Jannek,

did you read the Operator news 'Playing with wildcards'? See https://code.mpimet.mpg.de/boards/53/topics/3662
Maybe this will solve your problem.

cdo -select,name=q 'file*.nc' outfile.nc

Without your data I can only guess that you maybe have to take care of the different levels inside your file1, file2, ...
You can use the operator sellevel or select to select the same level for all files.

For example select level 10000

cdo -select,name=q,sellevel=10000 'file*.nc' outfile.nc

-Karin

RE: merge scattered timesteps - Added by Jannek Squar over 5 years ago

Hey Karin,

thanks for your feedback. I separated the files by hand, which works just fine. If there is spare time I'll try the seconds approach (seems more promising regarding the runtime)

Greetings
Jannek

    (1-2/2)