Project

General

Profile

How to extract every nth element of a variable

Added by Jonas Hesemann about 7 years ago

Dear community,
best greetings!

My question: Is there a way to extract every nth (e.g. 100th) element of one variable out of a very large 3D-(ICON)-dataset?

I've searched in the CDO documentation and I haven't found this functionality (yet). I'm aware that I can extract data from a given region.

However for statistics I'd like to more properly represent the whole dataset - without taking the time for calculations for my whole 3.3-billion-elements-per-variable(260-GB)-file (260 GB per single timestep). Matlab, for example, can not to this. In my tests it went exponentially slow at about data point 1 billion, when subsetting every 5'000th element. Taking a whole lot more elements from the first 1% of the same file worked surprisingly smooth for statistics. So Matlab has some speed problem with reading data parts at very large indices. I would be especially happy if I could somehow preprocess my large data to feed "only" 30 million elements to my existing Matlab statistic visualization tools - by evenly spaced extraction rather than averaging/interpolating, because the latter methods might destroy what high resolution data have to tell. If I cut the whole file into "subregion files" to process statistics one subregion by another, this takes too much resources in both hard drive space and calculation time.

If CDO can't take out evenly distributed elements by index, any other ideas how I could try to solve this would be apreciated.

Thank you & All the best,
Jonas Hesemann


Replies (1)

RE: How to extract every nth element of a variable - Added by Uwe Schulzweida about 7 years ago

The next CDO release comes with the new operator selindex. This operator selects grid cells by their index. The parameter for selindex is a list of cell indices. Here is an example to extract every 100th element:

cdo  selindex,1/<N>/100  infile  outfile
The maximum number of grid cells in CDO is INT_MAX (2147483647).
A prerelease of the next CDO version is available in the download area.

Cheers,
Uwe

    (1-1/1)