Project

General

Profile

LOOPING OF MULTIPLE FILES

Added by selam haftu over 3 years ago

i am new to CDO, i have gone through the manual and i can perform some operations. however, my work is too bulky, i have about 1500 files from ESGF with different variables. for each file, i have to remap, extract the study area, and calculate area mean. i can however perform this operation for individual file, but to perform a loop to ease this task is my difficulties. i cant find such operation in the manual. please provide me with the command on how to perform the loop operation for my multiple files so that i can perform at once. Also, can i perform all the three task together or it must be one after the other.
Thank you for your time and effort.
CDO version: 1.9.4rc1


Replies (4)

RE: LOOPING OF MULTIPLE FILES - Added by Ralf Mueller over 3 years ago

Hi Selam!

1500 files is a lot indeed. CDO does not offer looping itself, but this seem to be a perfect example for a shell loop. I posted some techniques how to speed up data analysis some time ago here

you might have a look at point 1) and 2). Remapping benefits from OpenMP-parallelization (1) and a proper for-loop is shown in (2). Appending

 | parallel -v -j 12
will process things in parallel, but for a first try I would leave this out (and also the "echo" stuff around the CDO command line).

hth
ralf

RE: LOOPING OF MULTIPLE FILES - Added by selam haftu over 3 years ago

thank you for your response. i saw a similar question by a user in the forum, i don't know if i can use the the same method she used for few files at a given time with her loop (for loop).
Her question can be found here https://code.mpimet.mpg.de/boards/2/topics/6396 if its possible, kindly assist me in defining it. thank you for all your support, we really appreciate.

RE: LOOPING OF MULTIPLE FILES - Added by Ralf Mueller over 3 years ago

sure, her solution is also fine

for i in $(ls); do cdo remapnn,grid2.txt ${i} regrid/${i}wgs.nc; done
Itis the same that I pointed to, but without the extra parallelization command. But that's no problem - it will just take a bit longer.

cheers
ralf

RE: LOOPING OF MULTIPLE FILES - Added by selam haftu over 3 years ago

thank you Ralf.

    (1-4/4)