Project

General

Profile

operator piping help: extracting and merging at the same time?

Added by Erik Noble over 12 years ago

Dear CDO community; thank you for providing CDO.

Could I please have some help?
I have 100 files in a directory. I want to extract 4 variables from the files and make merge the results together to make one file.
My current script snippet is below and it works. I do a loop for the extraction and then merge outside of the loop; it takes a long time to finish. I am worried because I have to do this iteratively hundreds of times. There has to be a more efficient method. I am new to CDO. Perhaps I missed it?
How can I do the 2 cdo commands together (extract and merge or vice-versa) to reduce unnecessary disk I/O (like the example from the CDO tutorial below)? Is this possible?
Any advice, pointers would be greatly appreciated.
Sincerely
-Erik Noble

< code snippet in a bash shell > #######################
export FILES=`ls post_wrfout_d01*`
for fili in $FILES
do
if [ ! -e "$fili" ] # Check if file exists.
then
echo "$fili does not exist."; echo
continue # On to next.
fi
echo "Using CDO selname"

  1. Run script
    cdo selname,u_gr_p,v_gr_p,u_10m_gr,v_10m_gr,DateTime $fili temp_$fili # CDO command 1: Extraction
    done
    echo "Merging to make Experiment_01_diagnostics.nc"
    cdo mergetime temp_* Experiment_01_diagnostics.nc # CDO command 2: timemerge
    rm temp_*
    done

#######################

Example from CDO Tutorial
Operator piping
All operators with one output stream can pipe the result directly to an other operator. The operator must begin with "-", in order to combine it with others. This can improve the performance by:
reducing unnecessary disk I/O
parallel processing
Use
cdo sub -dayavg ifile2 -timavg ifile1 ofile
instead of
cdo timavg ifile1 tmp1
cdo dayavg ifile2 tmp2
cdo sub tmp2 tmp1 ofile
rm tmp1 tmp2


Replies (2)

RE: operator piping help: extracting and merging at the same time? - Added by Uwe Schulzweida over 12 years ago

Dear Erik,

Please try the undocumented CDO operator select with the parameter name= to select variables by name from an unlimited number of inputfiles:

cdo select,name=u_gr_p,v_gr_p,u_10m_gr,v_10m_gr,DateTime $FILES Experiment_01_diagnostics.nc
The names of the input files have to be sorted alphanumeric in order to get a correct time axis.
Please let us know your experiences about the performance.

Best regards,
Uwe

RE: operator piping help: extracting and merging at the same time? - Added by nitin patil over 7 years ago

Dear All,

I have multiple files and need to merge into single netcdf with only selected variables for this I am trying the following and getting error:

cdo mergetime -selname,variable1,variable2 ifile1.nc ifile2.nc ifile3.nc ofile.nc

error:
cdo mergetime: Started child process "select,name=TAU_2D_550nm EACSST_197101.01_rad.nc (pipe1.1)".
cdo mergetime (Abort): Input streams have different number of variables per timestep!

Any mistake I am doing?

Thanks,
Nitin

    (1-2/2)