Project

General

Profile

cdo selvar with multiple input files

Added by Stefan Muthers about 13 years ago

Hello,

is it possible to select a variable from multiple input files with cdo, similar to the 'ncrcat -v var1 *.nc out.nc' command?

thank you and best regards,
stefan


Replies (12)

RE: cdo selvar with multiple input files - Added by Uwe Schulzweida about 13 years ago

yes of course, but this feature is still not documented. The name of this powerful operator is select. Use the parameter 'name=' to select variables by name:

cdo select,name=var1,var2,...,varN ifiles ofile

As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.
This operator is still under construction, so please use the latest CDO version.
Your feedback is very welcome.

Best regards,
Uwe

RE: cdo selvar with multiple input files - Added by Stefan Muthers about 13 years ago

wonderful! thank you for the fast response.

best regards,
stefan

RE: cdo selvar with multiple input files - Added by Stefan Muthers about 13 years ago

The combination of select with other operations produces unexpected results.

See the following example:

  • input files are daily values (grb format) with multiple variables per file
  • output should be a file with monthly mean values of a single variable

$ cdo select,name=t -monmean echam5_160001.grb echam5_160002.grb test.nc
cdo select: Started child process "monmean ../1600-ohne/outdata/echam5/1600-ohne_echam5_160001.grb (pipe1.1)".
cdo(2) monmean: Processed 29623680 values from 111 variables over 30 timesteps. ( 0.36s )
cdo select: Processed 601920 values from 222 variables over 30 timesteps. ( 0.48s )

The output already show it, monthly means are only calculated for the first input file.

$ cdo showdate test.nc
1600-01-31 1600-02-01 1600-02-02 1600-02-03 1600-02-04 1600-02-05 1600-02-06 1600-02-07 1600-02-08 1600-02-09 1600-02-10 1600-02-11 1600-02-12 1600-02-13 1600-02-14 1600-02-15 1600-02-16 1600-02-17 1600-02-18 1600-02-19 1600-02-20 1600-02-21 1600-02-22 1600-02-23 1600-02-24 1600-02-25 1600-02-26 1600-02-27 1600-02-28 1600-02-29
cdo showdate: Processed 1 variable over 30 timesteps. ( 0.00s )

I think this is a bug, should a open a new issue?

best regards,
stefan

RE: cdo selvar with multiple input files - Added by Uwe Schulzweida about 13 years ago

PLEASE NOTE: As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.

RE: cdo selvar with multiple input files - Added by Stefan Muthers about 13 years ago

ok. Isn't it possible to interrupt the execution of this command in this case, the same way it is done when an operator that expects one input stream is executed with multiple input files:

Error (cdo sellevel) : Too many streams! Operator needs 1 input and 1 output streams.

(Or at least raise a warning)

Otherwise the user assumes that everything went fine.

Thanks you and best regards,
stefan

RE: cdo selvar with multiple input files - Added by ahmad frahmand over 3 years ago

Uwe Schulzweida wrote:

yes of course, but this feature is still not documented. The name of this powerful operator is select. Use the parameter 'name=' to select variables by name:
[...]
As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.
This operator is still under construction, so please use the latest CDO version.
Your feedback is very welcome.

Best regards,
Uwe

Respected Schulzweids,

I have a same question. i have 4262 nc4 files from GPM daily rainfall data, i would like to select one sub dataset from this nc4 file and save it as new nc file. i can do it one by one using
cdo -selvar,precipitationCal 3B-DAY.MS.MRG.3IMERG.20150108-S000000-E235959.V06.nc4 GMP.nc
but doing for 4262 is impossible to do it one by one. how i can crate a loop for these file.

thank you sir/
Regards

RE: cdo selvar with multiple input files - Added by Ralf Mueller over 3 years ago

Dead Ahmad!

https://tldp.org/LDP/abs/html/loops1.html - this is a basic introduction in to loops with bash. Many other shells behave identical.

for file in 3B-DAY*.nc4; do
  cdo -selvar,precipitationCal $file precip_$file
done

After that you can call merge on the result files
cdo merge precip_*nc4 prec.nc4

For doing this in a single call, you need at least cdo-1.9.8:

cdo -merge -apply,selvar,precipitationCal '3B-DAY*.nc4' precip.nc4

It could be though, that you are not allowed (by the operating system) to open so many files at the same time. In this case use the for-loop, please.

cheers
ralf

RE: cdo selvar with multiple input files - Added by Kasra Keshavarz about 2 years ago

Hi Ralf,

I am trying to use your script and I get the following error...

This is the script:
cdo -merge -apply,selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX [wrf2d_d01_2000-12-31?] ~/h2d_d01_2000-12-31.nc

This is the error:
cdo apply: missing pipe symbol in apply argument: selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX

Wondering what the problem could be...

RE: cdo selvar with multiple input files - Added by Ralf Mueller about 2 years ago

hi!

I think your for got the spaced before and after the brackets. Try:

cdo -merge -apply,selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX [ wrf2d_d01_2000-12-31? ] ~/h2d_d01_2000-12-31.nc

BTW: If possible, please post the complete loop. Otherwise the error might come from code I don't know.

cheers
ralf

RE: cdo selvar with multiple input files - Added by Kasra Keshavarz about 2 years ago

Hi Ralf,

I still get the same error with the spaced brackets; the cdo version is 2.0.4 on a CentOS 7.

This is all the code that I am using, so there is no loop. If I want to use a loop instead of the chained operators of cdo, then it would be something like the following:

for file in wrf2d_d01_2000-12-31*; do
  cdo selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX "$file" "~/${file}.nc" 
done

and then merging:

cdo merge ~/*.nc final_netcdf.nc

Please let me know if I am missing anything else.

RE: cdo selvar with multiple input files - Added by Ralf Mueller about 2 years ago

can you upload at least 2 of the input files?

RE: cdo selvar with multiple input files - Added by Ralf Mueller about 2 years ago

Kasra Keshavarz wrote in RE: cdo selvar with multiple input files:

Hi Ralf,

I still get the same error with the spaced brackets; the cdo version is 2.0.4 on a CentOS 7.

This is all the code that I am using, so there is no loop. If I want to use a loop instead of the chained operators of cdo, then it would be something like the following:
[...]

and then merging:
[...]

Please let me know if I am missing anything else.

the loop above looks good to me.

    (1-12/12)