cdo selvar with multiple input files
Added by Stefan Muthers about 14 years ago
Hello,
is it possible to select a variable from multiple input files with cdo, similar to the 'ncrcat -v var1 *.nc out.nc' command?
thank you and best regards,
stefan
Replies (12)
RE: cdo selvar with multiple input files - Added by Uwe Schulzweida about 14 years ago
yes of course, but this feature is still not documented. The name of this powerful operator is select. Use the parameter 'name=' to select variables by name:
cdo select,name=var1,var2,...,varN ifiles ofile
As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.
This operator is still under construction, so please use the latest CDO version.
Your feedback is very welcome.
Best regards,
Uwe
RE: cdo selvar with multiple input files - Added by Stefan Muthers about 14 years ago
wonderful! thank you for the fast response.
best regards,
stefan
RE: cdo selvar with multiple input files - Added by Stefan Muthers about 14 years ago
The combination of select with other operations produces unexpected results.
See the following example:
- input files are daily values (grb format) with multiple variables per file
- output should be a file with monthly mean values of a single variable
$ cdo select,name=t -monmean echam5_160001.grb echam5_160002.grb test.nc
cdo select: Started child process "monmean ../1600-ohne/outdata/echam5/1600-ohne_echam5_160001.grb (pipe1.1)".
cdo(2) monmean: Processed 29623680 values from 111 variables over 30 timesteps. ( 0.36s )
cdo select: Processed 601920 values from 222 variables over 30 timesteps. ( 0.48s )
The output already show it, monthly means are only calculated for the first input file.
$ cdo showdate test.nc
1600-01-31 1600-02-01 1600-02-02 1600-02-03 1600-02-04 1600-02-05 1600-02-06 1600-02-07 1600-02-08 1600-02-09 1600-02-10 1600-02-11 1600-02-12 1600-02-13 1600-02-14 1600-02-15 1600-02-16 1600-02-17 1600-02-18 1600-02-19 1600-02-20 1600-02-21 1600-02-22 1600-02-23 1600-02-24 1600-02-25 1600-02-26 1600-02-27 1600-02-28 1600-02-29
cdo showdate: Processed 1 variable over 30 timesteps. ( 0.00s )
I think this is a bug, should a open a new issue?
best regards,
stefan
RE: cdo selvar with multiple input files - Added by Uwe Schulzweida about 14 years ago
PLEASE NOTE: As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.
RE: cdo selvar with multiple input files - Added by Stefan Muthers about 14 years ago
ok. Isn't it possible to interrupt the execution of this command in this case, the same way it is done when an operator that expects one input stream is executed with multiple input files:
Error (cdo sellevel) : Too many streams! Operator needs 1 input and 1 output streams.
(Or at least raise a warning)
Otherwise the user assumes that everything went fine.
Thanks you and best regards,
stefan
RE: cdo selvar with multiple input files - Added by ahmad frahmand over 4 years ago
Uwe Schulzweida wrote:
yes of course, but this feature is still not documented. The name of this powerful operator is select. Use the parameter 'name=' to select variables by name:
[...]
As all CDO operators with an unlimited number of input files this operator can't be combined with other operators.
This operator is still under construction, so please use the latest CDO version.
Your feedback is very welcome.Best regards,
Uwe
Respected Schulzweids,
I have a same question. i have 4262 nc4 files from GPM daily rainfall data, i would like to select one sub dataset from this nc4 file and save it as new nc file. i can do it one by one using
cdo -selvar,precipitationCal 3B-DAY.MS.MRG.3IMERG.20150108-S000000-E235959.V06.nc4 GMP.nc
but doing for 4262 is impossible to do it one by one. how i can crate a loop for these file.
thank you sir/
Regards
RE: cdo selvar with multiple input files - Added by Ralf Mueller over 4 years ago
Dead Ahmad!
https://tldp.org/LDP/abs/html/loops1.html - this is a basic introduction in to loops with bash. Many other shells behave identical.
for file in 3B-DAY*.nc4; do cdo -selvar,precipitationCal $file precip_$file done
After that you can call merge on the result files
cdo merge precip_*nc4 prec.nc4
For doing this in a single call, you need at least cdo-1.9.8:
cdo -merge -apply,selvar,precipitationCal '3B-DAY*.nc4' precip.nc4
It could be though, that you are not allowed (by the operating system) to open so many files at the same time. In this case use the for-loop, please.
cheers
ralf
RE: cdo selvar with multiple input files - Added by Kasra Keshavarz about 3 years ago
Hi Ralf,
I am trying to use your script and I get the following error...
This is the script:cdo -merge -apply,selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX [wrf2d_d01_2000-12-31?] ~/h2d_d01_2000-12-31.nc
This is the error:cdo apply: missing pipe symbol in apply argument: selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX
Wondering what the problem could be...
RE: cdo selvar with multiple input files - Added by Ralf Mueller about 3 years ago
hi!
I think your for got the spaced before and after the brackets. Try:
cdo -merge -apply,selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX [ wrf2d_d01_2000-12-31? ] ~/h2d_d01_2000-12-31.nc
BTW: If possible, please post the complete loop. Otherwise the error might come from code I don't know.
cheers
ralf
RE: cdo selvar with multiple input files - Added by Kasra Keshavarz about 3 years ago
Hi Ralf,
I still get the same error with the spaced brackets; the cdo version is 2.0.4 on a CentOS 7.
This is all the code that I am using, so there is no loop. If I want to use a loop instead of the chained operators of cdo, then it would be something like the following:
for file in wrf2d_d01_2000-12-31*; do cdo selvar,T2,Q2,PSFC,U,V,GLW,LH,SWDOWN,QFX,HFX "$file" "~/${file}.nc" done
and then merging:
cdo merge ~/*.nc final_netcdf.nc
Please let me know if I am missing anything else.
RE: cdo selvar with multiple input files - Added by Ralf Mueller about 3 years ago
can you upload at least 2 of the input files?
RE: cdo selvar with multiple input files - Added by Ralf Mueller about 3 years ago
Kasra Keshavarz wrote in RE: cdo selvar with multiple input files:
Hi Ralf,
I still get the same error with the spaced brackets; the cdo version is 2.0.4 on a CentOS 7.
This is all the code that I am using, so there is no loop. If I want to use a loop instead of the chained operators of cdo, then it would be something like the following:
[...]and then merging:
[...]Please let me know if I am missing anything else.
the loop above looks good to me.