How to create a timeseries from multiple nc data files using -seltimestep or any other operator.
Added by Mudit Mudit over 3 years ago
Hello everyone,
I am new with CDO and I wish to know that how can I create a timeseries data from multiple nc files.
I tried to write bash sh script but I am not sure whether it is correct or not.
Also I'm getting error message:
./code2.sh: line 2: syntax error near unexpected token `$'do\r''
'/code2.sh: line 2: `for year in 'seq -w 1961 2014' ; do
Each file has 10year monthly data in it having total 120 month data in a single file, and I want to extract 1st year data i.e. 1/12 months data from each file and make a timeseries from it in a single file. Kindly help me on how to write the CDO command/code so that I can get the result.
I tried to write the code as:
#!/bin/bash
for year in 'seq -w 1961 2014' ; do
cdo -seltimestep,1/12 pr_Amon_BCC-CSM2-MR_dcppA-hindcast_s${year}.nc series${year}.nc
done
I am attaching four sample files.
Thanks for Help in advance
Replies (10)
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller over 3 years ago
hi Mudit!
the error is thrown by bash not by CDO. So in order to check what's up, you need to upload the bash script you are using. for me it looks like MS-DOS file format problem regarding the line endings:
- unix: \n
- dos: \n\r
in this case, use a tool like dos2unix
for go to unix format and rerun. That would be my recommendation.
hth
ralf
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit over 3 years ago
Hello Ralf,
I am attaching the script file, please let me know what mistake I am doing, also kindly correct the script if I had written it wrongly.
Thanks and Regards
Mudit.
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller over 3 years ago
❯ file code2.sh code2.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators ❯ sh code2.sh code2.sh: line 2: syntax error near unexpected token `$'do\r'' 'ode2.sh: line 2: `for year in 'seq -w 1961 2014' ; do ❯ dos2unix code2.sh dos2unix: converting file code2.sh to Unix format... ❯ file code2.sh code2.sh: Bourne-Again shell script, ASCII text executable ❯ sh code2.sh cdo (Abort): Unprocessed Input, could not process all Operators/Files
Not the loop is at least started, but CDO does not seem to find the input.
I noticed that you used single quotes "'" around the
seq
call. This turns the call into a string, which is probably not what you want. Instead you should use backticks `
for use the result of the command as input. I uploaded a corrected version and also another version that used $() instead of backticks. I like the $() construction more, because it can be nested.
hth
ralf
code2_cor1.sh (139 Bytes) code2_cor1.sh | |||
code2_cor2.sh (140 Bytes) code2_cor2.sh |
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit over 3 years ago
Thanks sir for the help, both the updated files are working. there is one more query, I wish to store the selected time steps in a single file, what changes do I need to do in the file? I am getting different files for each year.
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit over 3 years ago
I am attaching the code that I am doing right now but again the error is coming:
cdo mergetime: Open failed on >.nc<
No such file or directory
*
code2_cor3.sh (198 Bytes) code2_cor3.sh |
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller over 3 years ago
the tmpdir
variable is not set. I corrected this in my version. then the script runs through. I also had to limit the list of years because I only have the first 4 files.
#!/bin/bash
tmpdir=./tmp
mkdir -p $tmpdir
for year in $(seq -w 1961 1964) ; do
echo $year
cdo -seltimestep,1/12 pr_Amon_BCC-CSM2-MR_dcppA-hindcast_s${year}.nc $tmpdir/series${year}.nc
done
cd $tmpdir
cdo mergetime '*.nc' series_t1.nc
code2_cor4.sh (245 Bytes) code2_cor4.sh |
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit about 3 years ago
Hello Sir,
The previous files ran perfectly.
I have some new files with a pattern in filename as:
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1961-r1i1p1f1_gn_196101-197012.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1962-r1i1p1f1_gn_196201-197112.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1963-r1i1p1f1_gn_196301-197212.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1964-r1i1p1f1_gn_196401-197312.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1965-r1i1p1f1_gn_196501-197412.nc
I am trying to write its filename pattern, but I am doing some kind of error in writing the same, here is the code which I am trying:
#!/bin/bash tmpdir=./tmp mkdir -p $tmpdir for year in $(seq -w 1961 2014) ; do echo $year cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-${year+9}12.nc $tmpdir/series${year}.nc done cd $tmpdir cdo mergetime '*.nc' series_s1961_t1_tos_Omon_BCC-CSM2-MR_dcppA-hindcast.nc exit 0
I am also attaching the sample files.
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller about 3 years ago
hi!
First let we give a more general advice: If you write loops with more complex variable handling or pattern matching, it can be very helpful to print the generated commands before executing them. This way you can analyze the results of the variable expansion:
for year in $(seq -w 1961 2014) ; do echo $year echo cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-${year+9}12.nc $tmpdir/series${year}.nc done
That way you can see that the term ${year+9}
does not perform an arithmetic expression. Those have to be done in a different manor:
cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-$((year+9))12.nc $tmpdir/series${year}.nc
some general remaps in code
a=1 ((a=a+13)) echo $a 14 b=$((a=a+13)) echo $b 27
IMO the best documentation is the advanced bash scripting guide
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit about 3 years ago
Thank you sir, its working perfectly now, I was unable to perform that arithmetic expression.
RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller about 3 years ago
as is said: the advanced bash scripting guide is very good - totally worth reading