Project

General

Profile

How to create a timeseries from multiple nc data files using -seltimestep or any other operator.

Added by Mudit Mudit about 3 years ago

Hello everyone,
I am new with CDO and I wish to know that how can I create a timeseries data from multiple nc files.
I tried to write bash sh script but I am not sure whether it is correct or not.

Also I'm getting error message:
./code2.sh: line 2: syntax error near unexpected token `$'do\r''
'/code2.sh: line 2: `for year in 'seq -w 1961 2014' ; do

Each file has 10year monthly data in it having total 120 month data in a single file, and I want to extract 1st year data i.e. 1/12 months data from each file and make a timeseries from it in a single file. Kindly help me on how to write the CDO command/code so that I can get the result.
I tried to write the code as:
#!/bin/bash
for year in 'seq -w 1961 2014' ; do
cdo -seltimestep,1/12 pr_Amon_BCC-CSM2-MR_dcppA-hindcast_s${year}.nc series${year}.nc
done

I am attaching four sample files.
Thanks for Help in advance :)


Replies (10)

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller about 3 years ago

hi Mudit!

the error is thrown by bash not by CDO. So in order to check what's up, you need to upload the bash script you are using. for me it looks like MS-DOS file format problem regarding the line endings:

  • unix: \n
  • dos: \n\r

in this case, use a tool like dos2unix for go to unix format and rerun. That would be my recommendation.

hth
ralf

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit about 3 years ago

Hello Ralf,
I am attaching the script file, please let me know what mistake I am doing, also kindly correct the script if I had written it wrongly.
Thanks and Regards
Mudit.

code2.sh (141 Bytes) code2.sh

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller about 3 years ago

❯ file code2.sh
code2.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators
❯ sh code2.sh                                                                                                             
code2.sh: line 2: syntax error near unexpected token `$'do\r''
'ode2.sh: line 2: `for year in 'seq -w 1961 2014' ; do
❯ dos2unix code2.sh                                                                                       
dos2unix: converting file code2.sh to Unix format...
❯ file code2.sh            
code2.sh: Bourne-Again shell script, ASCII text executable
❯ sh code2.sh                                                                                    
cdo (Abort): Unprocessed Input, could not process all Operators/Files

Not the loop is at least started, but CDO does not seem to find the input.
I noticed that you used single quotes "'" around the seq call. This turns the call into a string, which is probably not what you want. Instead you should use backticks ` for use the result of the command as input. I uploaded a corrected version and also another version that used $() instead of backticks. I like the $() construction more, because it can be nested.

hth
ralf

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit about 3 years ago

Thanks sir for the help, both the updated files are working. there is one more query, I wish to store the selected time steps in a single file, what changes do I need to do in the file? I am getting different files for each year.

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit about 3 years ago

I am attaching the code that I am doing right now but again the error is coming:
cdo mergetime: Open failed on >.nc<
No such file or directory *

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller about 3 years ago

the tmpdir variable is not set. I corrected this in my version. then the script runs through. I also had to limit the list of years because I only have the first 4 files.

#!/bin/bash
tmpdir=./tmp
mkdir -p $tmpdir
for year in $(seq -w 1961 1964) ; do
  echo $year
  cdo -seltimestep,1/12 pr_Amon_BCC-CSM2-MR_dcppA-hindcast_s${year}.nc $tmpdir/series${year}.nc
done
cd $tmpdir
cdo mergetime '*.nc' series_t1.nc

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit over 2 years ago

Hello Sir,
The previous files ran perfectly.
I have some new files with a pattern in filename as:

tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1961-r1i1p1f1_gn_196101-197012.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1962-r1i1p1f1_gn_196201-197112.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1963-r1i1p1f1_gn_196301-197212.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1964-r1i1p1f1_gn_196401-197312.nc
tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s1965-r1i1p1f1_gn_196501-197412.nc

I am trying to write its filename pattern, but I am doing some kind of error in writing the same, here is the code which I am trying:

#!/bin/bash
tmpdir=./tmp
mkdir -p $tmpdir
for year in $(seq -w 1961 2014) ; do
  echo $year
  cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-${year+9}12.nc $tmpdir/series${year}.nc
done
cd $tmpdir
cdo mergetime '*.nc' series_s1961_t1_tos_Omon_BCC-CSM2-MR_dcppA-hindcast.nc
exit 0

I am also attaching the sample files.

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller over 2 years ago

hi!

First let we give a more general advice: If you write loops with more complex variable handling or pattern matching, it can be very helpful to print the generated commands before executing them. This way you can analyze the results of the variable expansion:

for year in $(seq -w 1961 2014) ; do
  echo $year
  echo cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-${year+9}12.nc $tmpdir/series${year}.nc
done

That way you can see that the term ${year+9} does not perform an arithmetic expression. Those have to be done in a different manor:

cdo -seltimestep,1/12 tos_Omon_BCC-CSM2-MR_dcppA-hindcast_s${year}-r1i1p1f1_gn_${year}01-$((year+9))12.nc $tmpdir/series${year}.nc

some general remaps in code

a=1
((a=a+13))
echo $a
14
b=$((a=a+13))
echo $b
27

IMO the best documentation is the advanced bash scripting guide

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Mudit Mudit over 2 years ago

Thank you sir, its working perfectly now, I was unable to perform that arithmetic expression.

RE: How to create a timeseries from multiple nc data files using -seltimestep or any other operator. - Added by Ralf Mueller over 2 years ago

as is said: the advanced bash scripting guide is very good - totally worth reading

    (1-10/10)