CDO

http://www.tamsat.org.uk/regridded

I should add that I am using TAMSAT regridded data at 0.25 degrees resolution, and that my CHIRTS data is also at 0.25 degrees resolution. I also realised I could attach an example TAMSAT file if I compressed it, so here it is attached. Thanks in advance!

Hi Jonathan,

1. the last time step of the file Tmax.1983.nc is wrong
2. you can delete the 29th of February with the operator del29feb

You can fix it all:

cdo -seltimestep,1/-2 Tmax.1983.nc Tmax.1983_365.nc

cdo -del29feb Tmax.2016.nc Tmax.2016_365.nc

cdo -del29feb TAMSAT_2016.nc TAMSAT_2016_365.nc

Hi Karin,

Thank you very much for your help! The last two bits of code worked, but I got this error for the first section of code:

'cdo -seltimestep,1/-2 Tmax.2016.nc Tmax.2016_365.nc

cdo seltimestep (Abort): No timestep selected!'

Best regards,

Jonathan

I've copied the first command and it works. Do you have a typo?

Only Tmax.1983.nc has the wrong date at last timestep that has to be skipped.

cdo sinfon Tmax.1983.nc

   File format : NetCDF4
    -1 : Institut Source   T Steptype Levels Num    Points Num Dtype : Parameter name
     1 : unknown  unknown  v instant       1   1     96000   1  F32  : Tmax          
   Grid coordinates :
     1 : lonlat                   : points=96000 (300x320)
                        longitude : -19.875 to 54.875 by 0.25 degrees_east
                         latitude : -39.875 to 39.875 by 0.25 degrees_north
   Vertical coordinates :
     1 : surface                  : levels=1
   Time coordinate :
                             time : 366 steps
     RefTime =  1980-01-01 00:00:00  Units = days  Calendar = gregorian
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  1983-01-01 00:00:00  1983-01-02 00:00:00  1983-01-03 00:00:00  1983-01-04 00:00:00
  1983-01-05 00:00:00  1983-01-06 00:00:00  1983-01-07 00:00:00  1983-01-08 00:00:00
  1983-01-09 00:00:00  1983-01-10 00:00:00  1983-01-11 00:00:00  1983-01-12 00:00:00
  1983-01-13 00:00:00  1983-01-14 00:00:00  1983-01-15 00:00:00  1983-01-16 00:00:00
  1983-01-17 00:00:00  1983-01-18 00:00:00  1983-01-19 00:00:00  1983-01-20 00:00:00
  1983-01-21 00:00:00  1983-01-22 00:00:00  1983-01-23 00:00:00  1983-01-24 00:00:00
  1983-01-25 00:00:00  1983-01-26 00:00:00  1983-01-27 00:00:00  1983-01-28 00:00:00
  1983-01-29 00:00:00  1983-01-30 00:00:00  1983-01-31 00:00:00  1983-02-01 00:00:00
  1983-02-02 00:00:00  1983-02-03 00:00:00  1983-02-04 00:00:00  1983-02-05 00:00:00
  1983-02-06 00:00:00  1983-02-07 00:00:00  1983-02-08 00:00:00  1983-02-09 00:00:00
  1983-02-10 00:00:00  1983-02-11 00:00:00  1983-02-12 00:00:00  1983-02-13 00:00:00
  1983-02-14 00:00:00  1983-02-15 00:00:00  1983-02-16 00:00:00  1983-02-17 00:00:00
  1983-02-18 00:00:00  1983-02-19 00:00:00  1983-02-20 00:00:00  1983-02-21 00:00:00
  1983-02-22 00:00:00  1983-02-23 00:00:00  1983-02-24 00:00:00  1983-02-25 00:00:00
  1983-02-26 00:00:00  1983-02-27 00:00:00  1983-02-28 00:00:00  1983-03-01 00:00:00
   ................................................................................
   .................
  1983-11-05 00:00:00  1983-11-06 00:00:00  1983-11-07 00:00:00  1983-11-08 00:00:00
  1983-11-09 00:00:00  1983-11-10 00:00:00  1983-11-11 00:00:00  1983-11-12 00:00:00
  1983-11-13 00:00:00  1983-11-14 00:00:00  1983-11-15 00:00:00  1983-11-16 00:00:00
  1983-11-17 00:00:00  1983-11-18 00:00:00  1983-11-19 00:00:00  1983-11-20 00:00:00
  1983-11-21 00:00:00  1983-11-22 00:00:00  1983-11-23 00:00:00  1983-11-24 00:00:00
  1983-11-25 00:00:00  1983-11-26 00:00:00  1983-11-27 00:00:00  1983-11-28 00:00:00
  1983-11-29 00:00:00  1983-11-30 00:00:00  1983-12-01 00:00:00  1983-12-02 00:00:00
  1983-12-03 00:00:00  1983-12-04 00:00:00  1983-12-05 00:00:00  1983-12-06 00:00:00
  1983-12-07 00:00:00  1983-12-08 00:00:00  1983-12-09 00:00:00  1983-12-10 00:00:00
  1983-12-11 00:00:00  1983-12-12 00:00:00  1983-12-13 00:00:00  1983-12-14 00:00:00
  1983-12-15 00:00:00  1983-12-16 00:00:00  1983-12-17 00:00:00  1983-12-18 00:00:00
  1983-12-19 00:00:00  1983-12-20 00:00:00  1983-12-21 00:00:00  1983-12-22 00:00:00
  1983-12-23 00:00:00  1983-12-24 00:00:00  1983-12-25 00:00:00  1983-12-26 00:00:00
  1983-12-27 00:00:00  1983-12-28 00:00:00  1983-12-29 00:00:00  1983-12-30 00:00:00
  1983-12-31 00:00:00  1980-01-01 00:00:00

Tmax.2016.nc and TAMSAT_2016.nc have correct 366 days including 29th February.

Hi Karin,

You're right, that's why it didnt work! It is working with non-leap year files that have 366 days.

Thanks for your help!

Best regards

Jonathan

Hi Karin,

Just wondering if it is possible to do if then else statements in a loop?

I have a series of .nc files over 30 years. When there is a leap year, I want to remove the leap year day, and if there is not a leap year, it means the 366th day will be 1980-01-01 00:00:00 so I want to remove that one. Is there a way to loop over the files and apply the operators below?

I want to make a looping function for the following for my TAMSAT data (TAMSAT_YYYY.nc):

1) if leap year cdo -del29feb
2) if not cdo -seltimestep,1/-2
3) regrid cdo -rempabil,GRID_CHIRTS.txt
3) change calendar cdo setcalendar,365_day
4) change reference period, cdo setreftime,1980-01-01,00:00:00,1days

Best regards

Jonathan

You have to check the input files if the last timestep is equal to '1980-01-01T00:00:00' therefore you need to write a shell script.

For the example script below I used the Tmax.*.nc files from above for testing.

#!/usr/bin/env bash

wrong_time='1980-01-01T00:00:00'

infiles=$(ls Tmax.*.nc)

for f in $infiles
do
    outfile="${f%.*}_remap.nc" 

    last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

    if [ $last_t == $wrong_time ]; then
        cmd_deltime="-seltimestep,1/-2" 
    fi

    cdo -setreftime,1980-01-01,00:00:00,1days \
        -setcalendar,365_day \
        -rempabil,GRID_CHIRTS.txt \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile
done

This script produces some warnings but that can be ignored.

Hi Karin,

Thanks so much for your help! I ran the script as below:

#!/usr/bin/env bash

wrong_time='1980-01-01T00:00:00'

infiles=$(ls TAMSAT_*.nc)

for f in $infiles
do
outfile="${f%_*}_remap.nc"

last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

if [ $last_t == $wrong_time ]; then
        cmd_deltime="-seltimestep,1/-2" 
    fi

cdo -setreftime,1980-01-01,00:00:00,1days \
        -setcalendar,365_day \
        -remapbil,GRID_CHIRTS.txt \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile
done

But it only saved one file: TAMSAT_remap.nc. This was for the year 2015.. any idea how to get it to save individually processed files for each year?

Thanks so much for this help!

Jonathan

What does the command returns in a terminal?

ls TAMSAT_*.nc

It returns the following:

$ ls TAMSAT_*.nc
TAMSAT_1986.nc TAMSAT_1989.nc TAMSAT_1992.nc TAMSAT_1995.nc TAMSAT_1998.nc TAMSAT_2001.nc TAMSAT_2004.nc TAMSAT_2007.nc TAMSAT_2010.nc TAMSAT_2013.nc TAMSAT_remap.nc
TAMSAT_1987.nc TAMSAT_1990.nc TAMSAT_1993.nc TAMSAT_1996.nc TAMSAT_1999.nc TAMSAT_2002.nc TAMSAT_2005.nc TAMSAT_2008.nc TAMSAT_2011.nc TAMSAT_2014.nc
TAMSAT_1988.nc TAMSAT_1991.nc TAMSAT_1994.nc TAMSAT_1997.nc TAMSAT_2000.nc TAMSAT_2003.nc TAMSAT_2006.nc TAMSAT_2009.nc TAMSAT_2012.nc TAMSAT_2015.nc

Hm, that is strange, it should find and use all TAMSAT files as listed.

Insert a command line after the infiles definition 'echo $infiles; exit' to see if the infiles variables contains the file names. Or call the script with the -xv options.

Just a further note: change the infiles assignment to

infiles=$(ls TAMSAT_*[0-9]*.nc)

to not do the computing with TAMSAT_remap.nc when it already exists.

Ah, I see the failure! The outfile definition is wrong, you have changed it to

outfile="${f%_*}_remap.nc" 

that means that all output files have the same name TAMSAT_remap.nc!

Change it to the following to get for each single infile a single outfile use

outfile="${f%.*}_remap.nc" 

Hi Karin,

Ah yes you are correct! Thanks for pointing that out, and for your support with helping navigate my request. It is now saving individual files!

Best regards

Jonathan

Hi Karin,

So sorry, again having an issue. Tried to use exact same loop for my Tmin files, and it came back with this:

40314820@BIO2-147749 /bin/rfe1983
$ wrong_time='1980-01-01T00:00:00'

40314820@BIO2-147749 /bin/rfe1983
$

40314820@BIO2-147749 /bin/rfe1983
$ infiles=$(ls Tmin.*.nc)

40314820@BIO2-147749 /bin/rfe1983
$

40314820@BIO2-147749 /bin/rfe1983
$ for f in $infiles

do
outfile="${f%.*}_remap.nc"

last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

if [ $last_t == $wrong_time ]; then
cmd_deltime="-seltimestep,1/-2"
fi

cdo -setcalendar,365_day \
${cmd_deltime} \
-del29feb \
$f $outfile
done

cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

☻cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

I was using the below:

#!usr/bin/env
wrong_time='1980-01-01T00:00:00'

infiles=$(ls Tmin.*.nc)

for f in $infiles
do
outfile="${f%.*}_remap.nc"

last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

if [ $last_t == $wrong_time ]; then
        cmd_deltime="-seltimestep,1/-2" 
    fi

cdo -setcalendar,365_day \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile
done

infiles=$(ls Tmin.*.nc)

for f in $infiles
do
outfile="${f%.*}_remap.nc"

cdo -setcalendar,365_day \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile
done

I shortened to above as this is all that needs to be done for this file. Error is:

cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

cdo(2) seltimestep: Process started
cdo(3) del29feb
: cdo(2) seltimestep (Abort): Process startedNo timestep selected!

etc.

Jonathan

#!/usr/bin/env bash

wrong_time='1980-01-01T00:00:00'

infiles=$(ls Tmin.*.nc)    #-- take care of the dots in file names, should be used for extensions only

for f in $infiles
do
    outfile="${f%.nc}_remap.nc"    #-- take care of the dots in file names, should be used for extensions only

    last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

    cmd_deltime=""   #-- reset cmd_deltime

    if [ $last_t == $wrong_time ]; then
        cmd_deltime="-seltimestep,1/-2" 
    fi

    cdo -setreftime,1980-01-01,00:00:00,1days \
        -setcalendar,365_day \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile

done

This code works with the files containing leap years, but it is not deleting the 366th '1980-01-01 00:00:00' on all the other .nc files? Strange because I think copied and pasted the exact code that worked for my TAMSAT files and it didnt work for the Tmin files..

Attached are some of the files, in case this helps with figuring what I am doing wrong.

Note I tried removing the 'dots' from the filename which is why they now don't have them but it didnt make a difference

Bash script run.sh:

#!/usr/bin/env bash

wrong_time='1980-01-01T00:00:00'

infiles=$(ls Tmin*.nc)

for f in $infiles
do
    echo "$f\n----------------------------------" 
    outfile="${f%.nc}_remap.nc" 

    last_t=$(cdo -s -showtimestamp -seltimestep,-1 $f | sed -e 's/ //g')

    cmd_deltime="" 

    if [ $last_t == $wrong_time ]; then
        cmd_deltime="-seltimestep,1/-2" 
        echo "---- $f  -- $cmd_deltime" 
    fi

    cdo -setreftime,1980-01-01,00:00:00,1days \
        -setcalendar,365_day \
        ${cmd_deltime} \
        -del29feb \
        $f $outfile
done

Output:

bash run.sh

Tmin1986.nc\n----------------------------------
---- Tmin1986.nc  -- -seltimestep,1/-2
cdo(1) setcalendar: Process started
cdo(2) seltimestep: Process started
cdo(3) del29feb: Process started
cdo(3) del29feb (Warning): Day 29feb not found!
cdo(3) del29feb: Processed 35136000 values from 1 variable over 366 timesteps.
cdo(2) seltimestep: Processed 35040000 values from 1 variable over 366 timesteps.
cdo(1) setcalendar: Processed 35040000 values from 1 variable over 365 timesteps.
cdo    setreftime: Processed 35040000 values from 1 variable over 365 timesteps [0.25s 58MB].
Tmin1987.nc\n----------------------------------
---- Tmin1987.nc  -- -seltimestep,1/-2
cdo(1) setcalendar: Process started
cdo(2) seltimestep: Process started
cdo(3) del29feb: Process started
cdo(3) del29feb (Warning): Day 29feb not found!
cdo(3) del29feb: Processed 35136000 values from 1 variable over 366 timesteps.
cdo(2) seltimestep: Processed 35040000 values from 1 variable over 366 timesteps.
cdo(1) setcalendar: Processed 35040000 values from 1 variable over 365 timesteps.
cdo    setreftime: Processed 35040000 values from 1 variable over 365 timesteps [0.28s 58MB].
Tmin1988.nc\n----------------------------------
cdo(1) setcalendar: Process started
cdo(2) del29feb: Process started
cdo(2) del29feb: Processed 35040000 values from 1 variable over 366 timesteps.
cdo(1) setcalendar: Processed 35040000 values from 1 variable over 365 timesteps.
cdo    setreftime: Processed 35040000 values from 1 variable over 365 timesteps [0.28s 58MB].
Tmin1989.nc\n----------------------------------
---- Tmin1989.nc  -- -seltimestep,1/-2
cdo(1) setcalendar: Process started
cdo(2) seltimestep: Process started
cdo(3) del29feb: Process started
cdo(3) del29feb (Warning): Day 29feb not found!
cdo(3) del29feb: Processed 35136000 values from 1 variable over 366 timesteps.
cdo(2) seltimestep: Processed 35040000 values from 1 variable over 366 timesteps.
cdo(1) setcalendar: Processed 35040000 values from 1 variable over 365 timesteps.
cdo    setreftime: Processed 35040000 values from 1 variable over 365 timesteps [0.30s 58MB].
Tmin1990.nc\n----------------------------------
---- Tmin1990.nc  -- -seltimestep,1/-2
cdo(1) setcalendar: Process started
cdo(2) seltimestep: Process started
cdo(3) del29feb: Process started
cdo(3) del29feb (Warning): Day 29feb not found!
cdo(3) del29feb: Processed 35136000 values from 1 variable over 366 timesteps.
cdo(2) seltimestep: Processed 35040000 values from 1 variable over 366 timesteps.
cdo(1) setcalendar: Processed 35040000 values from 1 variable over 365 timesteps.
cdo    setreftime: Processed 35040000 values from 1 variable over 365 timesteps [0.29s 58MB].

I tried to do this by using:

bash run.sh

using the attached file and it returned this error:

$ bash run.sh
run.sh: line 2: $'\r': command not found
run.sh: line 4: $'\r': command not found
run.sh: line 6: $'\r': command not found
run.sh: line 8: syntax error near unexpected token `$'do\r''
'un.sh: line 8: `do

There is something wrong after your copy. You have to delete the \r from each line:

cat run-1.sh | tr -d '\r' > run-2.sh