Project

General

Profile

insert leap day

Added by Noelia Otero over 8 years ago

Hi everyone!

I have model output with work with a 365-calendar. I would like to ask if there is any way to add the "02-29" and put a missing value, in those years that are leap days but not considered by the model.
I've read in an older post that it can be remove by doing something like: cdo delete,month=2,day=29 ifile ofile.. so I was trying to do something similar ..like :cdo setmissval,-999,month=2,day=29 (I know this is not correct).
Any idea if this would be possible?

Many thanks in advance

Noelia


Replies (15)

RE: insert leap day - Added by Jaison-Thomas Ambadan over 8 years ago

I think, you could use setmon and mergetime to do this task. For example,

cdo -setmon,2 -selday,29 -selmon,1 ifile.nc feb29.nc

will extract the 29th day of January and convert that to 29th February. [To set the missing value, use setrtomiss on feb29.nc]. Now you can use the mergetime to merge that to original data

cdo -mergetime ifile.nc feb29.nc new_ifile.nc

RE: insert leap day - Added by Jaison-Thomas Ambadan over 8 years ago

Hi again, if your input file is multi-year then, the above steps create extra 29th feb for all years including non-leap years, so that could be a problem - you could extract 29th feb from a multi year reference data set and merge that to your model data.

RE: insert leap day - Added by Noelia Otero over 8 years ago

Hi,

Many thanks for your idea, I can try it. Although I am not sure how use "setrtomiss" here...I mean, should I apply something like this:
cdo setrtomiss,-999, feb29.nc newfew29.nc ? and then merge to the original one ?
My other problem, as you mentioned is that I have multiple years, so I am afraid that it is a problem, and I should do another step to take care of that.

Thanks a lot!
cheers

RE: insert leap day - Added by Poulomi Ganguli over 4 years ago

Hello,

I tried this code but instead of giving me date as yy-2-29 the time dimension, I am getting is of 1st March of the particular year.

RE: insert leap day - Added by Utkarsh Verma about 3 years ago

Hi,

I have 30 year modeled data and I was trying for 5 year where 2 leap year are there

cdo -setday,29 -setmon,2 -setyear,1988 -fldavg -seldate,1988-02-28,1988-03-01 tt_88-92_ctl2nc.nc tt_dummy.nc

but after doing this there is 3 files created as folows:

File format : NetCDF
-1 : Institut Source T Steptype Levels Num Points Num Dtype : Parameter name
1 : unknown unknown v instant 1 1 1 1 F32 : pr
Grid coordinates :
1 : lonlat : points=1 (1x1)
lon : 0 degrees_east
lat : 0 degrees_north
Vertical coordinates :
1 : generic : levels=1
z : 0 level
Time coordinate : 3 steps
RefTime = 1988-01-02 00:00:00 Units = days Calendar = standard
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
1988-02-29 00:00:00 1988-02-29 00:00:00 1988-02-29 00:00:00

I guess this is wrong output, also please answer the question asked by Noelia which I am quoting here
"Hi,

Many thanks for your idea, I can try it. Although I am not sure how use "setrtomiss" here...I mean, should I apply something like this:
cdo setrtomiss,-999, feb29.nc newfew29.nc ? and then merge to the original one ?
My other problem, as you mentioned is that I have multiple years, so I am afraid that it is a problem, and I should do another step to take care of that.

Thanks a lot!
cheers"

Thanks,

Utkarsh

RE: insert leap day - Added by Karin Meier-Fleischer about 3 years ago

Hi Utkarsh,

please, upload the input file, it is hard to say what is going wrong without it.

RE: insert leap day - Added by Utkarsh Verma about 3 years ago

Hi Karin,

Here is the input file for one year which is leap year.
please help me with handling for large time scale data where non leap years will also be there and other leap years.

Thanks,

Utkarsh

cdo_sample.nc (77.1 MB) cdo_sample.nc this is sample daily data for year 1988(leap year)

RE: insert leap day - Added by Brendan DeTracey about 3 years ago

If you search the forums for leap day and persistently make it to the second page of search results, you will find the following: https://code.mpimet.mpg.de/boards/2/topics/8634?r=8722#message-8722

RE: insert leap day - Added by Brendan DeTracey about 3 years ago

Plus your files are set to calendar=365_day. The link I posted shows that after creating a leap day and stuffing it in, your last step must be using the cdo setcalendar command to change from a 365 day calendar to a standard calendar:

$ cdo setcalendar,'standard' in.nc out.nc

RE: insert leap day - Added by Utkarsh Verma about 3 years ago

Hi Brendan,

I have one doubt

In the bash script you have used I don't have to do the time reference part so, excluding that my doubt is if I have insert the data by averaging the data in 28th feb and 1st march of leap year day, how to insert this computation in the given bash script by you?

for year in $years; do
echo "$year"
if [ $(isleap "$year") -eq 0 ] ; then
echo "$year"
cdo -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,"$year" "$infile" -selday,1 -selmon,3 -selyear,"$year" "$infile" tmp."$year".nc
fi
done
cdo mergetime "$infile" 'tmp.????.nc' tmp.nc
cdo setcalendar,'standard' tmp.nc "$outfile"

Thanks,
Utkarsh

RE: insert leap day - Added by Utkarsh Verma about 3 years ago

Hi Brendan,

I tried running the code which you provided as given below:

infile_bad='pr_day_CMCC-CM2-SR5_historical_r1i1p1f1_gn_19850101-20141231.nc'
infile='CMCC_1985-2014_365_day.nc'
outfile='CMCC_1985-2014_leapyear.nc'

isleap() { date -d $1-02-29 &>/dev/null && echo 0 || echo 1 ;}

cdo -setreftime,1979-01-01,00:00:00,days "$infile_bad" tmp1.nc
ncatted -a calendar,time,m,c,365_day tmp1.nc tmp2.nc
cdo -a copy tmp2.nc "$infile"
#\rm tmp1.nc tmp2.nc

#isleap() { date -d $1-02-29 &>/dev/null && echo 0 || echo 1 ;}

years=$(cdo -s showyear "$infile")

for year in $years; do
echo "$year"
if [ $(isleap "$year") -eq 0 ] ; then
echo "$year"
cdo -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,"$year" "$infile" -selday,1 -selmon,3 -selyear,"$year" "$infile" tmp."$year".nc
fi
done
cdo mergetime "$infile" 'tmp.????.nc' tmp.nc
cdo setcalendar,'standard' tmp.nc "$outfile"
#\rm tmp.nc
#\rm tmp.????.nc

I have tried without making my data to 365_day but in both case I am getting following error:

lp_year.sh: line 19: 45248 Segmentation fault (core dumped) cdo -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,"$year" "$infile" -selday,1 -selmon,3 -selyear,"$year" "$infile" tmp."$year".nc

lp_year.sh: line 19: 45267 Bus error (core dumped) cdo -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,"$year" "$infile" -selday,1 -selmon,3 -selyear,"$year" "$infile" tmp."$year".nc

cdf_get_vara_double: ncid = 65536 varid = 1

Error (cdf_get_vara_double): NetCDF: HDF error
lp_year.sh: line 19: 45288 Bus error (core dumped) cdo -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,"$year" "$infile" -selday,1 -selmon,3 -selyear,"$year" "$infile" tmp."$year".nc

RE: insert leap day - Added by Brendan DeTracey about 3 years ago

Yes, that post was just an example. This is what works in term of commands, without scripting for batch processing.
$ cdo  -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,1988 cdo_sample.nc -selday,1 -selmon,3 -selyear,1988 cdo_sample.nc tmp1.nc
cdo(1) setmon: Process started
cdo(2) divc: Process started
cdo(3) add: Process started
cdo(4) selday: Process started
cdo(5) selmonth: Process started
cdo(6) selyear: Process started
cdo(7) selday: Process started
cdo(8) selmonth: Process started
cdo(9) selyear: Process started
cdo(1) setmon (Warning): Time bounds unsupported by this operator, removed!
cdo(9) selyear: Processed 20183040 values from 1 variable over 365 timesteps.
cdo(8) selmonth: Processed 1714176 values from 1 variable over 365 timesteps.
cdo(7) selday: Processed 55296 values from 1 variable over 31 timesteps.
cdo(6) selyear: Processed 20183040 values from 1 variable over 365 timesteps.
cdo(5) selmonth: Processed 1548288 values from 1 variable over 365 timesteps.
cdo(4) selday: Processed 55296 values from 1 variable over 28 timesteps.
cdo(3) add: Processed 110592 values from 2 variables over 2 timesteps.
cdo(2) divc: Processed 55296 values from 1 variable over 1 timestep.
cdo(1) setmon: Processed 55296 values from 1 variable over 1 timestep.
cdo    setday: Processed 55296 values from 1 variable over 1 timestep [0.25s 50MB].
$ cdo -a mergetime cdo_sample.nc tmp1.nc tmp2.nc
cdo    mergetime: Processed 20238336 values from 2 variables over 366 timesteps [3.70s 56MB].
$ cdo setcalendar,'standard' tmp2.nc cdo_sample_with_leap_day.nc                                           
cdo    setcalendar: Changing absolute to relative time axis!
cdo    setcalendar: Processed 20238336 values from 1 variable over 366 timesteps [0.30s 57MB].

The main issue is dealing with the calendar. The 365 day calendar will interfere with inserting a leap day.
  1. Calculate the leap day with simple average. Write to a tmp file.
  2. Mergetime the original file with the tmp file to insert the leap day. Note the -a option which forces an absolute time axis. This is very important. Write to another tmp file.
  3. Set the calendar to 'standard' and write to final file. This also converts back to a relative time axis.
    Just tried it and you can boil it down to a single command, no tmp files:
    $ cdo -a setcalendar,'standard' -mergetime cdo_sample.nc -setday,29 -setmon,2 -divc,2 -add -selday,28 -selmon,2 -selyear,1988 cdo_sample.nc -selday,1 -selmon,3 -selyear,1988 cdo_sample.nc cdo_sample_with_leap_day.nc

RE: insert leap day - Added by Brendan DeTracey about 3 years ago

Ack, no edit button. DO NOT use the final example. It does not work as the conversion to an absolute time axis must follow the creation of the leap day.

RE: insert leap day - Added by Utkarsh Verma about 3 years ago

Hi Bredan,
Thank you for helping in detail.

I tried your earlier code it worked and actually the Bus error was due to multiple cdo processes in a single shot which was solved by putting -L after cdo.

Also, without using conversion to 365_days calendar (like in your bash script) I was getting same input file which I have given initially but gives desired file using conversion 365_days calendar and providing reference point.

Now, it's working fine.

cheers!
Utkarsh

RE: insert leap day - Added by Brendan DeTracey about 3 years ago

Good to hear. :)

    (1-15/15)