CDO

Tufan Turp wrote:

Hi all,

I'm trying to calculate the index of wet days with respect to 90th percentile of reference period (ECAR90P). For the reference period I have a 20-year daily dataset covering 1986-2005 period. It means my timestep is 7305. Firstly, I calculated the daily precipitation amount at wet days by using the following commands:

cdo mul input.nc -gec,1 input.nc wet_input.nc

cdo ifthen -gtc,0 wet_input.nc wet_input.nc wet_days_precipitation.nc

And then,

export CDO_PCTL_NBINS=1001

cdo ydaypctl,90 wet_days_precipitation.nc -ydaymin wet_days_precipitation.nc -ydaymax wet_days_precipitation.nc pctl90.nc

During this step it was killed!

My input file has a 25-km grid space for whole globe and since it is a huge file I cannot share here. My PC is has a 8 GB memory.

Thanks for your suggestions.

Best,

Tufan

Hi Tufan,

without the data it's not possible to say what's going wrong. Is it possible to get the input file?

-Karin

Dear Karin,

First of all, thank your for your quick reply. My input file is 30.3 GB since it includes 25-km resolution global daily precipitation data. I thought it might happen because of memory size of the PC. This is why I tried to change default bin value from 101 to 1001.

Tufan

I've tried to reproduce your problem but got a different error message caused by different number of timesteps for input.nc and pctl90.nc.

To use the eca_r90p operator the following works:

cdo -mul input.nc -gec,1 input.nc wet_input.nc

cdo -ifthen -gtc,0 wet_input.nc wet_input.nc wet_days_precipitation.nc

export CDO_PCTL_NBINS=1001

cdo -daypctl,90 wet_days_precipitation.nc \
    -daymin wet_days_precipitation.nc \
    -daymax wet_days_precipitation.nc \
     day_pctl90.nc

cdo -eca_r90p wet_input.nc day_pctl90.nc outfile_daypctl90.nc

-Karin

Dear Tufan,

let me give you some comments since I work on the climate extreme indices of the CDOs:

- Note that if you want to calculate a time series, e.g. yearly values of eca_r90p, starting inside the reference period, you should use another software. I recommend the climdex package. See: http://slides.com/wachsylon/cdoetccdi#/1/9 for more infos.

- Otherwise (if you want to calculate a time series which starts outside the reference period or if you try to calculate only one value) you can use CDOs. Note that if you apply

cdo ydaypctl

on a 20yr time series, only 20 values are taken for each day to calculate the 90th percentile. Since this is a small subset, you could extend it by defining a running window so that more values are taken into account for each day. The corresponding CDO operator is

cdo ydrunpctl

which additionally requires a window parameter.

Here is an idea to solve your problem:
Your NBINS variable

export CDO_PCTL_NBINS=1001

is too big. Your sample size per grid cell is 20 since you only have 20 values per each day. The operator starts to use BINs (instead of the actual values of the sample) for calculating the percentile if your sample size is about at least half of the maximal nbins depending on your system. Therefore, it should not make a difference in the calculation whether you set
export CDO_PCTL_NBINS=80 or
export CDO_PCTL_NBINS=160 or
export CDO_PCTL_NBINS=1001
however it will make a difference in the allocation of memory in your system.

Best regards,
Fabi

Dear Karin and Fabi,

Many thanks to both of you.

@Karin:
-------

Yes, it is an expected error during the step of eca_r90p command. If I could pass the first steps, I would encounter the same error because of the algorithm behind CDO.

@Fabi:
-------

I'm glad to receive a reply from you. I already know CLIMDEX and I've already read your slides before. Frankly speaking, the comparison of CDO and CLIMDEX packages for calculation of the core climate extreme indices was also in my to do list till I've seen your research I think it is a good effort and really beneficial to better understand of mentality of these packages. So thank you very much again! CDO is a great package for us and it is getting better and better day by day. But I think the extreme indices module is not so clear and useful, especially for the percentile indices. Actually, I have many questions But we may have chance talk about them in details soon.

Now, if we come to my current problem:

Actually, ydaypctl has never made a sense for me. I tried it since it is suggested in CDO user guide. I was planning to apply all possible calculation and compare the results to find scientifically information and also understand the approach of CDO.

I also tried ydrunpctl command before, because I think it is the command what I am looking for. After your suggestion I tried again and got the following error:

cdo ydrunpctl (Abort): Couldn't read data from input stream (pipe1.2)!
ERROR, namespaceSetActive, namespace.c, line 223
errorString: "assertion `(unsigned)nId < namespacesSize && namespaces[nId].resStage != STAGE_UNUSED` failed"

• Error in `cdo': double free or corruption (fasttop): 0x0000000002b17c00 ***
  Aborted (core dumped)

Here I have two questions about ydrunpctl command:

1. Should I use the precipitation amount at wet days data or merely daily precipitation data?

2. My timestep is 7305. What should I enter for the timestep when the screen asks?

cdo(2) ydrunmin : Enter number of timesteps > cdo(3) ydrunmax : Enter number of timesteps >

What kind of distribution is assumed by CDO for percentile calculations? Gamma or it is free of distribution? I guess it is basically a percentile rank.

Thank you very much Karin & Fabi.

Best,

Tufan