Project

General

Profile

eca_r90tot gives only fill values

Added by Micah Wilhelm almost 12 years ago

I'm trying to do an extreme value analysis of global precipitation data from ECHAM6 using the following command:
cdo eca_r90ptot pr_data pr_ref_period_data output

I get an output consisting entirely of fill values (1e+20). If I use the function eca_r90p, I get an output with '_'..

Is the 90th percentile of the reference data calculated within the function? If not, is there a way to do this in one step (avoiding the memory allocation problems associated with timpctl ) ? It seems strange that the percentile wouldn't be calculated automatically since there are functions like eca_r75p, eca_r95p, eca_r99p with the percentiles right in the name..

Anyway, I'm very new to cdo and there are no examples in the documentation.. So I would appreciate any help you can offer :)

Cheers,
Micah


Replies (13)

RE: eca_r90tot gives only fill values - Added by Jaison-Thomas Ambadan almost 12 years ago

Hi Micah,

Can you please upload a sample of your data? If the file size is > 50MB, you may select a very small geographical region using "cdo -sellonlatbox".

In the meantime, setting the fill/missing value before doing the percentile may help to get the correct answer:

cdo -eca_r90ptot -setmissval,1e20 pr_data -setmissval,1e20 pr_ref_period_data output

Cheers,
J.

RE: eca_r90tot gives only fill values - Added by Micah Wilhelm almost 12 years ago

Hi Jaison,

Thanks for the fast reply! I tried your suggestion, but still no luck. Here is a sample of my data. I checked it out with ncview, and it seems to be alright :/

Thanks again for you help,
Micah

ref.nc (2.7 MB) ref.nc pr reference period data
data.nc (2.43 MB) data.nc pr data

RE: eca_r90tot gives only fill values - Added by Jaison-Thomas Ambadan almost 12 years ago

Hi Micah,

The operator "eca_r90ptot" works fine! BUT the problem is it is NOT supposed to operate on that type of data (data.nc). I'll explain

Here is the cdo help from: cdo -h eca_r90ptot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let ifile1 be a time series RR of the daily precipitation amount at wet days (precipitation >= 1 mm)
and ifile2 be the 90th percentile RRn90 of the daily precipitation amount at wet days for any period
used as reference. Then the ratio of the precipitation sum at wet days with RR > RRn90 to the total
precipitation sum is calculated.
RRn90 is calculated as the 90th percentile of all wet days of a given climate reference period.
Usually ifile2 is generated by the operator ydaypctl,90.
The date information of a timestep in ofile is the date of the last contributing timestep in ifile1.
The following variables are created:
- precipitation_percent_due_to_R90p_days
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The first line says that the daily precipitation values must be >=1mm

Now the data that you provided (data.nc), I guess, is the precipitation rate, and the maximum value is 0.001558 and minimum is zero

You can simply test the operator by multiplying your input data by say, 1000

ie, "cdo -eca_r90ptot -mulc,1000 data.nc -mulc,1000 ref.nc output" will give you the correct output value (NOT missing value)

so, I suggest you to calculate precipitation sum (daily/monthly etc) and then apply the operator. OR simply adjust the unit/scale and then apply the operator.

Hope this helps!

Cheers,
J.

RE: eca_r90tot gives only fill values - Added by Micah Wilhelm almost 12 years ago

Hi Jaison,
Thank you so much for the help! Such a silly mistake on my part.. units units units..
One last question: I would like the equivalent function for temperature. However, I can only find eca_tx90p which reports the relative number of days that exceed the 90th percentile. Can you think of a way to report the total number of days instead?

All the best,
Micah

RE: eca_r90tot gives only fill values - Added by Ralf Mueller almost 12 years ago

I'm not totally sure, if this could solve your problem, but if this is only about counting days (or timesteps in general) for which a certain value is greater/lesser than a given local threshold, this might be done with the consects operator (see #14). A short skeleton could look like this:

  • for values above the threshold
    cdo timsum -consects -gtc,0 -sub data.nc threshold.nc
  • for values below the threshold
    cdo timsum -consects -ltc,0 -sub data.nc threshold.nc

hth,
ralf

RE: eca_r90tot gives only fill values - Added by Suman B over 4 years ago

Hi,
Could you please give an example code on how to calculate the "ifile2 be the 90th percentile RRn90 of the daily precipitation amount at wet days for any period
used as reference"???

I have used the following one but the operator is not working

cdo ydaypctl,95 dayp.nc -ydaymin dayp.nc -ydaymax dayp.nc infile2_dap.nc

and for the infile1 I have used the following code

cdo -ifthen -gec,1 dayp.nc dayp.nc infile1_gth1mm.nc

Please let me know if these are correct

RE: eca_r90tot gives only fill values - Added by Karin Meier-Fleischer over 4 years ago

Hi Suman,

what do you mean with its not working?

From the ECA documentation:

To compute the percentage of wet days where the daily precipitation amount is greater than the 90thpercentile of the daily precipitation amount at wet days for a given reference period use:

cdo eca_r90p rrfile rrn90file outfile

The file dayp.nc has to be a time serie (see https://code.mpimet.mpg.de/projects/cdo/embedded/cdo_eca.pdf#subsection.2.0.16):

cdo -ydaypctl,90 dayp.nc -ydaymin dayp.nc -ydaymax dayp.nc dayp90.nc
cdo -gec,1 dayp.nc dayp_wet.nc
cdo eca_r90p dayp_wet.nc dayp90.nc outfile.nc

-Karin

RE: eca_r90tot gives only fill values - Added by Suman B over 4 years ago

I have tried your example codes but the error is the same---"cdo eca_r95p (Abort): "

could please the input files please.. I attach them if u want to check i am using CDO version 1.9.7.1

cdo -ydaypctl,95 P1986_2015.nc -ydaymin P1986_2015.nc -ydaymax P1986_2015.nc dayp95.nc
cdo -gec,1 P1986_2015.nc dayp_wet.nc
cdo eca_r95p dayp_wet.nc dayp95.nc out.nc

cdo eca_r95p (Abort):

RE: eca_r90tot gives only fill values - Added by Karin Meier-Fleischer over 4 years ago

Which version do you use (cdo -V)? Can you upload your input files.

RE: eca_r90tot gives only fill values - Added by Suman B over 4 years ago

Hi Karin,
I am using cdo version --1.9.7.1 (below is all details (cdo -V)
Climate Data Operators version 1.9.7.1 (http://mpimet.mpg.de/cdo)
System: x86_64-unknown-cygwin
CXX Compiler: g++ -g -Wall -O2 -DPIC -pie -mwindows -ftree-vectorize -fopenmp
CXX version : g++ (GCC) 7.4.0
C Compiler: gcc -g -Wall -O2 -DPIC -pie -mwindows -ftree-vectorize -fopenmp
C version : gcc (GCC) 7.4.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (GCC) 7.4.0
Features: 15GB 12threads C++14 Fortran DATA PTHREADS OpenMP45 HDF5 NC4/HDF5 OPeNDAP UDUNITS2 PROJ.4 SSE2
Libraries: HDF5/1.10.2 proj/5.2
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.7.1
cgribex library version : 1.9.3
ecCodes library version : 2.9.0
NetCDF library version : 4.6.1 of May 26 2018 18:07:01 $
hdf5 library version : 1.10.2
exse library version : 1.4.1
FILE library version : 1.8.3

And these are my input files dayp.c is the daily precipitation series. dayp_wet is the wet day precipitation above 1 mm and dayp95.nc is 95th percentile of daily precipitation

and here are the codes to derive the inputs from dayp.nc file
cdo -ydaypctl,95 dayp.nc -ydaymin dayp.nc -ydaymax dayp.nc dayp95.nc
cdo -gec,1 dayp.nc dayp_wet.nc
cdo eca_r95p dayp_wet.nc day95p.nc result.nc
#...but this code is throwing error.....
cdo eca_r95p dayp_wet.nc dayp95.nc result.nc

cdo eca_r95p (Abort):

Thank you very much..

Suman

dayp95.nc (2.58 MB) dayp95.nc 95th percentile precipitation obtained using the first line of the codes
dayp.nc (10.3 MB) dayp.nc daily precipitation
dayp_wet.nc (10.3 MB) dayp_wet.nc wet day precipitation above 1 mm obtained using the second line of the code

RE: eca_r90tot gives only fill values - Added by Suman B over 4 years ago

Hi,
any solution please?

RE: eca_r90tot gives only fill values - Added by Fabian Wachsmann over 4 years ago

Hi Suman,

there are 2 issues with the procedure you chose:

  1. The operator eca_r95p reads and compares timesteps of both input files in a loop so that both input files needs to have the same number of time steps.
    Right now, there is no other option to choose the output frequency of eca_r95p but by creating a loop around it: If you want to have monthly output, you split your timeseries and the percentile file into monthly chunks. If you however want to have the average over the entire time series, you need to enlarge the percentile file dayp95.nc for the time period of the timeseries. If you do so, consider that not every year is a leap year.
    A shell like framework for yearly output frequency could be:
    cdo delete,month=2,day=29 day95p.nc day95p_noleap.nc
    for year in timeseries; do
      if ( $year == leapyear ); then
        cdo eca_r95p -selyear,$year dayp_wet.nc day95p.nc result_${year}.nc
      else
        cdo eca_r95p -selyear,$year dayp_wet.nc day95p_noleap.nc result_${year}.nc
      fi
    done
    cdo mergetime result_* result.nc
    
  2. In your example files, you have a small number of values going into the calculation of the percentile. So even if eca_r95p works, it will produce a field of zeros because the 95th percentile of 4 values (4 years of the timeseries) will be the highest of these values and eca_r95p counts values that are higher than the 95th percentile.
    Depending on your timeseries, the ydrun* operators could be an alternative because they take into account a window of 5 days to enhance the histogram used for the percentile.

Best regards,
Fabi

RE: eca_r90tot gives only fill values - Added by Suman B over 4 years ago

Hi,
I am trying to run this code on cygwin. the following error is comming. Do I need to separately define the year?
cdo(1) selyear: Process started

cdo(1) selyear (Abort): Integer parameter >timeseries< contains invalid character at position 1!

    (1-13/13)