timpctl not giving the expected results
Added by e ddc over 9 years ago
Hello
Just to let you know that timpctl is not giving the expected results.
The attached file (cdo_location.nc) contains a timeseries of "vas" at one specific location.
The percentile was calculated as below:
cdo -timmin cdo_location.nc min.nc
cdo -timmax cdo_location.nc max.nc
cdo -timpctl,99 cdo_location.nc min.nc max.nc ou.nc
and a ncdump ou.nc lets you know that 6.357008 was the value found for the percentile:
"""
data: lon = -127.613986951547 ; lat = 23.4235115115724 ; time = 42399.5 ; time_bnds = 35095, 35096 ; vas = 6.357008 ;
"""
One can verify this result by hand After sorting "vas" values. Let "sorted_vas" be the ordered version of "vas", then you can expect that the 99% percentile be "sorted_vas(index)" where index is 7305*99/100 = 7231.9. (Note that the initial time series has 7305 values). At this stage one can choose to use "sorted_vas(7232)=6.16" for the value of the percentile, even if "sorted_vas(7233)=6.21" might be a justifiable choice, or even an interpolation between those values. However, "timpctl,99" gives a value between "sorted_vas(7235)=6.32" and "sorted_vas(7236)=6.43" which is clearly larger than the sensible expectated value for this percentile.
This issue is also also apparent in ymonpctl.
Cheers,
eddc
| cdo_location.nc (4.89 MB) cdo_location.nc |
Replies (1)
RE: timpctl not giving the expected results - Added by Uwe Schulzweida over 9 years ago
Thanks for your post! Your are right the default precision of the percentile operators is not exact. CDO uses an algorithm with histograms. This algorithm is used because to be flexible in memory requirements. CDO operators are designed to process very large data sets on high resolution and many timesteps. You can increase the precision by setting the envirionment variable CDO_PCTL_NBINS to a large number, the default is 101. Using a value of 1001 will increase the precision and the memory requirement by a factor of 10.
export CDO_PCTL_NBINS=1001 cdo timpctl,99 ifile ofile