Creating probability forecasts of binary events from ensemble predictions and prior information - A comparison of method

1. Creating probability forecasts of binary events from ensemble predictions and prior information - A comparison of methods Ian Jolliffe, Chris A. T. Ferro and David B. Stephenson Climate Analysis Group Department of Meteorology University of Reading Cristina Primo Institute Pierre Simon Laplace (IPSL)

2. Outline: • How to improve probabilistic forecasts of a binary event? • - Use prior information: Bayesian methods. • - Calibrate the model: Logistic regression. • Illustration of methods with an example: • 3-day ahead precipitation in Reading (UK) • 5-month ahead forecast of Dec. Niño 3.4 Index • Conclusions

3. NOTATION: Numerical models provide an ensemble of forecasts for time : ensemble size 1 if the -th member forecast the event at time 0 otherwise Number of members that forecast the event. Aim: Forecast a binary event at a future time 1 if the event is observed to occur 0 otherwise

4. How to estimate the probability of the event?

5. How to estimate the probability of the event? (a) Temporal series Do not use ensemble forecasts.

6. How to estimate the probability of the event? (b) Frequentist approach Just use ensemble forecasts (do not use past observations).

7. How to estimate the probability of the event? (c) Bayesian approach Use ensemble forecasts and prior information about past data, expert opinion or a combination between them.

8. How to estimate the probability of the event? (d) Calibration approach Incorporate the relationship between past observations and past ensemble forecasts.

9. Frequentist approach • The probability is estimated by the relative frequency: • Easy to obtain. • When and , the forecaster issues probabilities of 0 and 1 (event completely impossible or completely certain to occur). • There is no estimate of uncertainty on the predicted probability. • The probabilities take only a finite set of discrete values. It is unlikely the forecaster really believes this statement !! = Probability that the event is observed to occur (unknown).

10. 2) Bayesian approach : provides us with a posterior distribution of the predicted probability. If the model is perfectly calibrated, the probability that an ensemble member forecast the event is also . Davison A. C.,Cambridge University Press (2003) • Estimate a distribution a priori including the uncertainty in the parameters: • Model uncertainty of the ensemble forecasts (likelihood) as a conditional distribution: • Obtain a posterior distribution (Bayes’ theorem):

11. 2) Beta approach Bayes´ theorem Posterior distribution Beta(0.5,0.5) Beta(1,1) Beta(2,2) Beta(10,20) Observations Forecasts likelihood choose a Prior distribution + p.d.f. Beta(a,b) But both a and b are unknown !! Katz and Ehrendorfer, Weather and Forecasting (2005).

12. How to choose the prior distribution? 1) a = b = 0 . This is equivalent to the frequentist approach. • 2) Calculate: • a central point: • a measure of the spread: = climatology Rajagopalan et al. (2002) methodis a particular case, where: = weight, = Number of past observations The weight gives different importance to prior belief and model forecasts and is chosen to minimize the logarithmic score.

13. 3) Calibration Technique: If is an explanatory variable calculated from the ensemble forecasts, then: Logistic regression The parameters of the logistic regression are calculated to maximize the likelihood. Predictor (given by the ensemble forecasts). Link Function Both unknown

14. Which explanatory variable to use? • Relative frequencies: • Logit transformation of the relative frequencies: • Include prior information: Roulston and Smith, Mon. Wea. Rev. (2002)

15. Summary of methods • Relative frequencies • 2) Beta approach • Rajagopalan et al. (2002) • 3) Logistic Regression

16. Example: Daily winter precipitation at Reading (UK) Forecasts: 3-day ahead 50-member forecasts of daily total precipitation from Ensemble Prediction System (EPS) at ECMWF for a grid point near Reading (UK) forecast Observations: total daily precipitation observed at the University of Readingatmospheric observatory. Binary event:precipitation above a threshold. Period: Dec-Jan-Feb from 1997 to 2006. n= 812 daily observations m x n = 50 x 812 = 40600 forecasts

17. Precipitation in Reading: 10 mm (perc.=97%, extreme event). 2 mm (perc= 75.6%) 0.1 mm (WMO def. of wet day) The model is not perfectly calibrated

18. The predicted probability can be expressed as a function of the frequency approach: F is a lineal function in the RLZ method and non linear in the Logistic Regression approach. Threshold=0.1mm Threshold=2mm Threshold=10mm

19. Brier Score: All the BS improve the frequencies one (a =0.05).

20. Example 2: Niño-3.4 SST index Forecasts: 5-month ahead 9-member ensemble forecasts of Niño 3.4 SST index from the coupled ECMWF model ( DEMETER). (Palmer et al. Bull. Am. Meteorol. Soc., 2004) Observations: Niño-3.4 SST index Binary event: Index above the median. Period: hindcasts for December started in the 1st of August of each year from 1958 to 2001. n = 44 observations m x n = 9 x 44 = 396 forecasts

21. Niño 3.4 SST index: Observations and Forecasts Observations Forecasts Years

22. Niño 3.4 SST index: Median 75% 90% 90% perc. 75% perc. median We calibrate the data when we codify them.

23. Brier Score:

24. Conclusions based on this example: - Use of prior information via the Beta distribution gives forecasts that have more skill than the frequentist ones - Calibration using logistic regression gives forecasts that have more skill than the frequentist ones - A combination of Beta technique and calibration improves each technique separately. Work is still necessary to choose the best predictor for the logistic regression and the best way to combine both techniques. c.primo@reading.ac.uk http://www.met.rdg.ac.uk/~sws05cp/

25. END