Understanding Probabilistic Forecast Verification in Meteorology

1. Probability Forecasts and Verification

2. Uncertainty • Forecasts have uncertainty that should be taken into account: • Observational Uncertainty (initial conditions) • Instrument errors, bias in frequency of measurements, representativeness, reporting errors, random errors, precision errors, conversion errors … • Model Uncertainty • Insufficient spatial resolution, truncation errors, parameterization errors, coding errors, initialization errors …

3. Deterministic and Probabilistic Forecasts • Deterministic Forecast: Specific Outcome is Predicted • Probabilistic Forecast: Range of Possible Outcomes Accompanied by Confidence Estimate

4. Ex: Tercile Probabilities

5. Probability Forecast

10. Probabilistic Verification • Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast • Compare Forecast Distribution to a Distribution of Observations • Rank Probability Score (RPS) • Rank Probability SkillScore (RPSS) • Relative Operating Characteristics (ROC) • Reliability (Not Really Verification)

11. Rank Probability Score

13. Rank Probability Skill Score

14. NMME July 1 start DJF prec forecast RPSS CFSv2

15. Relative Operating Characteristics

16. Relative Operating Characteristics Hit Rate = hits/(hits+misses) False Alarm Rate = false alarms/(hits+false alarms)

17. Relative Operating Characteristics For Each Threshold Build a Contingency Table Summing Over All Forecasts

18. Relative Operating Characteristics

20. Reliability • Prediction System is Reliable if Predicted Probabilities Agree with Observed Frequencies • Ex: Climatological Probability has Perfect Reliability (33%, 33%, 33%) • Reliability Diagram Measure whether the Prediction System is Reliable, Overconfident or Underconfident • Reliability Diagram also Shows Resolution and Sharpness

21. Reliability Diagram

22. Reliability Diagram

28. T2m October Starts JFM Verification

29. T2m October Starts JFM Verification

30. Probabilistic Verification • Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast • Compare Forecast Distribution to a Distribution of Observations • Rank Probability Score (RPS) • Rank Probability SkillScore (RPSS) • Relative Operating Characteristics (ROC) • Reliability (Not Really Verification)