1 / 30

Probability Forecasts and Verification

Probability Forecasts and Verification. Uncertainty. Forecasts have uncertainty that should be taken into account: Observational Uncertainty (initial conditions)

murphy-eaton
Download Presentation

Probability Forecasts and Verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Probability Forecasts and Verification

  2. Uncertainty • Forecasts have uncertainty that should be taken into account: • Observational Uncertainty (initial conditions) • Instrument errors, bias in frequency of measurements, representativeness, reporting errors, random errors, precision errors, conversion errors … • Model Uncertainty • Insufficient spatial resolution, truncation errors, parameterization errors, coding errors, initialization errors …

  3. Deterministic and Probabilistic Forecasts • Deterministic Forecast: Specific Outcome is Predicted • Probabilistic Forecast: Range of Possible Outcomes Accompanied by Confidence Estimate

  4. Ex: Tercile Probabilities

  5. Probability Forecast

  6. Probabilistic Verification • Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast • Compare Forecast Distribution to a Distribution of Observations • Rank Probability Score (RPS) • Rank Probability SkillScore (RPSS) • Relative Operating Characteristics (ROC) • Reliability (Not Really Verification)

  7. Rank Probability Score

  8. Rank Probability Skill Score

  9. NMME July 1 start DJF prec forecast RPSS CFSv2

  10. Relative Operating Characteristics

  11. Relative Operating Characteristics Hit Rate = hits/(hits+misses) False Alarm Rate = false alarms/(hits+false alarms)

  12. Relative Operating Characteristics For Each Threshold Build a Contingency Table Summing Over All Forecasts

  13. Relative Operating Characteristics

  14. Reliability • Prediction System is Reliable if Predicted Probabilities Agree with Observed Frequencies • Ex: Climatological Probability has Perfect Reliability (33%, 33%, 33%) • Reliability Diagram Measure whether the Prediction System is Reliable, Overconfident or Underconfident • Reliability Diagram also Shows Resolution and Sharpness

  15. Reliability Diagram

  16. Reliability Diagram

  17. T2m October Starts JFM Verification

  18. T2m October Starts JFM Verification

  19. Probabilistic Verification • Requires Several Forecast Cases – Cannot Verify a Single Probabilistic Forecast • Compare Forecast Distribution to a Distribution of Observations • Rank Probability Score (RPS) • Rank Probability SkillScore (RPSS) • Relative Operating Characteristics (ROC) • Reliability (Not Really Verification)

More Related