1 / 48

Probabilistic Forecast Verification

Probabilistic Forecast Verification. Allen Bradley IIHR Hydroscience & Engineering The University of Iowa. RFC Verification Workshop 16 August 2007 Salt Lake City. Advanced Hydrologic Prediction Service. Ensemble streamflow forecasts. Advanced Hydrologic Prediction Service.

tale
Download Presentation

Probabilistic Forecast Verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Probabilistic Forecast Verification Allen Bradley IIHR Hydroscience & Engineering The University of Iowa RFC Verification Workshop 16 August 2007 Salt Lake City

  2. Advanced Hydrologic Prediction Service • Ensemble streamflow forecasts

  3. Advanced Hydrologic Prediction Service • Ensemble streamflow forecasts • Multiple forecast locations

  4. Advanced Hydrologic Prediction Service • Ensemble streamflow forecasts • Multiple forecast locations • Throughout the United States

  5. AHPS Verification Forecast Location How good are the ensemble forecasts produced by AHPS? Forecast Date Forecast Variable

  6. Outline • Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecasts • Describe a prototype interactive web-based system for implementing this verification framework within an RFC • Present a future vision for the role of verification archives in AHPS forecasting 6

  7. Forecast Verification Framework

  8. Perspective: Forecast Users • Evaluate the quality of forecasts at a specific location for a particular forecast variable and date Examine one “element” in the data cube 8

  9. Elemental Problem • Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow threshold • Forecast quality attributes quantified over a range of flow thresholds 9

  10. Ensemble Forecast

  11. Ensemble Forecast y f • Probability forecast of a discrete event

  12. Wet Near Avg Dry fdry favg fwet Ensemble Forecast • Probability forecast of a discrete event • Probability forecasts of multicategory events 12

  13. y0.75 y0.50 y0.25 f0.25 f0.50 f0.75 Ensemble Forecast • Generalize by defining event forecasts as a continuous function of threshold Index function by the threshold’s climatological probability 13

  14. Ensemble Forecast Verification • Compute forecast-observations pairs for specific thresholds yp • Evaluate forecast quality for a range of thresholds yp

  15. Des Moines River near Stratford • Skill depends on the threshold • Uncertainty is greater for extremes Skill Standard Errors April 1st Forecasts 15

  16. Distributions-Oriented Measures • Skill Score Decomposition: (SS) Skill (RES) Resolution (CB) Conditional Bias (UB) Unconditional Bias Potential Skill Slope Reliability Standardized Mean Error

  17. Eliminate with bias- correction Implications for Verification SS RES CB UB Increase Probability forecast skill Minimum 7-Day Flow 17

  18. Perspective: NWS RFC Forecaster • Assess the overall performance of the forecasting system • Diagnose attributes limiting forecast skill (e.g., biases) Examine “slices” and “blocks” of the data cube 18

  19. Multidimensional Problem • The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different dates • Summary measures describe attributes of the skill functions derived from the elemental verification problem 19

  20. Summary Verification Measures • Ranked Probability Skill Score (RPSS): Weighted-average skill over probability thresholds

  21. RPSS Center of Mass Summary Verification Measures • RPSS shows average skill • Center of mass shows asymmetries in the skill function Skill 21

  22. Hypothetical Skill Functions • All skill functions have same average skill 22

  23. Hypothetical Skill Functions • All skill functions have same average skill • Second central moment shows shape 23

  24. Hypothetical Skill Functions • All skill functions have same average skill • Second central moment shows shape 24

  25. NCRFC Forecasts • 7-day minimum flow forecasts for mainstem locations for three rivers Minnesota River (MIN) Des Moines River (DES) Rock River (RCK) 25

  26. Forecast Skill Attributes • Forecasts made at the 1st of the month 26

  27. Forecast Skill Attributes • Average skill is highest for DES • The skill function is peaked in the middle 27

  28. Summary Measure Decomposition • Skill Score Decomposition: Skill Resolution Conditional Bias Unconditional Bias Weighted-average measures of resolution and biases

  29. AHPS Minimum 7-Day Flow • A single MIN site has large biases for low flows • The largest biases for other sites centered on higher flows 29

  30. Forecast Bias Attributes • Unconditionalbias is dominate • Simple bias-correction can significantly improve forecasts Simple bias- correction Post-hoc calibration 30

  31. Verification Framework • Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability) • Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality function • This interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts 31

  32. AHPSVerificationSystem

  33. AHPS Verification System Web-based tools for online access, analysis, and comparison of retrospective AHPS forecasts for River Forecast Centers (RFCs) http://www.iihr.uiowa.edu/ahps_ver

  34. Map-Based Navigation 3 1 2

  35. Verification Data Archive 35

  36. Verification Data Archive • Retrospective forecasts for a 50-year period 36

  37. Verification Data Archive • Retrospective forecasts for a 50-year period • Processed ensemble forecasts & observations 37

  38. Verification Data Archive • Retrospective forecasts for a 50-year period • Processed ensemble forecasts & observations • Verification results 38

  39. Verification System Concepts • Retrospective ensemble traces available in their native format (*.VS files) • Processed ensemble forecasts & observations for a suite of variables • Uses *.qme files from the calb system • Forecast quality measures based on the ensemble forecasts 39

  40. Disk Requirements • 6 Forecast periods per month (72 per year) • All segments have 50 years observed record

  41. Advantages • Interactive exploration of verification results • Provides a diagnostic “report card” for sites within an RFC • Instant access to forecasts and quality measures for verification sites • Seamless integration with other components of the NWSRFS system 41

  42. A Vision for theFuture

  43. Vision Generation and archival of retrospective forecasts will be a routine component of forecasting systems Verification methods can assess quality Verification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting system Archival information will form the basis for generating improved forecast products

  44. Product Generation with Archive Optimized CS • Raw ESP forecast • Archive verification indicates biases and skill • Optimal merging and bias correction Verification Archive Ensemble Forecast

  45. Conclusions

  46. Conclusions A consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”) AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFC Retrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation

  47. DESI4 Total Bias STRI4 JCKM5 Des Moines Forecast Skill • Skill is higher (lower) downstream (upstream) • Skill decline from April to June

More Related