Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service

1. Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service Gary A. Wick NOAA Environmental Technology Laboratory On Rotational Assignment with the NWS Office of Hydrologic Development November 2003

2. Overview • Performance measures for the Advanced Hydrologic Prediction Service (AHPS) • Review of probabilistic forecast verification measures • Trial application with operational forecast data • Recommendations

3. Advanced Hydrologic Prediction Service (AHPS) AHPS Program Performance Existing Measures Future Measures Deployment Number of forecast points Coverage area Maturity, e.g. metrics addressing Usage of forecast information Probabilistic forecast effectiveness Science Projects Performance AHPS Science Activities Development Activity Effectiveness Contribution to forecast maturity Number of science tools deployed per year Contribution to information content

4. Probabilistic Forecast Verification • Categorical forecasts • Brier Score • Rank Probability Score (RPS)

5. Categorical Forecasts • Transforms probabilistic forecast into a categorical forecast through selection of a probability threshold • Simple but doesn’t fully address probability

6. Brier Score • Simple extension that fully characterizes probabilistic forecasts • Limited to occurrence of a specific event

7. Rank Probability Score • Extension characterizing full distribution of forecasts • Ideal as science measure but added complexity a concern at program level

8. Application of Accuracy Measures • Deterministic Measures • Probabilistic Brier Score

9. Deterministic Application • Used National Weather Service verification database • Monthly data for 177 sites starting April 2001 • Results computed “on-the-fly” • Evaluated accuracy difference between AHPS and non-AHPS points • Considered subset of points for the North Central, Ohio, and Missouri Basin River Forecast Centers • Expressed in terms of mean absolute error and root mean square error

10. Deterministic Results

11. Deterministic Results

12. Implications: Deterministic • Possible to implement something rapidly • Characterization must be defined • Existing verification database and interface inadequate

13. Brier Score Evaluation • Sample ensemble forecasts and verification provided by Kristie Franz • 43 sites from the Ohio River Forecast Center • 11 weekly mean and monthly maximum exceedance forecasts • Forecast traces, verification, and historical data • Evaluated accuracy of forecasts for exceedance of flood stage

14. Brier Score Evaluation • Use of all forecasts suggested very high accuracy • Use of all forecasts suggests very high accuracy • Only 17% improvement over forecasts for no flooding

15. Brier Score Evaluation • Most revealing results obtained for forecasts where flooding occurred

16. Rank Probability Score • Could interpret as accuracy from 76 to 91%

17. Implications: Probabilistic • Brier score can be presented simply and meaningfully • Application limited by constraint to instances of flooding • Rank probability score addresses all forecasts but meaning is harder to express • Necessary to regularly archive ensemble forecasts and verification

18. Recommendations • Performance measures • Data collection • Additional analyses

19. Recommended AHPS Accuracy Measures • Deterministic River Forecast Accuracy • Probabilistic River Forecast Accuracy • Flood Forecast Accuracy

20. AHPS Deterministic River Forecast Accuracy • Percent accuracy of mean daily streamflow for days 1-3 • Evaluates short-term hydrograph forecasts

21. AHPS Probabilistic River Forecast Accuracy • RPS derived accuracy of weekly mean streamflow exceedance • Evaluates AHPS weekly chance of exceedance forecasts • Express as percent accuracy • Apply to week 2 and week 4 forecasts

22. AHPS Flood Forecast Accuracy • Derived from Brier score and weekly maximum stage forecasts • Simple evaluation of weekly exceedance forecasts • Express as percent accuracy for cases where flooding occurred • Apply to week 2 and week 4 forecasts

23. Recommended Data Archival • Forecast Data • Ensemble forecast traces at selected points • Verification • Corresponding stage/streamflow observations • Historical Data • Ensure consistency with forecast quantities

24. Further Analysis • Apply proposed measures to enhanced set of archived data • Evaluations will help illustrate where forecast skill exists and improvements are possible • Explore possible alternatives for collection of climatological data • Final metric selection best made after more comprehensive evaluation

25. Closing Notes • Accuracy measures provide a bridge between programmatic and science activities and metrics • Important to recognize limitations of accuracy measures and continue consideration of other metrics