Hydrologic Verification Verification of Deterministic River Stage Forecasts

1. Edwin Welles OST Seminar June 6, 2007 Hydrologic Verification Verification of Deterministic River Stage Forecasts

2. 2 Outline of This Talk • Introductory comments about river forecasts. • Is there value in verifying stage forecasts? • Results of evaluating limited sample of NWS stage forecasts. • Proposal: Hydrologists should verify their forecasts. • Results of a hindcasting study on headwater basins. • Proposal 2: We can use verification information to guide our forecast process development. • Standardized Hydrologic Verification • Just advertising

3. 3 The beneficial ..... and not so beneficial uses of water.

4. Average Annual Deaths (1990-99): 502 14 99 55 Flood 57 Heat 58 193 26 4 Impacts of Floods and Droughts • Floods kill about 100 people a year • Approximate annual economic losses: Floods - $5 billion Droughts - $6-8 billion

5. 5 Mitigating the Impacts • Forecasts and data can help us manage our water resources and mitigate the impacts of floods. • NWS started issuing hydrologic forecasts in 1890.

6. 6 The Approach to River Forecasting Satellite Data Precipitation Estimates River Gage Data Weather Observations Radar Data Models Snow Cover/ Melt Data Precipitation Forecasts Reservoir Releases Climate Predictions Data and Forecasts to Meet Diverse Needs

7. 7 Forecast Modeling Structure Forecast Point Snow Elevation zone boundary ...for many rivers, Runoff Overland Flow Routing Reservoir with many reservoirs. Channel Routing Rating Curves

8. 8 The Forecast Process Observed: p, T, PE, Qres, Stg Verification BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR 13 2005 Forecast: p, T, PE, Qres, Q Data Assimilation

9. 9 Where’s the …..?

10. Where’s the …..? 10 • … Verification • Little verification of hydrologic forecasts has been conducted to date. • My Main Point • Need to fill this void in both Research and Operations. • So let’s look at some verification metrics

11. 11 Description of the Data • Two sets of River Stage forecasts from NWS RFCs • OK Dataset • 4 Locations in Oklahoma • 1993 to 2002 • 1 to 4 basins above the forecast point • Response time measured in hours • MM Dataset • 11 locations along the mainstem of the Missouri river • 1983 to 2002 • 500 to 1000 basins above the forecast point • Response time measured in days • Generated a Persistence forecast as a reference • Observation at time of forecast persisted into the future

12. 12

13. 13

14. 14

15. 15 Initial Observationsfrom this limited sample • Day 1 and Day 2 forecasts are accurate and skillful (as compared to persistence). • Little skill in day 3 • Little change over the periods of record. • Need to conduct a more complete study to establish a comprehensive baseline so we can answer this basic question. • What is the skill of hydrologic forecasts?

16. V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N Improving The Approach to River Forecasting 16 Leverage Data and Systems from NOAA and Collaborators Satellite Data Precipitation Estimates River Gage Data Weather Observations Radar Data Models Snow Cover/ Melt Data Precipitation Forecasts Reservoir Releases Valuable Data and Forecasts to Satisfy Diverse Customer Needs Climate Predictions

17. 17 Some More Basic Questionsfor which we do not have answers • How is new science improving the forecasts? • What are the largest sources of error in the forecasts? • What should be done to improve the forecasts? How can verification can help us answer these questions?

18. 18 A Hindcasting Experiment to Analyze Sources of Error • Generated hindcasts • Three headwater basisns in Oklahoma and Missouri. • Out to three day lead times • Four years (1997 to 2000) • Twelve total scenarios using… • 2 calibrations • A “good” and an “a priori” calibration • 2 state updating methods • Updated initial conditions and not-updated initial conditions • 3 QPF scenarios • Perfect QPF • The observations • Actual QPF • Computed QPF for 24 hours and then zero for days 2 and 3 • Zero QPF • Zero for all 3 days

19. 19 Analysis Method • For each scenario • Computed standard verifiction metrics on 4 subsets of the data • Sorted into high and low stages • Sorted by observations and by forecasts • Sorting by observations – Discrimination • Sorting by forecasts – Reliability • Presenting just RMSE for High stage discrimination • Compared scenarios with simple differences between the metrics. • Evaluated error in the QPF also.

20. 20 Comparing the Calibrations

21. 21 • The Skill of the Input Forecasts

22. 22 Comparing the State Updating Scenarios • State updating appears to bring similar skill to the hindcasts as the calibration in the early periods. • QPF and state updating skill appear independent.

23. 23

24. 24 Comparing theCalibration Scenarios • State updating appears to bring similar skill to the hindcasts as the calibration in the early periods. • Improving the calibration may degrade forecast skill depending upon the QPF characteristics. • Need good QPF to realize benefits of improved calibration.

25. 25

26. 26 Comparing the QPF Scenarios • Using some QPF almost always improves the forecasts (1 exception) for the High Stages • Not so for the low stages. • Can capitalize on the improved QPF even with poor calibration. • But get more improvement with better calibration

27. 27

28. 28

29. 29 Improving The Forecast Process The hindcasts tell us .... Observed: p, T, PE, Qres, Stg Help needed for > day 1 forecasts Verification Verification Verification Verification Verification Verification Verification Verification Verification BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR 13 2005 Verification Forecast: p, T, PE, Qres, Q Data Assimilation Critical for < day 1 forecasts

30. 30 Standardized Verification for Hydrologic Forecasts This Professor has a really good idea. This one used verification to explain her idea. U N W S U NWS Without Verification With Verification It is hard to communicate without a common language. Standardized Verification is the common language forecasters and researchers need.

31. 31 Standardized Verification for Hydrologic Forecasts • Supports research by identifying needs AND by clarifying the value of results. • Supports operational agencies by defining acceptable methods. • They are expected to evolve, but we must start somewhere. • Use peer review to establish validity.

32. 32 Hydrologic Forecast Process Supported by a complete verification system.

33. 33 Summary • Identified a need • Fill the hydrologic verification void. • Documented an initial description of NWS river stage forecast skill. • Need to verify more forecasts to update baseline • Verification methods can identify sources of error. • Demonstrated with hindcast experiment • Propose Standardized Verification will enhance verification efforts and moving research to operations.