RFC Verification Workshop What to Expect

1. RFC Verification WorkshopWhat to Expect Julie Demargne and Mary Mullusky RFC Verification Workshop, August 14-16 2007

2. Outline • Context • Goals • Agenda • Survey Analysis • Verification Questions • Logistics

3. Approach to River Forecasting Observations Forecasts Models Predictions BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR 13 2005

4. Where is the …? • … Verification • Little verification of hydrologic forecasts has been conducted to date. • Key question: How good are the forecasts? • Need to fill this void for Research, Operations and Users ???

5. Improving the Approach to River Forecasting Observations V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N Forecasts Models Predictions BULLETIN FLOOD WARNING NATIONAL WEATHER SERVICE SHREVEPORT LA 1033 AM CDT WED APR 13 2005

6. Hydrologic Verification • Verification helps us answer these questions: • How good are the forecasts? • What are the sources of skill and error? • How is new science improving the forecasts? • What should be done to improve the forecasts? • Verification supports: • Research by identifying needs and clarifying value of results • Operational agencies by defining acceptable methods and satisfying user demands • Users by quantifying forecast skill and uncertainty for use in decision making

7. Hydrologic Verification in NWS • NWS River Forecast Verification System under development • Requirements and Verification Plan defined by Verification System Requirements Team in October 2006 • On-going verification activities to be coordinated and expanded Model Setup Calibration Operation Installation State Updating Data Quality Control Runtime Simulations Data Assimilation Verification Forecast Computation Hydrologic and Hydraulic Models Postprocessor Product Review and Issuance Forecaster Analysis and Quality Control

8. Hydrologic Verification in NWS !!! Papers Verification Experts Forecasts + Verification Info x Verification System

9. Workshop Goals • OBJ1: Review verification concepts and methodologies • OBJ2: Review existing verification software capabilities • OBJ3: Discuss strategies to communicate verification results • OBJ4: Discuss recommendations in NWS River Forecast Verification Plan report Answer your verification questionsHelp you develop verification expertise

10. Workshop Agenda: Highlights • Day 1 • Verification concepts and metrics • Verification strategy -> lab with your forecast products • Local verification activities • Day 2 • National verification • Archive database • Logistical verification -> lab • Deterministic verification: IVP -> lab • Day 3 • Probabilistic forecast verification at University of Iowa • Ensemble verification: EVS -> lab

11. Survey Analysis • Goal: better understand forecast processes and forecaster expertise • Forecast Processes • Deterministic forecast • Ensemble forecast • Verification Processes and Expertise

12. Survey Analysis: Mods All sorts of Mods are used Significant effort from forecasters

13. Survey Analysis: Mods Various factors used to define Mods Large expertise of forecasters

14. Survey Analysis: Mods Numerous iterations of Mods Significant effort from forecasters

15. Survey Analysis: Variables All sorts of variables are important Each variable may require a different verification strategy

16. Survey Analysis: Sources of Error Many sources of error How do you separate the different sources of error

17. Survey Analysis: Verification Metrics A significant part of Day 1 will be devoted to verification metrics

18. Survey Analysis: Reference Forecast Discussion on reference forecasts will be part of the presentation of skill score metrics (Day 1)

19. Survey Analysis: Other Findings • Ensembles • Benefits not identified primarily because of lack of rigorous verification • Quality of real-time forecasts (deterministic and ensemble) • Role of hindcasting experiments (deterministic and ensemble)

20. Verification Questions • List of questions provided (sorted by agenda topics) • Verification Concepts and methodologies • Effective Strategies to Communicate Verification (deterministic and probabilistic) • Findings and Recommendations of the Hydrologic Verification System Requirements Team • National Verification Program • Archiving requirements (deterministic and probabilistic) • IVP Software Capabilities • EVS software capabilities • Verification Teams (Workshop Follow Up and Future Training) • Strategy: Questions answered during the session. Review questions at the end of the day to be certain questions were answered. Remaining questions will be placed on a list for the new NWS Hydrology Forecast Verification Team.

21. Workshop Logistics • To eat • Lunch • Dinner at Squatters Brew Pub at 6:30pm tonight (head count) • Lab sessions • Workstation • Internet access • CBRFC workshop contact point • Carpool CBRFC staff: thank you!

22. Thank you … let’s start!