1 / 34

ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE. Jeff Waldstreicher Scientific Services Division – Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS.

eamon
Download Presentation

ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE Jeff Waldstreicher Scientific Services Division – Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY

  2. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS • IMPACT ON VERIFICATION SCORES • Performance Metrics • CASE STUDY ANALYSIS • Event Verification • Subjective Evaluation of Impact on Forecast Process • AFDs • Event Reviews and Impact Reports

  3. COMPLICATIONS • Many Factors Influence Performance Metrics • Factors Not Independent • Difficult to tie performance changes to a specific factor • Cannot Analyze “Null Case” • If forecasters have “knowledge” or “data,” cannot directly answer “What if they did not have the knowledge?”

  4. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES • Infusion of New Technologies • Hardware (New systems or Processors) • Software (New algorithms or Models) • Applied Research and Development • National (Research Laboratory) • Local/Regional • Independent • Collaborative • Changes to Operational Procedures • Implementation of Best Practices

  5. FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES • Climate Variability • Variations in frequency and type of events • External Outreach and Education • Including development of spotter networks • Personnel/Staffing Issues • Including forecaster experience levels • Training • All other factors are also tied to training issues

  6. HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS? • Can be global (national), regional, or local • Technology usually national • Climate Variability typically regional • Staffing usually local • Impact can be long or short term • Technology infusion typically has long term impact • Climate Variability is usually short term • Factors are often inter-related • New technology or staffing changes require training • Applied R&D often yields/suggests changes to operations

  7. COMET PROJECTS IN EASTERN REGION • Different types of projects • Cooperative (2-3 year ~$35K/yr) • 19 ER Cooperative Projects funded since 1991 • Partners (1 year ~$9K/yr) • 39 ER Partners Projects funded since 1991 • 20 ER WFOs and 3 RFC have participated • 90 Offices Nationwide • 21 Universities have participated in ER projects • 70+ Universities Nationwide

  8. EVALUATING COMET PROJECTS • Examined COMET Cooperative and Partners Projects in Eastern Region • Projects completed between 1995 and mid-2001 • Projects specifically addressing warning programs • Tornado • Severe Thunderstorms • Flash Flooding • Winter Storms • Study was designed to minimize as much as possible the impact of the factors previously discussed.

  9. METHODOLOGY • 3-year running verification scores used • Minimize impact of short-term factors such as variability of events • Compared 3 years before project to 3 years following project • 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores and a full 3 years of “after” scores • 3-year “expected” improvements calculated based on long term trend of ER-wide scores • Compared rate of improvement for WFO involved in collaborative project to ER-wide improvement • Use of ER scores as a baseline minimizes the impact of national/region-wide factors such as AWIPS and radar system improvements • Very difficult to evaluate the impact of project results beyond the primary WFO

  10. INDIVIDUAL PROJECT COMPARISON • Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

  11. PROBABILITY OF DETECTION (POD) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 3 2 1 Severe Tstm Warnings 6 1 0 Flash Flood Warnings 5 1 0 Winter Storm Warnings 4 3 0

  12. FALSE ALARM RATIO (FAR) # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 2 4 0 Severe Tstm Warnings 3 4 0 Flash Flood Warnings 3 3 0 Winter Storm Warnings 3 3 1

  13. LEAD TIME # of COMET Projects With WFO Greater Improvement # of COMET Projects With ER Greater Improvement # of COMET Projects With No Difference in Improvement Tornado Warnings 5 1 0 Severe Tstm Warnings 5 1 1 Flash Flood Warnings 4 1 1 Winter Storm Warnings 4 3 0

  14. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES • WFO RAH and NCSU • Continuous collaborative projects since January 1991 • COMET Projects 1991-2000 • 3 Cooperative Projects • 3 Partners Projects • 1 Graduate Fellowship • CSTAR Projects • 2000-2003 • 2003-2006 • Unique “laboratory” to examine the impact of long-term collaborative activities

  15. IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES • WFO ALY and U-Albany • Continuous since February 1995 • Numerous COMET Projects • 3 Cooperative Projects • 7 Partners Projects • Several additional projects between UA and NCEP • CSTAR PROJECT 2001-2004

  16. SUMMARY • Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance • There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.

More Related