1 / 28

How Are We Doing? A Verification Briefing for the SAWS III Workshop April 23, 2010 Chuck Kluepfel

How Are We Doing? A Verification Briefing for the SAWS III Workshop April 23, 2010 Chuck Kluepfel National Weather Service Headquarters Silver Spring, Maryland 301-713-0090 x132 Charles.Kluepfel@noaa.gov Prepared October 2008. Part 1 Traditional Statistics. The Basics: POD and FAR.

felcia
Download Presentation

How Are We Doing? A Verification Briefing for the SAWS III Workshop April 23, 2010 Chuck Kluepfel

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Are We Doing? A Verification Briefing for the SAWS III Workshop April 23, 2010 Chuck Kluepfel National Weather Service Headquarters Silver Spring, Maryland 301-713-0090 x132 Charles.Kluepfel@noaa.gov Prepared October 2008 1

  2. Part 1 Traditional Statistics 2

  3. The Basics: POD and FAR • You can drive up your POD (also called hit rate) by over-forecasting IFR and below conditions. • This practice simultaneously drives up the FAR. • The CSI provides a mathematical way of correcting an inflated POD by using the FAR. The 2-category Heidke Skill Score has a similar affect, and it passes tests for equitability (statistical balance). Heidke also considers the “not forecast / not observed” situations (in an appropriately balanced manner), which are ignored by the CSI. 3

  4. Prevailing vs. Operational Impact Forecast (OIF) • Which should we use? • OIF considers TEMPO groups. • GPRA system uses OIF. • MOS / LAMP – Do not produce TEMPOs • When comparing to guidance, I used prevailing. 4

  5. Modified SW US, March 2009 to Feb 2010Flight Category: IFR and Below 5

  6. Traditional Stats for Modified Southwest United States: Colorado New Mexico Utah Arizona Nevada California Minus these WFOs: San Diego, Los Angeles, San Francisco 6

  7. 7

  8. Modified SW US, March 2009 to Feb 2010Flight Category: IFR and Below 8

  9. Modified SW US, March 2009 to Feb 2010Flight Category: IFR and Below 9

  10. Modified SW United StatesTAF Performance vs. ProjectionMarch 2009 to February 2010 10

  11. Modified SW US - March 2009 to Feb 2010Scheduled 3-6 hr IFR and Below Prevailing GFS LAMP (77K - 36K) K ÷ (77K – 63K) ~ 2.9 The 3-6 hr GFS LAMP false alarmed almost 3 times for every additional hit it got over the forecasters! 11

  12. Modified SW US - March 2009 to Feb 2010Scheduled 3-6 hr IFR and Below Prevailing NAM MOS (117K – 36K) ÷ (75K - 63 K) ~ 6.2 3-6 hr NAM MOS false alarmed over 6 times for every additional hit it got over the forecasters! 12

  13. WFOs San Diego, Los Angeles, San FranciscoTAF Performance vs. ProjectionIFR and BelowMarch 2009 to February 2010 13

  14. WFOs El Paso, Tucson, Phoenix(Low Desert Southwest)IFR and BelowMarch 2009 to February 2010 14

  15. Part 2 Lead-Time Software 15

  16. 16

  17. Part 3 The Future 17

  18. The Future Output to CSV files (just posted) Starting with Flight Category and Sig Wx Data Ceiling / Visibility (next) Winds (last) Plots of POD / FAR / CSI Sort Elements by Sig Wx Type 18

  19. Improving the Current System: Geometric Interpretation (X) FOH FAR (Z) 1 FOM (Ma,Mf) PON Ma: Average of Observations (x+y)/N Mf: Average of Forecasts (x+z)/N FORECAST (F) POD POFD 0 (W) (Y) DFR FOCN 1 0 OBSERVED (A) A(F): Regression of Observations upon the forecast F(A): Regression of Forecast upon observations

  20. Basic Interpretation: Extreme Cases PERFECTFORECAST RESIGN FROM THE NWS 20

  21. Basic Interpretation: Random Chance RANDOM CHANCE 21

  22. Basic Interpretation: False Alarms vs. Misses Over-forecast Under-forecast Assesses Bias in one glance!! 22

  23. Finis 23

  24. Traditional Stats Entire National Weather Service 24

  25. Nation, March 2009 to Feb 2010Flight Category: IFR and Below 25

  26. Nation, March 2009 to Feb 2010 Flight Category: IFR and Below 26

  27. Nation, March 2009 to Feb 2010 Flight Category: IFR and Below 27

  28. NationPerformance vs. ProjectionMarch 2009 to February 2010 28

More Related