1 / 30

Evaluation of a Mesoscale Short-Range Ensemble Forecasting System

Evaluation of a Mesoscale Short-Range Ensemble Forecasting System over the Northeast United States Matt Jones & Brian A. Colle NROW, 2004 Institute for Terrestrial and Planetary Atmospheres Stony Brook University Stony Brook, New York. OUTLINE Verification Method - Northeast

aida
Download Presentation

Evaluation of a Mesoscale Short-Range Ensemble Forecasting System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of a Mesoscale Short-Range Ensemble Forecasting System over the Northeast United States Matt Jones & Brian A. Colle NROW, 2004 Institute for Terrestrial and Planetary Atmospheres Stony Brook University Stony Brook, New York

  2. OUTLINE • Verification Method - Northeast - Seasonal view - Multiple parameters • Results • Conclusions

  3. Verification Method SUMMER = May – September 2003 WINTER = October 2003 – March 2004 Scalar Measures: Contingency-based Measures: Prob.-based Measures:

  4. SUMMER MAE SUMMER ME 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD night day night day night day night day

  5. Near-Surface T Lowest level cloud water (~3K ft.) Example of PHYS-member spread – Eta-PBL 2mT Warm Cool Moist Dry

  6. WINTER MAE WINTER ME 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD NCEP BREDS GFS night day night day night day night day

  7. 21zEta-1 21zEta-2 21zEta+1 21zEta+2 21zEta-CTL 00zEta 00zGFS IC MEAN PHYS MEAN L992 L 2004102200 f48

  8. SUMMER MAE SUMMER MAE 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD 0000UTC Eta 0000UTC ensemble mean 0000UTC 4-km MM5 1200UTC 4-km MM5 0000UTC ensemble mean night day night day Can the ensemble-meanbeat 4km MM5and Etadeterminitistic forecasts? night day night day

  9. WINTER MAE WINTER MAE 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD 0000UTC 4-km MM5 1200UTC 4-km MM5 0000UTC ensemble mean 0000UTC Eta 0000UTC ensemble mean night day night day Can the ensemble-meanbeat 4km MM5and Etadeterminitistic forecasts? night day night day

  10. SUMMER 24HP BIAS WINTER 24HP BIAS Over Pred. Under Pred. PHYS IC ALL SUMMER 24HP ETS WINTER 24HP ETS Better Worse

  11. Verification Rank Histogram • All solutions of ensemble should be equally likely. • Observation should appear no different than any ensemble member. • Not a measure of skill; a necessary, but not sufficient condition for a good ensemble. Perfect “flat” Under-dispersed Over-Dispersed Biased “U-shaped” “N-shaped” “L-shaped”

  12. SUMMER RANK HIST / MISSING RATES WINTER RANK HIST / MISSING RATES 2mT 2mRH SLP 10mWS

  13. SUMMER MAE-VAR WINTER MAE-VAR 2mT 2mRH SLP 10mWS 10mWD Corr. Coeff. MAE MAE VAR VAR C.C. C.C.

  14. Probabilistic Precipitation Brier Score: REL = Reliability RES = Resolution (event discrimination) UNC = Uncertainty (dependent only on obs.) fi = forecast probability oi = observed probability (=1 for occurrence, =0 for non-occurrence) Nt = number of forecast/event pairs for threshold, t m = number of ensemble members (m+1 probability categories) Perfect Reliability No skill No resolution Skill

  15. PHYS IC ALL SUMMER 24HP Reliability Diagrams Sample SUMMER 24h MPC

  16. PHYS IC ALL WINTER 24HP Reliability Diagrams Sample WINTER 24h MPC

  17. Ensemble Post-processing • Due to model imperfections, significant bias is retained even after ensemble averaging. Day-15 Day-14 Day-13 Day-12 Day-11 Day-10 Day-9 Day-8 TODAY Day-7 Day-6 Day-5 Day-4 Day-3 Day-2 Day-1 Use previous 14 complete forecasts to correct forecasts starting 0000UTC today

  18. SUMMER MISSING RATE IMPROVEMENT WINTER MISSING RATE IMPROVEMENT Uncalibrated Calibrated 2mT 2mRH SLP 10mWS

  19. Conclusions (1) • The ensemble-mean is more skillful than component members on average for daytime 2mT/10mWS, SLP, and 10mWD. Persistent biases among component members reduce the skill advantage of the ensemble-mean during other periods (e.g. nighttime 2mT/10mWS). • The ensemble-mean can outperform the deterministic Eta model, and can equal the skill of a high-resolution deterministic MM5 initialized 12 hours later. • The PHYS ensemble is more beneficial for forecasting surface parameters during the warm season due to greater variation among component members. • The GFS initial condition leads to a superior SLP forecast compared to the poorly skilled NCEP Eta-bred members, especially during the cool season. The GFS member outperforms the ensemble-mean for SLP and 10mWD in the cool season. • The ensemble has some ability to predict forecast skill and estimate the uncertainty of a forecast through ensemble spread-error correlation, especially for 10mWD. Persistent biases among component members and ensemble underdispersion for other surface parameters reduce the spread-error correlation (e.g. 2mT, 10mWS).

  20. Conclusions (2) • In warm season, low POPs have reliability for low threshold precip. events. High POPs have reliability for all thresholds. • In cool season, low POPs have poor reliability for all precip. event thresholds. High POPs have reliability all precip. event thresholds. • The PHYS (IC) ensemble is more skillful in POPs during the warm (cool) season. In the warm season, the Hybrid ensemble has the greatest POP skill. • A 14-day bias calibration can reduce much of the bias for most parameters, improving ensemble MRs.

  21. REALTIME SBU-SREF PRODUCTS http://fractus.msrc.sunysb.edu/mm5rte 18-mbr Ens output Ensemble Stats Ensemble Verif.

  22. Acknowledgments • Eric Grimit – University of Washington • NWS – OKX • ITPA – SBU Website • http://fractus.msrc.sunysb.edu/mm5rte Publication • Jones, M.S., and B. A. Colle, 2004: Evaluation of a mesoscale short-range ensemble forecasting system over the Northeast United States. Wea. Forecasting, in preparation.

  23. OUTLINE • Verification Method • Results • Conclusions • Future Work

  24. Future Work • Investigate for which synoptic regimes ensemble variance is most/least useful. • Investigate for which synoptic regimes a post-processing technique is most beneficial (MOS vs. historical bias calibration). • Reduce the inequality of skill among members by removing poorly-performing members / replacing with multiple models, multiple analysis initial conditions. • Investigate alternative ensemble quantities (trimmed mean/variance, modal quantile value). • Continue efforts in improving presentation of forecast uncertainty/ensemble confidence.

  25. Verification Rank Histogram • All solutions of ensemble should be equally likely. • Observation should appear no different than any ensemble member. • Not a measure of skill; a necessary, but not sufficient condition for a good ensemble. Perfect MR = “Missing Rate” “flat” Under-dispersed Over-Dispersed Biased “U-shaped” “N-shaped” “L-shaped”

  26. Usability of Ensemble Variance • The variance of a properly dispersed ensemble is a good representation of forecast uncertainty. • Ensemble variance should be correlated with ensemble error, leading to an ability of the ensemble to predict ensemble skill (Houtekamer 1993). High skill Low spread Low skill High spread

  27. Ensemble Probability Forecasts • An ensemble distribution should present what is most probable and what is least probable, reducing the “element of surprise” (Brooks and Doswell 1993).

  28. SUMMER MAE REDUCTION WINTER MAE REDUCTION PHYS IC ALL

  29. SUMMER % BEST SUMMER % WORST

  30. 2mT 24HP 2003081000 CASE Near-Surface T Lowest level cloud water (~3K ft.) Composite “moist” case Composite “dry” case 2003081000 case KF2 KF BM GR Warm Cool Moist Dry

More Related