1 / 66

COSMO-DE-EPS: Setup, First Results, and Verification

This presentation provides an overview of the setup and first results of the COSMO-DE-EPS pre-operational phase, including verification and forecasters' feedback. It also discusses the enlargement of the sample at low cost and looks into past production cycles.

maegane
Download Presentation

COSMO-DE-EPS: Setup, First Results, and Verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COSMO-DE-EPS Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold

  2. Presentation Overview • setup of COSMO-DE-EPS • first results of pre-operational phase • verification • forecasters‘ feedback • enlarging the sample at low cost • looking into past production cycles

  3. Presentation Overview • setup of COSMO-DE-EPS • first results of pre-operational phase • verification • forecasters‘ feedback • enlarging the sample at low cost • looking into past production cycles new new

  4. Setup of COSMO-DE-EPS

  5. COSMO-DE-EPS status • pre-operational phase has started: Dec 9th, 2010 • pre-operational setup: • 20 members • grid size: 2.8 km convection-permitting • lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC) • variations in physics, initial conditions, lateral boundaries model domain

  6. Generation of Ensemble Members • plus variations of • initial conditions • model physics Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS GME, IFS, GFS, GSM

  7. Generation of Ensemble Members • plus variations of • initial conditions • model physics Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS is running as a time-critical application at ECMWF BC-EPS GME, IFS, GFS, GSM

  8. Generation of Ensemble Members Perturbation Methods Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115–123. Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, 168-177. (contains status of 2009)

  9. COSMO-DE-EPS plans (2011-2014) 2011 • upgrade to 40 members, redesign • statistical postprocessing • initial conditions by LETKF • lateral boundary conditions by ICON EPS reach operational status 2012 2013

  10. new First Results of Pre-operational Phase- verification - forecasters‘ feedback

  11. new First Results of Pre-operational Phase- verification - forecasters‘ feedback

  12. SYNOP Verification Method RADAR • Ensemble Members • Probabilities of Precipitation

  13. PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members • Do the ensemble members • have different long-term statistics? • (multi-model / multi-configuration) • Are there many cases with • the same „best member“ • or „wettest member“? • look at Equitable Threat Score • - look at Frequency Bias Index • (results similar, not shown) Equitable Threat Score 0.5 0.4 0.3 0.2 0.1 0.0 JUNE 2011 IFS GME GFS GSM }20 members 0 5 10 15 20 Forecast Time [h]

  14. PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members • Do the ensemble members • have different long-term statistics? • (multi-model / multi-configuration) • Are there many cases with • the same „best member“ • or „wettest member“? • look at Equitable Threat Score • - look at Frequency Bias Index • (results similar, not shown) Equitable Threat Score 0.5 0.4 0.3 0.2 0.1 0.0 JUNE 2011 IFS GME GFS GSM }20 members 0 5 10 15 20 Forecast Time [h] Only small differences in long-term statistics  Members may be treated as equally probable

  15. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JUNE 2011 Frequency 1 6 11 16 21 Observation Rank

  16. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank

  17. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank a) Underdispersiveness relatively small b) Four groups  Many cases with large influence by global models

  18. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h]

  19. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability)

  20. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JUNE 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability)

  21. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE MAY - JULY 2011 0.112 5 10 20 Threshold [mm/h] For larger precipitation amounts (summer): even more additional value

  22. PREC 1h accumulation Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm

  23. PREC 1h accumulation Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm Reliability diagram shows some bias and underdispersiveness Lines are not flat  additional calibration has good potential

  24. Summary of Verification (Precipitation) • Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) • Ensemble underdispersiveness is relatively small • Ensemble members may be treated as equally probable • Additional calibration has good potential

  25. Summary of Verification (Precipitation) • Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) • Ensemble underdispersiveness is relatively small • Ensemble members may be treated as equally probable • Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation

  26. Other Variables • T_2M and VMAX have been verified • ensemble spread is far too small • nevertheless, ensemble provides additional value to COSMO-DE

  27. Other Variables • T_2M and VMAX have been verified • ensemble spread is far too small • nevertheless, ensemble provides additional value to COSMO-DE COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables

  28. new First Results of Pre-operational Phase- verification - forecasters‘ feedback

  29. Forecasters‘ Feedback • available products: see figure precipitation, snow, wind gusts, T_2m probability thresholds: warning criteria • all products on grid-scale (2.8km) • in addition: precipitation probabilities for larger areas (10x10 grid boxes) „probability that the precipitation event will occur anywhere within the region“ probabilities, quantiles, ensemble mean, spread, min, max, …

  30. Forecasters‘ Feedback • evaluate „full package“ • including the visualization tool • consistency of products • select interesting cases • consider forecasters‘ interpretation • perception as intended? • is there any value in the forecast, additional to forecasters‘ knowledge?

  31. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points)

  32. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong

  33. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong • what they criticize: • jumpiness between subsequent runs • lack of spread in T_2M and VMAX

  34. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong • what they criticize: • jumpiness between subsequent runs • lack of spread in T_2M and VMAX • what they are learning: • dealing with low probabilities (10% probability for extreme weather  issue a warning?)

  35. new Enlarging the Sample at Low Cost - looking into past production cycles

  36. new Enlarging the Sample at Low Cost - looking into past production cycles - Introduction - Verification Study - Recommendation

  37. COSMO-DE-EPS Overview December 2010 • start of pre-operational phase / evaluation • 20 members  probabilities, quantiles, etc • runs at 00 UTC, 03 UTC, 06 UTC,…

  38. COSMO-DE-EPS Overview December 2010 • start of pre-operational phase / evaluation • 20 members  probabilities, quantiles, etc • runs at 00 UTC, 03 UTC, 06 UTC,… • start of pre-operational phase with 40 ensemble members • reach operational status • statistical postprocessing of probabilities of precipitation Q1 2012 End of 2012 2013

  39. COSMO-DE-EPS Overview December 2010 • start of pre-operational phase / evaluation • 20 members  probabilities, quantiles, etc • runs at 00 UTC, 03 UTC, 06 UTC,… • start of pre-operational phase with 40 ensemble members • reach operational status • statistical postprocessing of probabilities of precipitation further improvements ? Q1 2012 End of 2012 2013

  40. 21UTC 00UTC Enlarging the Sample at Low Cost • looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members } 20 members } 20 members

  41. 21UTC 00UTC Enlarging the Sample at Low Cost • looking into past production cycles 0 UTC 15 UTC 18UTC } 20 members } 20 members } 20 members } 60 members

  42. Important Check • How good are forecasts from past production cycles?

  43. Important Check • How good are forecasts from past production cycles? June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS • similar results for • different months • different starting times • different thresholds • 6h accumulations 0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14 time of day [UTC] time of day [UTC]

  44. Important Check • How good are forecasts from past production cycles? June 2011 precipitation 1h accumulations threshold: 1 mm/h Forecast Start: 00UTC 21UTC 18UTC FBI ETS • similar results for • different months • different starting times • different thresholds • 6h accumulations 0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14 time of day [UTC] time of day [UTC] The quality is similar. Exception: Most recent run with forecast lead time 0-3 hours is the best.

  45. Probability of Precipitation > 10 mm/h Resulting Probabilities 20 + 20 + 20 members 20 members % 2011-05-22 15UTC Forecast Start: 09 UTCForecast Start: 09 UTC, 06 UTC, 03 UTC

  46. Summary of Study • looking into past production cycles (20+20+20) • quality gain for precipitation (except first 2 forecast hours) • does not harm quality of T_2M and VMAX_10M • also applicable to „area probabilities“ Recommendation: Look into past production cycles

  47. 18UTC 21UTC 00UTC Technical aspect:how to derive probabilities from 20+20+20 members 0 UTC 15 UTC 21UTC 21h forecast in (pre-)operational mode + 6h additional lead time required

  48. Extra Slides

  49. Example

More Related