1 / 44

X General COSMO meeting Cracow, 15-19 September 2008

COSMO LEPS system: status, verification, applications Andrea Montani , C. Marsigli, T. Paccagnella ARPA-SIM, Bologna , Italy F. Fundel, A. Walser, M. A. Liniger, C. Appenzeller Meteoswiss. X General COSMO meeting Cracow, 15-19 September 2008. Introduction Present status and recent upgrades

phillip-contreras
Download Presentation

X General COSMO meeting Cracow, 15-19 September 2008

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COSMO LEPS system:status, verification, applicationsAndrea Montani, C. Marsigli, T. PaccagnellaARPA-SIM, Bologna, ItalyF. Fundel, A. Walser, M. A. Liniger, C. AppenzellerMeteoswiss X General COSMO meeting Cracow, 15-19 September 2008 A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  2. Introduction • Present status and recent upgrades • Verification results (SYNOP on the GTS): • performance from 2002 onwards • Application of COSMO-LEPS: the reforecast exercise • Main results Outline A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  3. COSMO-LEPS (developed at ARPA-SIM) What is it? It is a Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland). Why? It was developed to combine the advantages of global-model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …) generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of severe weather events. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  4. COSMO-LEPS suite @ ECMWFbefore and during D-PHASE Operations Period (from 1 June to 30 November 2007) 16 Representative Members driving the 16 COSMO-model integrations (weighted according to the cluster populations) employing either Tiedtke or Kain-Fristch convection scheme (randomly choosen) 3 levels 500 700 850 hPa 4 variables Z U V Q d+3 d+4 d d+5 d+1 d+2 d-1 Cluster Analysis and RM identification Cluster Analysis and RM identification older EPS 00 2 time steps younger EPS 12 European area clustering period Complete Linkage • suite runs as a ”time-critical application” managed by ARPA-SIM; • Δx ~ 10 km; 40 ML; • COSM0 v3.20; • fc length: 132h; • Computer time (4.7 million BU for 2007) provided by the COSMO partners which are ECMWF member states. COSMO-LEPS Integration Domain COSMO-LEPS clustering area A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  5. Recent updates Changes to the operational suite on 1 December 2007: • new model version (COSMO-LM 4.0); • new numerics (Runge-Kutta scheme); • use of multi-layer soil model; • new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill. • Old system and new system ran in parallel for 52 days (October and November 2007); • Larger T2m spread in new system for all forecast ranges; • Comparable skill (in terms of T2m root-mean-square error) of the two systems. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  6. COSMO-LEPS suite @ ECMWFafter D-PHASE Operations Period (since December 2007) 16 Representative Members driving the 16 COSMO-model integrations (weighted according to the cluster populations) employing either Tiedtke or Kain-Fristch convection scheme (randomly choosen) + perturbations in turbulence scheme 3 levels 500 700 850 hPa 4 variables Z U V Q d+3 d+4 d d+5 d+1 d+2 d-1 Cluster Analysis and RM identification Cluster Analysis and RM identification older EPS 00 2 time steps younger EPS 12 European area clustering period Complete Linkage • suite runs as a ”time-critical application” managed by ARPA-SIM; • Δx ~ 10 km; 40 ML; fc+ 132h; • COSM0 v4.0 since Dec07 (with RK + multi-layer); • computer time (5.8 million BU for 2008) provided by the COSMO partners which are ECMWF member states. COSMO-LEPS Integration Domain COSMO-LEPS clustering area A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  7. Objective verification of COSMO-LEPS • SYNOP on the GTS Main features: variable: 12h cumulated precip (18-06, 06-18 UTC); period : from Dec 2002 to Aug 2008; region: 43-50N, 2-18E (MAP D-PHASE area); method: nearest grid point; no-weighted fcst; obs: synop reports (about 470 stations/day); fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h; thresholds: 1, 5, 10, 15,25, 50mm/12h; system: COSMO-LEPS; scores: ROC area, RPSS, Outliers both (69) monthly and (23) seasonal scores were computed verification over the full domain (about 1400 stations/day) is underway A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  8. Time series of ROC area • Area under the curve in the HIT rate vs FAR diagram. • Valuable forecast systems have ROC area values > 0.6. ROC • The positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different forecast ranges. • Poor performance of the system in Spring and Summer 2006 (both particularly dry), despite system upgrades. • Good performance during DOP 2007. • Small positive impact of 2007 system upgrades. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  9. Seasonal scores of ROC area • Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns (SON). • Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6. ROC • Need to take into account the different statistics for each season (JJA 2004 and SON 2006 less rainy than the others). • Good scores obtained throughout the full DOP in 2007. • Best performance for JJA 2008, more evident for longer ranges. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  10. Outliers: time series + seasonal scores • How many times the analysis is out of the forecast interval spanned by the ensemble members. • … the lower the better … • Performance of the system assessed as time series and for the last 5 summers (JJA). OUTL • Evident seasonal cycle (more outliers in winter), but overall reduction of outliers in the years up to 2007. • Reduction of outliers from one summer to the other, related to the increase of ensemble size (evident for the 10m to 16m increase). • Need to take into account the different statistics for each season. • Best results for the last 2 summers. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  11. COSMO-LEPS reforecasts (v 4.0) Motivations: COSMO-LEPS is often not reliable; probabilities might be wrong!  need for calibration Setup Period of 30 years (1971-2000) 1 deterministic run of COSMO-LEPS fc+90h, 12:00 initial time (every 3rd day) ERA40 as initial/boundary conditions. Calculated and archived at ECMWF Convect. scheme = Tiedtke/Kain-Fritsch Random physics (turlen & patlen) Limitations New climatology needed with each model version change Needs time and is costly (~ 2000 KBU/yr) A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  12. Observation Data Verification of calibrated vs raw forecasts • 24h Tot_Prec (0600-0600 UTC) • Domain: Switzerland • interp. on CLEPS Grid (C.Frei) • Apr06 – Aug07 (verification) • 1971-2000 (calibration) model topography [m] A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  13. BSSD 24h precipitation calibrated raw • BSSD is written as 1-BS/(BSref +D). Useful forecast systems if BSSD > 0. • BS measures the mean squared difference between fcst and obs in probability space. • Equivalent to MSE for deterministic forecast. winter summer A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  14. Main results (1) • Since November 2002, COSMO-LEPS system has been running on a daily basis. • Since June 2008, COSMO-LEPS data are disseminated to JRC for the use in the European Flood Alert System (EFAS) as input to the flood forecasting system over the major European basins ==> success for COSMO both from a scientific and a visibility point of view. • COSMO-LEPS products routinely used in met-ops rooms across COSMO community as well as in research Projects (e.g. COPS, D-PHASE, PREVIEW: see the poster). Time-series verification scores indicate the following trends: • increase in ROC area scores and reduction in outliers percentages; • positive impact of increasing the population from 5 to 10 members (June 2004); • some deficiency in the skill of the system can be identified after the system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT • scores are encouraging throughout DPHASE Operations Period; • system upgrades of Dec07 (RK + multi-layer + new perturbations) brought small but positive impact. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  15. Main results (2) • Calibrating with reforecasts improves the forecast skill significantly (especially in winter). • Calibrated warnings without using observations are possible. • Calibrating frequent precipitation events does not require a large calibration period, while calibrating extreme events does (not shown). A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  16. Thank you ! A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  17. Feasibility study: COSMO-LEPS at 7 km (the answer to forecasters’ dream?) New system x = 7 km z = 40 ML t = 66-72 s ngp = 510x405x40 = 8.262.000 fcst range = 132h cost = 1925 BU x run elapsed time = 138 min Present system x = 10 km z = 40 ML t = 90 s ngp = 306x258x40 = 3.157.920 fcst range = 132h cost = 640 BU x run elapsed time = 45 min … cosmoleps_7 is about 3 times more expensive than present configuration new computer at ECMWF being installed Computer resources for each ECMWF member state will increase by a factor of 5 (five)  and …. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  18. … the dream is possible COSMO-LEPS 10 km COSMO-LEPS 7 km • the grid of cosmoleps_7 would be almost identical to that of COSMO-EU, this making easier and cleaner the use of initial fields provided by DWD (e.g. soil moisture analysis). A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  19. Future plans (2008 and 2009) • test the use of the Soil Moisture Analysis fields provided by DWD; • run cleps_7 for ~ 40 days in autumn 2008 and assess the impact; • within TIGGE-LAM, develop coding of COSMO-LEPS output files in GRIB2 format; • migration to the new machine at ECMWF; • use a better snow analysis (possibly provided by DWD or Meteoswiss); • extend the cluster analysis so as to consider not only ECMWF EPS, but also UKMO MOGREPS as global ensemble providing ic’s and bc’s (first tests); • implement cosmoleps_7; • gaining from COSMO-SREPS experience, introduce more model perturbations; • test COSMO-LEPS nested on the under-development ECMWF EDA over MAP D‑PHASE period; • optimise use of reforecasts + calibration of wind gust; • support CONSENS. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  20. Feasibility study of COSMOLEPS at 7 km “Keep the pace” with deterministic model (x~ 2-3 km): if the gap in resolutions between deterministic and probabilistic systems is too large, the two systems do not support each other and go for different solutions (that is, they forecast different weather!). Motivations: Provide a more detailed description of mesoscale processes by incresing the horizontal resolution. Do not lose a “reasonable advantage” against ECMWF EPS, which will go to x=25 km during 2009. from 10 to 7 km (plus small domain extensions) does not seem a lot A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  21. but … A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  22. Present methodology A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  23. Operational set-up Additional products: 1 deterministic run (ICs and 3-hourly BCs from the high-resolution deterministic ECMWF forecast) to “join” deterministic and probabilistic approaches (start at 12UTC; t = 132h); 1 “hindcast” run (ICs and 3-hourly BCs from ECMWF analyses) to “downscale” ECMWF information (start at 00UTC; t = 36h). Core products: 16 perturbed COSMO-model runs (ICs and 3-hourly BCs from 16 EPS members) to generate, “via weights”, probabilistic output (start at 12UTC; t = 132h); A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  24. Recent updates Changes to the operational suite on 1 December 2007: • new model version (COSMO-LM 4.0); • new numerics (Runge-Kutta scheme); • use of multi-layer soil model; • new perturbations introduced relative to the maximal turbulent length scale (tur_len) and to the length scale of thermal surface patterns (pat_len)  to increase spread and maintain skill. • Old system and new system ran in parallel for 52 days (October and November 2007); • Larger T2m spread in new system for all forecast ranges; • Comparable skill (in terms of T2m root-mean-square error) of the two systems. back to plans A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  25. fc step: 78-90h Jun04: 5m  10m Feb06: 10m  16m; 32ML  40 ML Time series of ROC area for COSMO-LEPS (2) • Area under the curve in the FAR vs HIT diagram. • Valuable forecast systems have ROC area values > 0.6 ROC • the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different fcst ranges. • poor performance of the system in Spring and Summer 2006, despite system upgrades. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  26. Time series of Brier Skill Score • BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0. • BS measures the mean squared difference between forecast and observation in probability space. • Equivalent to MSE for deterministic forecast. BSS fc step: 30-42h • improvement of performance detectable for all thresholds along the years; • still problems with high thresholds, but good trend in 2007. Jun04: 5m  10m Feb06: 10m16m; 32ML  40 ML A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  27. Semi-diurnal cycle in COSMO-LEPS scores • BSS score … the higher the better … • Performance of the system assessed for 5 different Summers (JJA). BSS • Evident 12-hour cycle in BSS scores (the same holds for RPSS, while less evident for ROC area scores). • Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …). • The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range. • The bad performance in Summer 2006 is confirmed. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  28. Score dependence on the domain size (1) • Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM) and the full domain (~ 1500 stations; fulldom):  different statistics of the verification samples;  up to now, performance of the system over the 2 domains assessed only for the last 6 months (March-August 2007).  difficult to draw general conclusions A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  29. OUTL RPSS ROC Score dependence on the domain size (2) • RPSS score… the higher the better… (and positive). • ROC area… the higher the better… (and above 0.6). • Outliers percentage … the lower the better. • Smoother transitions from month to month in “fulldom” scores. • Slightly better performance of COSMO-LEPS over the MAPDOM, but the signal varies from month to month. • Higher predictability with orographic forcing?  Need to check individual regions and/or to stratify for type of stations. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  30. COSMO-LEPS 16-MEMBER EPS 51-MEMBER EPS MAM06 Average values (boxes 0.5 x 0.5) tp > 1mm/24h tp > 5mm/24h • As regards AVERAGE precipitation above these two threshols, the 3 systems have similar performance. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  31. Main results • COSMO-LEPS system runs on a daily basis since November 2002 and it has become a “member-state time-critical application” at ECMWF ( partial involvement of ECMWF operators in the suite management). • Time series scores identify positive trends in the performance of the system: • increase in ROC area scores and reduction in outliers percentages; • positive impact of increasing the population from 5 to 10 members (June 2004); • some deficiency in the skill of the system identified after the system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!)  need of further investigation. • High-resolution verification shows better scores of COSMO-LEPS with respect to EPS in forecasting precipitation maxima within boxes. • No clear impact of the weighting procedure as regards precipitation, BUTCOSMO-LEPS ensemble mean for 2m temperature has better skill (lower standard deviation to observations) with weighting the ensemble members according to cluster populations compared to the mean with unweighted members. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  32. fc step: 78-90h fc step: 30-42h DOP Feb06: 10m  16m; 32ML  40 ML Jun04: 5m  10m DOP Jun04: 5m  10m Feb06: 10m  16m; 32ML  40 ML Time series of ROC area • Area under the curve in the HIT rate vs FAR diagram. • Valuable forecast systems have ROC area values > 0.6. ROC • the positive impact of increasing the ensemble size in 2004 is evident for all thresholds and for different forecast ranges. • poor performance of the system in Spring and Summer 2006 (both particularly dry), despite system upgrades. • good performance during DOP 2007. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  33. Seasonal scores of ROC area • Performance of the system assessed for 5 different summers (JJA) and 5 different autumns (SON). • Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6. ROC • Need to take into account the different statistics for each season (JJA 2003 and SON 2006 less rainy than the others). • Best scores obtained during DOP 2007 for either seasons. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  34. DOP Jun04: 5m  10m Feb06: 10m16m; 32ML  40 ML Time series of Ranked Probability Skill Score • A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference system. BS measures the mean squared difference between forecast and observation in probability space. • Useful forecast system, if RPSS > 0. RPSS • the improvement of the system performance is detectable for all forecast ranges along the years; • the poor performance of the system in summer 2006 is confirmed; • RPSS always positive throughout 2007 and especially high during DOP. A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  35. DOP Jun04: 5m  10m Feb06: 10m  16m; 32ML  40 ML Outliers: time series + seasonal scores • How many times the analysis is out of the forecast interval spanned by the ensemble members. • … the lower the better … • Performance of the system assessed as time series and for 5 different summers (JJA). OUTL • Evident seasonal cycle (more outliers in winter), but overall reduction of outliers in the years. • Reduction of outliers from one summer to the other, related to the increase of ensemble size (more evident for the 5 to 10 increase). • Need to take into account the different statistics for each season (JJA 2003 less rainy than the others). • Best results for D-PHASE summer! A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  36. Future plans • Verification vs “DPHASE observations”: • assess performance over different domains (North and South of the Alps), • study individual case studies, • consider basin-by-basin performance. • “Think about” increasing horizontal resolution to 7 km. • Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss). • Implement “seamless COSMO ensemble system” merging COSMO-LEPS and COSMO-SREPS. • Develop “hybrid” clustering technique (take boundary conditions from a “grand-global” ensemble provided by mixing ECMWF-EPS and UKMO-MOGREPS). A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  37. Dissemination during DOP probabilistic products (www.smr.arpa.emr.it/prodotti and www.d-phase.info ) “deterministic” products (individual 16 COSMO-LEPS runs) derived probability products (EM, ES) images and alerts A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  38. Time series of Ranked Probability Skill Score • A sort of BSS “cumulated” over all thresholds. BSS is written as 1-BS/BSref. Sample climate is the reference system. BS measures the mean squared difference between forecast and observation in probability space. • Useful forecast system, if RPSS > 0. RPSS • the improvement of the system performance is detectable for all forecast ranges along the years; • the poor performance of the system in summer 2006 is confirmed; • RPSS always positive throughout 2007 and especially high during DOP. • Lesser skill in 2008 (so far). A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  39. Dim 2 Possible evolution scenarios Cluster members chosen as representative members (RMs) Initial conditions Dim 1 LAM scenario Dim 2 LAM scenario LAM integrations driven by RMs LAM scenario Dim 1 Initial conditions COSMO-LEPS methodology ensemble size reduction A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  40. Why calibrate? OBS-CLEPS/(OBS+CLEPS)/2 1971-2000 LT: 42h Q0.8 Q0.95 Jan COSMO-LEPS is not reliable, Probabilities might be wrong! Need for calibration Jul A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  41. Reliability Diagram: Verification A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  42. Winter 06/07 Q0.8 24h precipitation calibrated raw + (18-42h) + (66-90h) • raw forecast overconfident, very limited skill • strong improvement in reliability • long lead-time forecasts more reliable A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  43. Summer 06 & 07 Q0.8 24h precipitation calibrated raw + (18-42h) + (66-90h) • raw forecast overconfident • modest improvement of reliability • long lead-time forecasts more reliable A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

  44. The BSS debiased Weigel et al. 2006, Mon. Wea. Rev. Special case BSS: M: Ensemble size A.Montani; COSMO-LEPS status, verification, … X COSMO meeting - Cracow - 15-19 September 2008

More Related