1 / 33

Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute )

Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute ) Probabilistic Forecasts of Near-Gale Force Winds in the Baltic Applying ECMWF, EPS and Other Methods ECMWF Forecast Products User Meeting 15 – 17 June 2005. Introduction :. A study with 2 frameworks:

matana
Download Presentation

Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute )

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute ) Probabilistic Forecasts of Near-Gale Force Winds in the Baltic Applying ECMWF, EPS and Other Methods ECMWF Forecast Products User Meeting 15 – 17 June 2005 ECMWF User Meeting

  2. Introduction: A study with 2 frameworks: • Develop warning criteria / Guidance methods to forecast probability of near-gale force winds in the Baltic  Joint Scandinavian research undertaking • e.g. Finland and Sweden issue near-gale & storm force wind warnings for same areas using different criteria => homogenise ! • Evaluation of ECMWF products • Deterministic and probabilistic forecasts • Two (maybethree) calibration methods • Here, only ECMWF data applied  Later, HIRLAM, too • Here, 6 Finnish coastal stations  Later, c. 15-20 stations from Sweden, Denmark, Norway • Goal: Common Scandinavian operational practice (?) ECMWF User Meeting

  3. ECMWF MARS u & v components at 10 m => wind speed at 10m Forecast lead times: +12 hr to +144 hr Data retrieval: 0.5 * 0.5 degree resolution Operational, Control, EPS data (interpolated to 0.5o * 0.5o) Nearest grid point used Forecasts / observations valid: 00, 06, 12, 18 utc Observations: 10 minute mean wind speed Data coverage: 1/10/04 – 30/4/05  212 days Data: ECMWF User Meeting

  4. with height of instrumentation ? with observing site surroundings and obstacles ? with the coast ? with nearby islands ? with barriers ? with installations ? with low-level stability ? NE We may have problems: ECMWF User Meeting

  5. Observing stations( 6 out of 39 ) 02_873- Hailuoto 02_910- Valassaaret 02_980- Nyhamn 02_979 - Bogskär 02_981 - Utö 02_987- Kalbådagrund ECMWF User Meeting

  6. Heights of the instrumentation ( in red, the 6 out of 39 ) (m) 55 50 45 40 35 30 25 20 15 10 5 ECMWF User Meeting

  7. 873-Hailuoto 46 / 8 m ECMWF User Meeting

  8. 910 - Valassaaret 22 / 18 m ECMWF User Meeting

  9. 979 - Bogskär 32 / 4 m ECMWF User Meeting

  10. 980 - Nyhamn 25 / 8 m ECMWF User Meeting

  11. 981 - Utö 31 / 9 m ECMWF User Meeting

  12. 987 - Kalbådagrund 32 / 23 m ECMWF User Meeting

  13. 979 - Bogskär Unstable 32 m Neutral Stable 10 m Wind speed dependence: Logarithmic wind profile 14 m/s 15 15,5 m/s threshold ECMWF User Meeting

  14. Methods for producing probabilistic forecasts: • EPS (51 members): Probability of wind speed > 14 m/s • Kalman filtering • Various approaches  No details given here • Deterministic forecasts, adjusted by “a posteriori” estimate of the observed error distribution of the dependent sample Probability distribution of near-gale • Gives an estimate of the upper limit of the probabilistic predictability. Deterministic forecasts: • Error distribution of original sample (212 cases) • Approximation of the error distribution with a Gaussian fit (m, s): • ”sample error” method ECMWF User Meeting

  15. Methods for producing probabilistic forecasts: • Deterministic forecasts, adjusted with a Gaussian distribution fitted to model forecasted stability(temperature forecasts at 2 adjacent model levels)  Probability distribution of near-gale, “stability” method • Scheme used at SMHI (H. Hultberg) • “Neighbourhood” method • Both spatial (right) and temporal “neighbours” • c. 25-75 “members” • Applicable primarily for hi-res models ? ECMWF User Meeting

  16. Calibration of EPS forecasts: • Traditionally, calibration of the EPS is done by re-labeling the probabilities using the information of the reliability diagram (large sample of past forecasts and observations is needed) • Here, Kalman filtering is used to calibrate the EPS mean (as well as operational and control forecasts). Then each EPS member is transformed with the same relationship (state vector). • This will calibrate the “mean” of the distribution, hopefully also the “spread”. • Kalman filtering is also used in the traditional way to correct the deterministic forecasts and then to estimate the probabilities using the observed error distribution. ECMWF User Meeting

  17. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics ECMWF User Meeting

  18. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

  19. 32 m 46 m 25 m 22 m 31 m 32 m Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

  20. Sample climatologic characteristics, ref. ECMWF ECMWF User Meeting

  21. Deterministic FCs: Bias - RMSE - 981_Utö w.r.t to FC lead time ME (Bias) RMSE “Ensemble spread” ECMWF User Meeting

  22. Deterministic FCs: Bias - RMSE - 987_Kalbåda w.r.t to FC lead time ME (Bias) RMSE ECMWF User Meeting

  23. Brier Score: • BS = ( 1/n ) Σ ( p i – o i ) 2 • Common accuracy measure of prob fcs • o iis binary (0 or 1) • Analogous to MSE in probability space • A quadratic scoring rule • Sensitive to large forecast errors ! • Careful with limited datasets ! • Influenced by climatologic frequency of the sample • Different samples not to be compared • Brier Skill Score: • BSS = [ 1 – BS / BS ref ] *100 Range: 0 to 1 Perfect score = 0 Range: - oo to 100 Perfect score = 100 Probabilistic FCs: BrierSkill w.r.t to FC lead time 987_Kalbåda ECMWF User Meeting

  24. Relative Operating Characteristic Probabilistic FCs: ROC • To determine the ability of a forecasting system to discriminate between situations when a signal is present (here, occurrence of gale) from no-signal cases (“noise”) • To test model performance relative to a specific threshold • Applicable for probability forecasts and also for categorical deterministic forecasts • Allows for their comparison • Gained popularity in forecast verification in recent years ECMWF User Meeting

  25. Probabilistic FCs: ROC Curve • Graphical representation in a square box of the Hit rate (H) (y-axis) against the False Alarm Rate (F) (x-axis) for different potential decision thresholds • Curve is plotted from a “binned” set of probability forecasts by stepping (or sliding) a decision threshold (e.g. 10% probability intervals) through the forecasts, each probability decision threshold generating a separate 2*2 contingency table • The probability forecast is transformed into a set of categorical “yes/no” forecasts • A set of value pairs of H and F is obtained, forming the curve • It is desirable that H be high and F be low, i.e. the closer the point is to the upper left-hand corner, the better the forecast • A perfect forecast system, with only correct forecasts & no false alarms, (regardless of the threshold chosen) has a “curve” that rises from (0,0) (H=F=0) along the y-axis to (0,1) (upper left-hand corner; H=1, F=0) and then straight to (1,1) (H=F=1) H = a / ( a + c ) F = b / ( b + d ) ECMWF User Meeting

  26. Probabilistic FCs: ROC Curve generation ( S a ) ( S b ) b+d =5351 a+c =1920 Example H = a / ( a + c ) F = b / ( b + d ) To learn more about ROC and Signal Detection Theory, check: http://wise.cgu.edu/ ECMWF User Meeting

  27. Probabilistic FCs: ROCA Area Range: 0 to 1 Perfect system = 1 • Area under the ROC curve • Decreases from 1 when curve moves downward from the ideal top-left corner • A useless forecast system is along the diagonal, when H=F and the area is = 0.5; Such system cannot discriminate between occurrences and non-occurrences of the event ROCA based skill score: ROC_SS = 2 * ROCA - 1 • Negative below the diagonal • At it’s minimum: ROC_SS = - 1, when ROCA = 0 • ROC is applicable for deterministic categorical forecasts • ROC_SS translates into KSS  TSS (= H – F ) • Only one single decision threshold - only a single ROC point results Typically, this is “inside“ the ROC area, i.e. indicating worse quality • ROC, ROCA and ROC_SS are directly related to a decision-theoretic approach • Can be related to the economic value of probability forecasts to end users • Allowing for the assessment of the costs of false alarms Range: -1 to 1 Perfect score = 1 ECMWF User Meeting

  28. Probabilistic FCs: ROC curve/area; T +48 hr 987_Kalbåda ROC_EPS ROC_Kalman (EPS) ROCA = 0.85 ROCA = 0.73 ECMWF User Meeting

  29. Probabilistic FCs: ROC curve/area; T + 24 hr 981_Utö ROC_”stability” ROC_”neighbour” ROCA = 0.88 ROCA = 0.96 ECMWF User Meeting

  30. EPS Probabilistic FCs: ROC Area w.r.t to FC lead time 873_Hailuoto 910_Valassaaret ECMWF User Meeting

  31. Probabilistic FCs: ROC Area w.r.t to FC lead time 981_Utö 987_Kalbåda ECMWF User Meeting

  32. Conclusions  Future: • So far we’ve just scratched the (sea) surface • Need much more experimentation with various methods • Different methods for different time/space scales • e.g. very-short vs. medium-range ? • Biases and other scores depend on station (e.g. observation height) • (Statistical) adjustment of original observations required ? • Finland has an operational scheme for this ! • EPS forecasts are slightly underdispersive • Kalman filtering reduces the biases and produces better prob. forecasts for most stations in terms of the ROC curve/area • Apply to data from other counterparts • Reach the goal… !!! ECMWF User Meeting

  33. Forecast Quality Project 2005 The Royal Meteorological Society at the behest of the UK weather forecasting industry and their customers, has undertaken a project to establish methodologies and metrics by which the quality of weather forecast services can be assessed from a user perspective on a basis that is clear, scientifically well founded, relevant to the users’ needs and easily applied and understood. UK forecast user and provider input is NOW needed!www.rmets.org/survey Almost finnish(ed), but one advertisement… ECMWF User Meeting

More Related