Verification of weather parameters
This presentation is the property of its rightful owner.
Sponsored Links
1 / 24

Verification of weather parameters PowerPoint PPT Presentation


  • 60 Views
  • Uploaded on
  • Presentation posted in: General

Verification of weather parameters. Anna Ghelli, ECMWF. Overview. Deterministic forecast performance for different weather parameters Precipitation forecast: scores and their confidence SYNOP on the GTS Precipitation analysis

Download Presentation

Verification of weather parameters

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Verification of weather parameters

Verification of weather parameters

Anna Ghelli, ECMWF

ECMWF forecast User Meeting -- Reading, June 2006


Overview

Overview

  • Deterministic forecast performance for different weather parameters

  • Precipitation forecast: scores and their confidence

    • SYNOP on the GTS

    • Precipitation analysis

  • Ensemble prediction System: its performance relative to precipitation and 10m wind

  • ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    2m Temperature

    Skill (rmse) for different forecast ranges

    North America

    Top panel:

    North America

    Strong seasonality in the skill with higher values during winters.

    Europe

    Bottom panel:

    Europe

    2mT skill have reached a plateau. The skills for different timesteps have levelled off. The two past winters were dominated by prolonged periods of cold weather.

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Performance for different forecast ranges

    Europe

    Top panel:

    Specific humidity

    Higher skill (MAE) in winter. Last four winters have consistently kept higher level of performance

    Europe

    Bottom panel:

    10m wind speed

    RMSE: Changes in the forecasting model have improved the performance of the model in forecasting wind speed.

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Observed yes

    Observed no

    Forecast

    yes

    Forecast

    no

    1. FREQUENCY BIAS INDEX

    3. HIT RATE

    2. TRUE SKILL SCORE

    3. FALSE ALARM RATE

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    15mm/24h

    t+42

    24 hour accumulated precipitation verified against SYNOP on GTS

    The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value.

    t+90

    High thresholds have large confidence intervals, important to remember when assessing performance of the system

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    t+42: solid shading

    t+90: dotted shading

    24 hour accumulated precipitation verified against SYNOP on GTS

    FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimate

    FBI<1 under-estimate

    The forecasting system over-estimate the number of events for thresholds of 1 mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. Further changes in the convection during 2003 have slightly increased FBI values

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    15mm/24h

    t+42: solid shading

    t+90: dotted shading

    24 hour accumulated precipitation verified against SYNOP on GTS

    FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals!

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Stratifying the sample: selecting seasons as rule

    DJF

    MAM

    15mm/24h

    15mm/24h

    Green: t+42

    Light blue: t+90

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Precipitation analysis for Europe

    • High density networks in Europe (Member and Co-operating states)

    • Upscaling (simple box averaging to obtain a areal precipitation value)

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Each grid box will contain a certain number of stations. The number of stations will not be constant every day.

    The number of stations per grid box indicates how representative the analysis is for the specific grid point.

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Which reference system?

    t+42

    Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.

    t+90

    1 mm/24h

    Green : proxy

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Which reference system?

    15 mm/24h

    Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.

    Green : proxy

    t+90

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    • FBI plotted for two thresholds (1mm/24h, and 15mm/24h)

    • Verification against SYNOP on the GTS (greenish shading),

    • Verification against precipitation analysis (blue dotted)

    FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds.

    Forecast range t+66

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    • TSS (threshold 25mm/24h) plotted for t+66

    • Verification against SYNOP on the GTS (greenish shading),

    • Verification against precipitation analysis (blue dotted)

    TSS values decrease as we increase forecast range.

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Timeseries of Brier Skill Score for Europe

    The BSS is written as 1- BS/BSref

    Sample climate is the reference system

    BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast

    Forecast vs. observations

    Increased resolution

    T255

    T399

    Improvements back in Autumn 1999 –

    High thresholds performance down at the beginning of 2005 probably linked to drier conditions over Europe. Dec-Apr 2006 performance better than previous 3 years.

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Forecast vs. observations

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Timeseries of Brier Skill Score for Europe

    The BSS is written as 1- BS/BSref

    Sample climate is the reference system

    BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C

    Forecast vs analysis

    T255

    Increased resolution

    T399

    10 m Wind speed

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    Rainy season (24h accum.): October to April

    Forecast range: t+96

    Verification against SYNOP on GTS

    Consistent picture for the two seasons

    2005-2006

    1mm/24h

    BS=0.079

    2004-2005

    1mm/24h

    BS=0.157

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    Rainy season (24h accum.): October to April

    Forecast range: t+96

    Verification against SYNOP on GTS

    Consistent picture for the two seasons. Higher thresholds better reliability

    2005-2006

    10 mm/24h

    BS=0.019

    2004-2005

    10 mm/24h

    BS=0.04

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    Rainy season (24h accum.): October to April

    Forecast range: t+96

    Verification against SYNOP on GTS

    Consistent picture for the two seasons.

    Full symbol: T511/799

    Shape: T255/T399

    2005-2006

    5 mm/24h

    t+96 : A=0.889

    t+120: A=0.860

    2004-2005

    5 mm/24h

    t+96 : A=0.864

    t+120: A=0.832

    ECMWF forecast User Meeting -- Reading, June 2006


    Verification of weather parameters

    Europe

    ROC Area

    Verification against SYNOP on GTS for t+96

    T255

    Increased resolution

    T399

    ECMWF forecast User Meeting -- Reading, June 2006


    Conclusion

    2m Temperature: the skill over Europe has reached a plateau. Over N.America winters are more skilful than summers.

    Specific humidity shows consistent skills. Winters more skilful than summers

    Wind: Recent changes in the model have resulted in a decrease of the RMSE over Europe.

    TCC: New cloud scheme was introduced in April 2005. Forecasts of certain cloud types have improved.

    Conclusion

    • Importance of confidence intervals

    • Precipitation forecast improvements are slow, but evident.

    • FBI indicates over-estimation of small threshold events  verification against precipitation analysis shows a better picture.

    • Precipitation analysis can be used for verification in a delayed mode. It can be used to define the performance of a reference system.

    ECMWF forecast User Meeting -- Reading, June 2006


    Conclusion1

    Conclusion

    • Brier skill score and ROC area: In recent year the system has maintained its good performance. The increase in horizontal resolution has had a small but positive impact on both precipitation and wind speed.

    • Reliability diagrams for the last two rainy seasons show a skilful system, with an evident decrease of Brier Score for the 2005-2006 season.

    ECMWF forecast User Meeting -- Reading, June 2006


  • Login