verification of weather parameters
Download
Skip this Video
Download Presentation
Verification of weather parameters

Loading in 2 Seconds...

play fullscreen
1 / 24

Verification of weather parameters - PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on

Verification of weather parameters. Anna Ghelli, ECMWF. Overview. Deterministic forecast performance for different weather parameters Precipitation forecast: scores and their confidence SYNOP on the GTS Precipitation analysis

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Verification of weather parameters' - asher


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
verification of weather parameters
Verification of weather parameters

Anna Ghelli, ECMWF

ECMWF forecast User Meeting -- Reading, June 2006

overview
Overview
  • Deterministic forecast performance for different weather parameters
  • Precipitation forecast: scores and their confidence
          • SYNOP on the GTS
          • Precipitation analysis
  • Ensemble prediction System: its performance relative to precipitation and 10m wind

ECMWF forecast User Meeting -- Reading, June 2006

slide3

2m Temperature

Skill (rmse) for different forecast ranges

North America

Top panel:

North America

Strong seasonality in the skill with higher values during winters.

Europe

Bottom panel:

Europe

2mT skill have reached a plateau. The skills for different timesteps have levelled off. The two past winters were dominated by prolonged periods of cold weather.

ECMWF forecast User Meeting -- Reading, June 2006

slide4

Performance for different forecast ranges

Europe

Top panel:

Specific humidity

Higher skill (MAE) in winter. Last four winters have consistently kept higher level of performance

Europe

Bottom panel:

10m wind speed

RMSE: Changes in the forecasting model have improved the performance of the model in forecasting wind speed.

ECMWF forecast User Meeting -- Reading, June 2006

slide5

Observed yes

Observed no

Forecast

yes

Forecast

no

1. FREQUENCY BIAS INDEX

3. HIT RATE

2. TRUE SKILL SCORE

3. FALSE ALARM RATE

ECMWF forecast User Meeting -- Reading, June 2006

slide6

Europe

15mm/24h

t+42

24 hour accumulated precipitation verified against SYNOP on GTS

The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value.

t+90

High thresholds have large confidence intervals, important to remember when assessing performance of the system

ECMWF forecast User Meeting -- Reading, June 2006

slide7

Europe

t+42: solid shading

t+90: dotted shading

24 hour accumulated precipitation verified against SYNOP on GTS

FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimate

FBI<1 under-estimate

The forecasting system over-estimate the number of events for thresholds of 1 mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. Further changes in the convection during 2003 have slightly increased FBI values

ECMWF forecast User Meeting -- Reading, June 2006

slide8

Europe

15mm/24h

t+42: solid shading

t+90: dotted shading

24 hour accumulated precipitation verified against SYNOP on GTS

FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals!

ECMWF forecast User Meeting -- Reading, June 2006

slide9

Stratifying the sample: selecting seasons as rule

DJF

MAM

15mm/24h

15mm/24h

Green: t+42

Light blue: t+90

ECMWF forecast User Meeting -- Reading, June 2006

slide10

Precipitation analysis for Europe

  • High density networks in Europe (Member and Co-operating states)
  • Upscaling (simple box averaging to obtain a areal precipitation value)

ECMWF forecast User Meeting -- Reading, June 2006

slide11

Each grid box will contain a certain number of stations. The number of stations will not be constant every day.

The number of stations per grid box indicates how representative the analysis is for the specific grid point.

ECMWF forecast User Meeting -- Reading, June 2006

slide12

Which reference system?

t+42

Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.

t+90

1 mm/24h

Green : proxy

ECMWF forecast User Meeting -- Reading, June 2006

slide13

Which reference system?

15 mm/24h

Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges.

Green : proxy

t+90

ECMWF forecast User Meeting -- Reading, June 2006

slide14

Europe

  • FBI plotted for two thresholds (1mm/24h, and 15mm/24h)
  • Verification against SYNOP on the GTS (greenish shading),
  • Verification against precipitation analysis (blue dotted)

FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds.

Forecast range t+66

ECMWF forecast User Meeting -- Reading, June 2006

slide15

Europe

  • TSS (threshold 25mm/24h) plotted for t+66
  • Verification against SYNOP on the GTS (greenish shading),
  • Verification against precipitation analysis (blue dotted)

TSS values decrease as we increase forecast range.

ECMWF forecast User Meeting -- Reading, June 2006

slide16

Timeseries of Brier Skill Score for Europe

The BSS is written as 1- BS/BSref

Sample climate is the reference system

BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast

Forecast vs. observations

Increased resolution

T255

T399

Improvements back in Autumn 1999 –

High thresholds performance down at the beginning of 2005 probably linked to drier conditions over Europe. Dec-Apr 2006 performance better than previous 3 years.

ECMWF forecast User Meeting -- Reading, June 2006

slide17

Forecast vs. observations

ECMWF forecast User Meeting -- Reading, June 2006

slide18

Timeseries of Brier Skill Score for Europe

The BSS is written as 1- BS/BSref

Sample climate is the reference system

BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C

Forecast vs analysis

T255

Increased resolution

T399

10 m Wind speed

ECMWF forecast User Meeting -- Reading, June 2006

slide19

Europe

Rainy season (24h accum.): October to April

Forecast range: t+96

Verification against SYNOP on GTS

Consistent picture for the two seasons

2005-2006

1mm/24h

BS=0.079

2004-2005

1mm/24h

BS=0.157

ECMWF forecast User Meeting -- Reading, June 2006

slide20

Europe

Rainy season (24h accum.): October to April

Forecast range: t+96

Verification against SYNOP on GTS

Consistent picture for the two seasons. Higher thresholds better reliability

2005-2006

10 mm/24h

BS=0.019

2004-2005

10 mm/24h

BS=0.04

ECMWF forecast User Meeting -- Reading, June 2006

slide21

Europe

Rainy season (24h accum.): October to April

Forecast range: t+96

Verification against SYNOP on GTS

Consistent picture for the two seasons.

Full symbol: T511/799

Shape: T255/T399

2005-2006

5 mm/24h

t+96 : A=0.889

t+120: A=0.860

2004-2005

5 mm/24h

t+96 : A=0.864

t+120: A=0.832

ECMWF forecast User Meeting -- Reading, June 2006

slide22

Europe

ROC Area

Verification against SYNOP on GTS for t+96

T255

Increased resolution

T399

ECMWF forecast User Meeting -- Reading, June 2006

conclusion
2m Temperature: the skill over Europe has reached a plateau. Over N.America winters are more skilful than summers.

Specific humidity shows consistent skills. Winters more skilful than summers

Wind: Recent changes in the model have resulted in a decrease of the RMSE over Europe.

TCC: New cloud scheme was introduced in April 2005. Forecasts of certain cloud types have improved.

Conclusion
  • Importance of confidence intervals
  • Precipitation forecast improvements are slow, but evident.
  • FBI indicates over-estimation of small threshold events  verification against precipitation analysis shows a better picture.
  • Precipitation analysis can be used for verification in a delayed mode. It can be used to define the performance of a reference system.

ECMWF forecast User Meeting -- Reading, June 2006

conclusion1
Conclusion
  • Brier skill score and ROC area: In recent year the system has maintained its good performance. The increase in horizontal resolution has had a small but positive impact on both precipitation and wind speed.
  • Reliability diagrams for the last two rainy seasons show a skilful system, with an evident decrease of Brier Score for the 2005-2006 season.

ECMWF forecast User Meeting -- Reading, June 2006

ad