Download
1 / 32

Rank Histograms – measuring the reliability of an ensemble forecast - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Rank Histograms – measuring the reliability of an ensemble forecast. You cannot verify an ensemble forecast with a single observation. The more data you have for verification, (as is true in general for other statistical measures) the more certain you are.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Rank Histograms – measuring the reliability of an ensemble forecast' - barth


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Rank Histograms – measuring the reliability of an ensemble forecast

  • You cannot verify an ensemble forecast with a single observation.

  • The more data you have for verification, (as is true in general for other statistical measures) the more certain you are.

  • Rare events (low probability) require more data to verify => as do systems with many ensemble members.

From Barb Brown


From Tom Hamill


Troubled Rank Histograms

Counts

0 10 20 30

Counts

0 10 20 30

1 2 3 4 5 6 7 8 9 10

Ensemble #

1 2 3 4 5 6 7 8 9 10

Ensemble #

Slide from Matt Pocernic


From Tom Hamill


From Tom Hamill


From Tom Hamill


From Tom Hamill


From Tom Hamill


Example of Quantile Regression (QR)

Our application

Fitting T quantiles using QR conditioned on:

Ranked forecast ens

ensemble mean

ensemble median

4) ensemble stdev

5) Persistence

R package: quantreg


Step 2: For each quan, use “forward step-wise

cross-validation” to iteratively select best subset

Selection requirements: a) QR cost function minimum,

b) Satisfy binomial distribution at 95% confidence

If requirements not met, retain climatological “prior”

Step I: Determine

climatological quantiles

Probability/°K

climatological

PDF

1.

Regressor set:

1. reforecast ens

2. ens mean

3. ens stdev

4. persistence

5. LR quantile

(not shown)

3.

T [K]

2.

4.

Temperature [K]

observed

forecasts

Time

Step 3: segregate forecasts into differing ranges of ensemble dispersion and refit models (Step 2) uniquely for each range

Final result: “sharper” posterior PDF

represented by interpolated quans

forecasts

Forecast

PDF

posterior

I.

II.

III.

II.

I.

Probability/°K

prior

T [K]

Temperature [K]

Time


Rank Probability Score

for multi-categorical or continuous variables


Scatter plot and contingency table
Scatter-plot and Contingency Table

Brier Score

Does the forecast detect correctly temperatures above 18 degrees ?

y = forecasted event occurence

o = observed occurrence (0 or 1)

i = sample # of total n samples

=> Note similarity to MSE

Slide from Barbara Casati


Other post-processing approaches …

1) Bayesian Model Averaging (BMA) –

Raftery et al (1997)

2) Analogue approaches –

Hopson and Webster, J. Hydromet (2010)

3) Kalman Filter with analogues –

DelleMonache et al (2010)

4) Quantile regression –

Hopson and Hacker, MWR (under review)

5) quantile-to-quantile (quantile matching) approach –

Hopson and Webster J. Hydromet (2010)

… many others


Quantile Matching: another approach when matched forecasts-observation

pairs are not available => useful for climate change studies

ECMWF 51-member Ensemble

Precipitation Forecasts compared

To observations

  • 2004 Brahmaputra Catchment-averaged Forecasts

  • black line satellite observations

  • colored lines ensemble forecasts

  • -Basic structure of catchment rainfall similar for both forecasts and observations

  • -But large relative over-bias in forecasts


Model Climatology CDF

“Observed” Climatology CDF

Pmax

Pmax

Precipitation

Pfcst

Padj

25th

50th

75th

100th

25th

50th

75th

100th

Quantile

Quantile

In practical terms …

ranked forecasts

ranked observations

0

1m

0

1m

Precipitation

Precipitation

Hopson and Webster (2010)


Bias-corrected Precipitation Forecasts

Original Forecast

Brahmaputra Corrected Forecasts

Corrected Forecast

=> Now observed precipitation within the “ensemble bundle”


ad