Rank histograms measuring the reliability of an ensemble forecast
This presentation is the property of its rightful owner.
Sponsored Links
1 / 32

Rank Histograms – measuring the reliability of an ensemble forecast PowerPoint PPT Presentation


  • 89 Views
  • Uploaded on
  • Presentation posted in: General

Rank Histograms – measuring the reliability of an ensemble forecast. You cannot verify an ensemble forecast with a single observation. The more data you have for verification, (as is true in general for other statistical measures) the more certain you are.

Download Presentation

Rank Histograms – measuring the reliability of an ensemble forecast

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Rank histograms measuring the reliability of an ensemble forecast

Rank Histograms – measuring the reliability of an ensemble forecast

  • You cannot verify an ensemble forecast with a single observation.

  • The more data you have for verification, (as is true in general for other statistical measures) the more certain you are.

  • Rare events (low probability) require more data to verify => as do systems with many ensemble members.

From Barb Brown


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

Troubled Rank Histograms

Counts

0102030

Counts

0102030

1 2 3 4 5 6 7 8 9 10

Ensemble #

1 2 3 4 5 6 7 8 9 10

Ensemble #

Slide from Matt Pocernic


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

From Tom Hamill


Rank histograms measuring the reliability of an ensemble forecast

Example of Quantile Regression (QR)

Our application

Fitting T quantiles using QR conditioned on:

Ranked forecast ens

ensemble mean

ensemble median

4) ensemble stdev

5) Persistence

R package: quantreg


Rank histograms measuring the reliability of an ensemble forecast

Step 2: For each quan, use “forward step-wise

cross-validation” to iteratively select best subset

Selection requirements: a) QR cost function minimum,

b) Satisfy binomial distribution at 95% confidence

If requirements not met, retain climatological “prior”

Step I: Determine

climatological quantiles

Probability/°K

climatological

PDF

1.

Regressor set:

1. reforecast ens

2. ens mean

3. ens stdev

4. persistence

5. LR quantile

(not shown)

3.

T [K]

2.

4.

Temperature [K]

observed

forecasts

Time

Step 3: segregate forecasts into differing ranges of ensemble dispersion and refit models (Step 2) uniquely for each range

Final result: “sharper” posterior PDF

represented by interpolated quans

forecasts

Forecast

PDF

posterior

I.

II.

III.

II.

I.

Probability/°K

prior

T [K]

Temperature [K]

Time


Rank histograms measuring the reliability of an ensemble forecast

Rank Probability Score

for multi-categorical or continuous variables


Scatter plot and contingency table

Scatter-plot and Contingency Table

Brier Score

Does the forecast detect correctly temperatures above 18 degrees ?

y = forecasted event occurence

o = observed occurrence (0 or 1)

i = sample # of total n samples

=> Note similarity to MSE

Slide from Barbara Casati


Rank histograms measuring the reliability of an ensemble forecast

Other post-processing approaches …

1) Bayesian Model Averaging (BMA) –

Raftery et al (1997)

2) Analogue approaches –

Hopson and Webster, J. Hydromet (2010)

3) Kalman Filter with analogues –

DelleMonache et al (2010)

4) Quantile regression –

Hopson and Hacker, MWR (under review)

5) quantile-to-quantile (quantile matching) approach –

Hopson and Webster J. Hydromet (2010)

… many others


Rank histograms measuring the reliability of an ensemble forecast

Quantile Matching: another approach when matched forecasts-observation

pairs are not available => useful for climate change studies

ECMWF 51-member Ensemble

Precipitation Forecasts compared

To observations

  • 2004 Brahmaputra Catchment-averaged Forecasts

  • black line satellite observations

  • colored lines ensemble forecasts

  • -Basic structure of catchment rainfall similar for both forecasts and observations

  • -But large relative over-bias in forecasts


Rank histograms measuring the reliability of an ensemble forecast

  • Forecast Bias Adjustment

  • done independently for each forecast grid

  • (bias-correct the whole PDF, not just the median)

Model Climatology CDF

“Observed” Climatology CDF

Pmax

Pmax

Precipitation

Pfcst

Padj

25th

50th

75th

100th

25th

50th

75th

100th

Quantile

Quantile

In practical terms …

ranked forecasts

ranked observations

0

1m

0

1m

Precipitation

Precipitation

Hopson and Webster (2010)


Rank histograms measuring the reliability of an ensemble forecast

Bias-corrected Precipitation Forecasts

Original Forecast

Brahmaputra Corrected Forecasts

Corrected Forecast

=> Now observed precipitation within the “ensemble bundle”


  • Login