huug van den dool dave unger consolidation of multi method seasonal forecasts at cpc part i l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I PowerPoint Presentation
Download Presentation
Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I

Loading in 2 Seconds...

play fullscreen
1 / 34

Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I - PowerPoint PPT Presentation


  • 159 Views
  • Uploaded on

Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I . LAST YEAR / THIS YEAR:. Last Year: Ridge Regression as a Consolidation Method, to yield, potentially, non-equal weights. This Year: see new Posters on Consolidation by

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Huug van den Dool / Dave Unger Consolidation of Multi-Method Seasonal Forecasts at CPC. Part I' - vanya


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
huug van den dool dave unger consolidation of multi method seasonal forecasts at cpc part i
Huug van den Dool / Dave Unger

Consolidation of Multi-Method Seasonal Forecasts at CPC.

Part I

last year this year
LAST YEAR / THIS YEAR:
  • Last Year: Ridge Regression as a Consolidation Method, to yield, potentially, non-equal weights.
  • This Year: see new Posters on Consolidation by

-) Malaquias Pena (methodological twists and SST application) and

-) Peitao Peng (application to US T&P)

  • Last Year: Conversion to pdf as per “Kernel method” (Dave Unger).

This Year: Time series approach is next.

slide3

Does the NCEP CFS add to the skill of the European DEMETER-3 to produce a viable International Multi Model Ensemble (IMME) ?“Much depends on which question we ask”

Input by Suranjana Saha and Ake Johansson is acknowledged.

slide4
DATA and DEFINITIONS USED
  • DEMETER-3 (DEM3) = ECMWF + METFR + UKMO
  • CFS
  • IMME = DEM3 + CFS
  • 1981 – 2001
  • 4 Initial condition months : Feb, May, Aug and Nov
  • Leads 1-5
  • Monthly means
slide5
DATA/Definitions USED (cont)
  • Deterministic : Anomaly Correlation
  • Probabilistic : Brier Score (BS) and Rank Probability Score (RPS)
  • Ensemble Mean and PDF
  • T2m and Prate
  • Europe and United States

Verification Data :

  • T2m : Fan and Van den Dool
  • Prate : CMAP
slide6

BRIER SCORE FOR 3-CLASS SYSTEM

1. Calculate tercile boundaries from observations 1981-2001 (1982-2002 for longer leads) at each gridpoint.

2. Assign departures from model’s own climatology (based on 21 years, all members) to one of the three classes: Below (B), Normal (N) and Above (A), and find the fraction of forecasts (F) among all participating ensemble members for these classes denoted by FB, FN and FA respectively, such that FB+ FN+FA=1 .

3. Denoting Observations as O, we calculate a Brier Score (BS) as :

BS={(FB-OB)**2 +(FN-ON)**2 + (FA-OA)**2}/3,

aggregated over all years and all grid points.

{{For example, when the observation is in the B class, we have (1,0,0) for (OB, ON, OA) etc.}}

4. BS for random deterministic prediction: 0.444

BS for ‘always climatology’ (1/3rd,1/3rd,1/3rd) : 0.222

5. RPS: The same as Brier Score, but for cumulative distribution (no-skill=0.148)

number of times imme improves upon dem 3 out of 20 cases 4 ic s x 5 leads
Number of times IMME improves upon DEM-3 :out of 20 cases (4 IC’s x 5 leads):

“The bottom line”

“ NO consolidation, equal weights, NO Cross-validation”

cross validation cv
Cross Validation (CV)
  • Why do we need CV?
  • Which aspects are CV- ed: a) systematic error correction ( i) the mean and ii) the stand.dev) and b) weights generated by Consolidation
  • How?? CV-1, CV-3, CV-3R
  • Don’t use CV-1!. CV-1 malfunctions for systematic error correction in combination with internal* climatology and suffers from degeneracy when weights generated by Consolidation are to be CV-ed.
  • *Define internal and external climatology
slide11

Last year

wrt OIv2 1971-2000 climatology

slide12

Last year

wrt OIv2 1971-2000 climatology

slide13

w1 w2 w3 w4

nov lead 3 check .35 .04 .27 .35 11 3 1981 left out (and 2 others)

nov lead 3 check .36 .00 .29 .35 11 3 1982 left out (and 2 others)

nov lead 3 check .29 .02 .33 .36 11 3

nov lead 3 check .40 .04 .31 .36 11 3

nov lead 3 check .38 -.01 .28 .34 11 3

nov lead 3 check .37 .00 .27 .33 11 3

nov lead 3 check .35 -.01 .25 .40 11 3

nov lead 3 check .31 .01 .29 .43 11 3

nov lead 3 check .28 .01 .31 .39 11 3

nov lead 3 check .38 .02 .30 .32 11 3

nov lead 3 check .36 .00 .39 .32 11 3

nov lead 3 check .31 .03 .29 .39 11 3

nov lead 3 check .45 -.01 .23 .35 11 3

nov lead 3 check .37 .00 .31 .41 11 3

nov lead 3 check .35 .03 .28 .40 11 3

nov lead 3 check .33 .02 .36 .35 11 3

nov lead 3 check .42 .01 .33 .31 11 3

nov lead 3 check .33 .04 .31 .42 11 3

nov lead 3 check .33 .00 .29 .35 11 3

nov lead 3 check .40 .02 .31 .35 11 3

nov lead 3 check .33 .00 .24 .38 11 3 2001 left out (and 2 others)

Feb forecast has high co-linearity. Model 2 has high –ve weights for unconstrained regression

overriding conclusion
Overriding conclusion

 With only 20+ years of hindcasts it is hard for any consolidation to be much better than equal weight MME. (Give us 5000 years.)

‘Pooling’ data helps stabilize weights and it increases skill, but is it enough?

 20+ years is a problem even for CV-ed systematic error correction.

further points of study
Further points of study
  • The nature of climatology (= control in verification), external, internal, fixed
  • Cross Validation method not settled
  • The Many Details of Consolidation as per Ridge Regression
  • Conversion to pdf can be done in very many different ways (including 3-class BS minimization, logistic regression, ‘count method’, Kernels)
forecast consolidation at cpc part 2 ensembles to probabilities

Forecast Consolidation at CPC – Part 2Ensembles to Probabilities

David Unger / Huug van den Dool

Acknowledgements: Dan Collins, Malaquias Pena, Peitao Peng

objectives
Objectives
  • Produce a single probabilistic forecast from many tools

- Single value estimates

- Ensemble sets

  • Utilize Individual ensemble members

- Assume individual forecasts represent possible realizations

- We want more than just the ensemble mean

  • Provide Standardized Probabilistic output

- More than just a 3-class forecast

ensemble regression
Ensemble Regression
  • A regression model designed for the kernel smoothing methodology

- Each member is equally likely to occur

- Each has the same conditional error distribution in the event it is

closest to the truth.

F= Forecast, σF = Forecast Standard Deviation

Obs=Observations, σObs= Standard Deviation of observations

R=Correlation between individual ensemble members and the observations

Rm = Correlation between ensemble mean and observations

a1 , a0 = Regression Coefficients,

F = a0 + a1 F

time series estimation
Time series estimation
  • Moving Average, Let X11 Be the 10-year running mean known on year 11. N=10

X11 = 1/N(x1+x2+x3+x4+x5+x6+x7+x9+x9+x10)

X12 = X11 + 1/N(x11-x1)

XY+1 = XY + 1/N(xY+1-xY-10)

  • Exponential Moving Average (EMA), α = 1/N

X12 = X11 + α(x11- X11)

XY+1 = (1- α)XY + αxY+1

adaptive ensemble regression
Adaptive Ensemble Regression

EMA estimates

  • F
  • F2
  • (Obs)
  • (Obs)2
  • F (Obs)
  • Fm2
  • (F-Fm)2
trends
Trends
  • Adaptive Regression “learns” recent bias, and is very good in compensating.
  • Most statistical tools also “learn” bias, and adapt to compensate.

Steps need to be taken to prevent doubling bias corrections.

trends continued
Trends (Continued)
  • Step 1. Detrend all models and Obs.

F = F – F10Obs = Obs – Obs10

F10 , Obs10 = The EMA approximating a 10-year mean

  • Step 2. Ensemble Regression

Final forecast set, F are anomalies.

  • Step 3. Restore the forecast.

A) F = F + F10

We believe the OCN trend estimate

B) F = F + C30: C30 = 30-year (1971-2000) Climatology

We have no trust in OCN.

C) F = F +C30+ROCN (F10 –C30) :

ROCN= Correlation ( F10,Obs)

Trust but verify.

weighting
Weighting
  • The chances of an individual ensemble member being “best” increases with the skill of the model.
  • The kernel distribution represents the expected error distribution of a correct forecast.
final forecast
Final Forecast
  • Consolidated Probabilities are the area under the PDF within each of three (Below, Near, Above median ) categories.

Call for ABOVE when the P(above)>36% and P(Below) <33.3%

Call for BELOW when P(Below > 36%) and P(Above) < 33.3%

White area = Equal Chances (We don’t as yet trust our Near Normal percentages)

performance
Performance

Tools 1995 – 2005.

  • CFS – 15 members hindcast

All Members weighted equally with combined area equal to the tool weighting

  • CCA - Single Valued Forecast Hindcasts from Cross Validation
  • SMLR - Single valued forecast – Hindcasts from Retroactive Real-time Validation
  • OCN incorporated with EMA rather than 10-year box car average.
performance continued
Performance (Continued)
  • First Guess EMA parameters provided by CCA, SMLR Statistics 1956-1980.
  • CFS spinup 1981-1994

Validation Period All Initial times, 1-month lead, Jan 1995-Dec 2005 (11 Years)

performance continued29
Performance (Continued)
  • Official Forecast: Hand-drawn, probabilities in 3 classes. PoE obtained from Normal distribution, Standard deviation based on tool skills)
  • CCA+SMLR: A Consolidation of the two Statistical Forecasts, equally Weighted.
  • CFS: A Consolidation of 15 CFS ensembles, equally weighted.
  • CFS+CCA+SMLR Wts: A Consolidation of CCA, CFS, and SMLR, weighted by R/(1-R) for each of the three tools. 15 CFS members each given 1/15th of the CFS weight. Also known as “All”
  • All – Equal Wts: CCA, SMLR and the 15 CFS members combined are given equal Weights.
  • All – No Trend. Anomalies applied to 30-year mean.
performance30
Performance

CRPSS

RPSS - 3

HSS

Bias (C)

% Cover

CCA+SMLR

CFS

CFS+CCA+SMLR, Wts.

All – No Trend

All – Equal Wts.

Official

slide31

CRPS Skill Scores: Temperature

1995 – 2005

Skill

.10

.05

.01

All

CCA+SMLR

1-Month Lead, All initial times

CFS

Official

slide32

CRPS Skill Scores / % cover: Temperature

1995 – 2005

Skill

.10

.05

.01

Trends

No Trends

CRPSS

1-Month Lead, All initial times

%cover

the winter forecast
The Winter Forecast

Skill Weighting

Equal Weighting

conclusions
Conclusions
  • Weighting is not critical – within reason.
  • Consolidation outperforms component models .
  • Getting the trends right is essential.
  • CFS + Trend consolidation provides an accurate continuous distribution.