Malaquias Pe ñ a and Huug van den Dool

1. NCEP Climate Meeting, April 4, 2007 Consolidation of Multi Method Forecasts Application to monthly predictions of Pacific SST Malaquias Peña and Huug van den Dool Acknowledgments: Suru Saha retrieved and organized the data, Dave Unger and Peitao Peng provided discussion to the subject

2. DATA • Forecasting tools: 8 CGCMs, 1 Statistical model • NCEP CFS: 1981-2006, 15 membs, 9 leads • DEMETER : 1980-2001, 9 membs, 6 leads • ECMWF • MPI • MF • UKMO • INGV • LODYC • CERFAX • CPC’ Constructed Analog (CA) : 1956-2006, 12 membs,12 leads This is what all have in common: • Monthly Forecasts, leads 0 to 5 • Initial months: Feb, May, Aug, Nov • Length of retrospective forecasts: 21 years, 1981-2001 FOCUS: TROPICAL PACIFIC SST: 12.5 S TO 12.5 N

3. Consolidation: Making the best single forecast out of a number of forecast inputs. Objective consolidation necessary as large supply of forecasts are available. If K is the number of participant forecast systems, ζ, predicting a particular target month with a given lead time, the consolidation is the following linear combination: DEFINITIONS • For convenience, systematic errors and observed climatology are removed in ζ. • The regression coefficients (weights), α, are based on past performance of the forecast system. • o is the verifying field (e.g. observation; climatology removed). Suppose there are N cases of retrospective forecasts, then one can train a consolidation method by comparing:

4. OPTIMIZING WEIGHTS • Find weights, αi ,for each forecasting tool, ζi, that minimizes the (sum of square of) errors εj in Where Z is a matrix whose columns are the forecasting tools and rows are the data points in the training period, o is the column vector containing the verifying field, and εis a vector of errors. • Least square method (unconstrained regression):

5. ILL-POSED MATRIX PROBLEM too large Corresponding weights for UR for lead 1, im 1

6. RIDGE REGRESSION Minimize: Constrained to: leads to Ridge Regression (DelSole, 2007) (ad hoc) and where • Van den Dool estimates  such that the weights are small and stable • Many more ways to find it • Depends on characteristics of covariance matrix ZTZ

7. RIDGE REGRESSION RID RIM RIW λ λ λ • Model weights (αi, i=1..9) as a function of λfor three ridge consolidation methods. • Figure illustrates asymptotic values. Our methods stop at λ=0.5. • Unconstrained regression (λ=0) results in a wide range (including negative values) of weights.

8. CONSOLIDATION METHODS ASSESED

9. CROSS-VALIDATION Anomaly Pattern correlation over the tropical Pacific. Average for all leads and initial months. Empty bar: Full (dependent), filled bar: 3-yr out cross-validated.

10. GRIDPOINT BY GRIDPOINT PERFORMANCE

11. EQUATORIAL PACIFIC

12. WESTERN TROPICAL PACIFIC Trust in good models when performed well in a gridpoint. It goes to the opposite direction of the bad models

13. WESTERN TROPICAL PACIFIC MIXES CLOSEST NEIGHBORING GRIDPOINT Trust in good models when performed well in a 3x3 box of gridpoints. It goes to the opposite direction of the bad models

14. WESTERN TROPICAL PACIFIC DOUBLE PASS AND MIXES CLOSEST NEIGHBORING GRIDPOINT Trust less good models, damps towards climatology as negative weights are set to zero

15. INCREASING EFFECTIVE SAMPLE Tropical Pacific SST. AC average for all leads and initial months • 1 GRIDPOINT BY GRIPOINT • 3X3 BOXES • 5X5 BOXES • ALL GRIDPOINTS IN THE DOMAIN • GRIDPOINTS IN AND OUT DOMAIN AC Skill of most consolidation methods improve when effective sampling size increases Multi-methods average

16. INCREASING EFFECTIVE SAMPLE Consistency: Percentage cases (leads and initial months) outperforming MM

17. RELATIVE OPERATIONING CURVES • Assess the ability to anticipate correctly the occurrence or non occurrence that SST anomalies will fall in the upper, middle and lower terciles. • Class limits defined by the observed SST during the training period • Probability information from the ensemble: counting the fraction of ensemble members that falls into the “above-normal”, “near-normal”, and “below-normal” categories, and interpreting this fraction as the probability that forecasts will fall in such categories. • Approach for the optimized weights: each ensemble member forecast is multiplied by normalized weights. Upper tercile Lead 3 Lower tercile

18. UPPER TERCILE

19. LOWER TERCILE

20. SUMMARY • All the points below are for the particular case of SST anomalies in the tropical Pacific. • Forecasts arising from a combination of multiple models of similar skill generally outperform those from individual models but not UR after CV-3. • Even the simple average of multi-methods shows consistent improvement over individual participant models. • Over all and after cross-validation, sophisticated consolidation methods marginally improve over the simple average. • Increasing the effective sampling size increases the skill and consistency of consolidation methods. • Consolidation methods improve significantly over the multi-methods average in the western Pacific. • Probabilistic assessment, as measured by ROC shows some improvement of consolidation methods over MM. Construction of the probability density function of the consolidation requires optimization.