Lorenz 1963 Multi-model Ensemble Test-bed

1. Christine Johnson NAEFS Teleconference September 2006 Lorenz 1963 Multi-model Ensemble Test-bed

2. Research Aim • The Met Office is currently developing a real-time multi-model ensemble using the TIGGE data. • Which methods should be used to calibrate and combine the multi-model ensemble? • Evaluate the benefits of the multi-model ensemble. • Investigate within an idealized framework using the Lorenz 1963 model.

3. Lorenz 1963 multi-model ensemble testbed Near =3/2 Near =/2 • The 1963 Lorenz equations, with a dimensional scaling so that the fast-timescale is similar to the synoptic timescale of the atmosphere. • Add ‘seasonally’ dependent biases that are sinusoidal and dependent on , which varies from 0 to 2 in one ‘model-year’ (300 days). • Generate 8-member, 15-day ensembles every 12 hours. Truth Model 1 Model 2

4. Ensemble members Truth Control Member • The models give different forecasts. • The true state tends to lie within the ensemble. • The ensemble spread is representative of the forecast error.

5. RMS Errors Control Mean Spread Climate • The deterministic (control) forecast error is the same as that for climate, by 15 days. • The ensemble mean is more accurate by the end of the forecast. • The ensemble spread is too small at the end of the forecast.

6. Calibration and Combination Data Use previous forecasts (in the blue box ) together with observations, to calibrate and combine the new forecast (red). Use a 50 day moving calibration window.

7. Calibration Method Choose the values of 0 and 1 to minimize the mean square error (MSE) over the calibration window (linear regression). Choose the value of  so that the ensemble variance is equal to the MSE of the bias-corrected ensemble mean.

8. Rank Histograms With bias correction and variance inflation No bias correction or variance inflation With bias correction There is no bias, and the verification lies within the ensemble. Ensemble is under-dispersive, so either all the members are larger or all the members are smaller than the verification. Most ensemble members are smaller than the verification: negative bias. Note: Rank i means that i ensemble members are larger than the verification

9. Impact of the position of the window What is the impact of the position of the calibration window, relative to the forecast? The best result is with a CENTRED window. It is difficult to correct the errors at LONG LEAD TIMES.

10. Bias coefficients Bias coefficients show the general seasonal variation. Bias coefficients have more variability at longer lead times and therefore are more sensitive to the calibration sample.

11. Use of reforecast data Following Hamill et al 2004, use 20 years of forecast data with a 50 day window centred on the forecast to be verified. Raw data With bias correction Bias correction gives a reduction in the errors at long lead times, but not as large as for the centred moving window.

12. Summary: Calibration • Bias coefficients exhibit large fluctuations at long lead times due to flow-dependent errors. • We can only expect realistic moving-window bias correction to correct the errors at short lead times. • Reforecast data can be used to correct the seasonally varying component at long lead times. A centred moving-window is needed to correct the flow dependent component.

13. Combination Method Assume that the multi-model pdf is formed by the sum of the pdfs of each single-model (Raftery et al 2005). Estimate the model-dependent weights, wk.

14. Use of the model-dependent weights wk Multi-model ensemble mean: Multi-model ensemble probabilities (Stefanova and Krishnamurti 2002): Where and are the single-model means and probabilities respectively.

15. Combination methods summary Note: For a fair comparison, the multi-model ensembles has the same number of members as the single model ensembles (8 – member).

16. Bayesian model averaging In a similar way to a mixture-density problem, estimate the weights and variances using the expectation-maximization algorithm The BMA estimated weights are sensitive to the sample due to ill-conditioning. Hence it is important to add prior information to the log-likelihood function, in the form of a beta-distribution on w (Fraley and Raftery, 2005): No prior With prior

17. The model-dependent weights • All three methods give similar estimates for the model dependent weights. • The weights show the ‘seasonal’ variation, despite having been bias-corrected. RMS-based Multiple-regression BMA

18. Combination methods results Single-model with bias correction & variance inflation The simple combination gives an improvement in RESOLUTION. The model-dependent weights gives an improvement in RELIABILITY. Resolution It is difficult to determine the best combination method. Reliability

19. Impact of the position of the window Reliability RMS error of Mean The model-dependent weights have more impact in improving the RMS errors and reliability when using a centred window. The multiple-regression method gives slightly better results using the ideal centred window. Realistic window Centred window

20. Summary: combination • Model combination gives improvements in resolution • Model-dependent weights give improvements in reliability. • All 3 methods give similar estimates for the weights.

21. Conclusions and future plans The results show that model combination gives the most improvements to the ensemble resolution. Calibration and model-dependent weights give improvements in the reliability. In the Met Office multi-model ensemble: • Apply Kalman-filter type bias correction. • Use the most simple method (skill-based) for estimating model-dependent weights. • Consider spatial correlations.

22. References Hamill, T. M., J. S. Whitaker and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Weather Rev., 132, 1434-1447. Fraley C. and A. Raftery, 2005: Bayesian regularization for normal mixture estimation and model-based clustering. Technical report 486, Dept. Statistics. Univ. Washington. Raftery A., T. Gneiting, F. Balabdaoui and M. Polakowski , 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Weather Rev.,133, 1155-1174. Stefanova L. and T. N. Krishnamurti, 2002: Interpretation of seasonal climate forecast using Brier skill score, the Florida state university superensembles and the AMIP I dataset. J. Climate, 15, 537-544.