Theoretical Review of Seasonal Predictability In-Sik Kang

1. Theoretical Review of Seasonal Predictability In-Sik Kang

2. (b) Forced variance Forced variance Climate signals caused by external forcing (a) Total variance (c) Free variance Free variance Intrinsic transients due to natural variability Analysis of Variance of JJA Precipitation Anomalies (SNU case)

3. Signal-to-noise ForcedVariance InternalVariance

4. Decomposition of climate variables • Climate state variable (X) consists of predictable and unpredictable part. • Predictable part = signal (Xs) : forced variability • Unpredictable part = noise (Xn) : internal variability • X = Xs + Xn • The dynamical forecast (Y) also have its forced and unforced part. • forecast signal (Ys) : forced variability of model forecast noise (Yn) : internal variability of model • Y = Ys + Yn The internal variability (noise) is stochastic If the forecast model is not perfect, Xs≠Ys. (there is a systematic error)

5. Prediction skill Noise and Error are not correlated with others Alpha : regression coeff. of signal Cor(x,y)= The correlation coefficient is maximized by removingV(ye)andV(yn) The most accurate forecast will be the SIGNAL of perfect model.

6. Maximum prediction skill : potential predictability When the forecast is perfect signal, the correlation coefficient is Cor(x,y)= Maximum prediction skill (= potential predictability of particular predictand) is a function of Signal to Noise Ratio

7. Perfect model correlation & Signal to Total variance ratio Z500 winter (C20C, 100 seasons, 4 member) Although the 4 member is not enough to estimate Potential predictability precisely, the patterns of 2 metrics are quite similar

8. Strategy of prediction The strategy of seasonal prediction is to obtain “perfect signal” as close as possible. (i.e. reducing variance of systematic error and variance of noise) • 1. Reduction of Noise • Averaging large ensemble members • (if number of ensemble members is infinte, Noise will be zero in the ensemble mean) • 2. Correct signal • Improving GCM • Statistical post-process (MOS)

9. Ensemble averaging : reduction of noise Correlation with perfect forecast (perfect signal) N : number of ensemble member Necessary number of ensemble is dependent on the signal to noise ratio Extratropical forecast needs larger ensemble members than tropics.

10. Multi Model Ensemble The averaging of large ensembles reduces noise in the forecast. The multi model ensemble combines ensembles with different forecast signal, the cancellation of systematic bias is possible : a sort of post processing. Thus the multi model ensemble (MME) can be more efficient ensemble technique to get “perfect signal” : it permits both of noise reduction and signal correction. And statistically optimized MME technique (eg. superensemble) can be more beneficial in the correction of systematic error like as usual statistical post-processes. However, there are still debates on benefit of multi model ensemble in seasonal prediction.

11. MMES (based on point-wise correction, CPPM) □ Pattern Correlation (0-360E, 40S-60N) 0.43, 0.44, 0.53 (a) 850 hPa Temperature (b) Precipitation 0.40, 0.47, 0.58 Single model MME1 MME2 MME3

12. Issues on Multi Model Ensemble prediction • Debates on MMEP of Seasonal forecast • Is a multi model system better than a single good model? (Graham et al. 2000; Peng et al. 2002; Doblas-Reyes et al. 2000) • Is a sophisticated technique better than a simple composite? (Krishnamurti et al. 2000; Kharin and Zwiers 2002; Pavan and Doblas-Reyes 2000 ) Strong limitation of seasonal predictability study : small samples MME prediction experiment in a simple climate system (Krishnamurti et al. 2000; Palmer 1993, 1999; Qin and Robinson 1995)

13. Multimodel ensemble forecast in simple model  Design of the simple climate model and predictability experiment Simple Chaotic Model 1 Low frequency forcing Simple Chaotic Model 2 • Equivalent to the seasonal forecast using multi-AGCMs with prescribed SST Simple Chaotic Model : Simple Chaotic Model 9 Forced variability Internal variability Different parameters Atmospheric process  Simple chaotic model : 2 layer Nonlinear QG spectral model, Reinhold & Pierrehumbert (1982) Time varying forcing (Interannual time scale)

14. Multimodel ensemble forecast in simple model Observation Prediction (20 ensembles) Ens. mean Case 2 Case 1 Physical parameters of each models Prediction experiments : 120cases, 20 ensembles

15. Interannual variability : a particular wave component Obs model Fcst (ensemble member)

16. Multimodel ensemble prediction schemes • MME1 : simple composite of individual forecast with equal weighting. (special case of MME2) • MME2 (Superensemble) : Optimally weighted composite of individual forecasts. The weighting coefficient is defined by regression of forecasts and observation during training period. • MME3 : simple composite of individual forecasts, which was corrected by statistical post process ■ Statistical correction (Kang et al. 2004) Corrected forecast Coupled Pattern Observation SVD Projection coeff. Hindcast New Forecast Coupled Pattern

17. Prediction skills of single and multi model ensemble Pattern correlation of indiv. Model and MME (60case avg.) MME is not better than a single good model !! It is due to the inclusion of bad model (M2, M6) What will happen if bad model excluded in MME? MME3 corrected MME1 MME2

18. Which models? : combination of models Comparing 219 combinations (3 to 8 models) Good combinations (MME1) Bad combinations (MME1) • Combining all available models does not guarantee best skill • “Systematically” bad model needs to be excluded. Good combinations (MME2) Bad combinations (MME2) • “Systematically” bad model can be useful : difficult to find good combination

19. Number of models MME1 MME2 MME3 NT=30 NT=60 NT=90 30 case mean skill averaged over each number of models • Composite forecast : more models, better forecasts • Due to overfitting, regression based forecast getting worse with increasing number of models. • When the signal to noise ratio is large, superensemble is more skillful than simple composite.

20. Results of simple model experiments • Debates on MMEP of Seasonal forecast • Is a multi model system better than a single good model? It depends on the combination of models, if there are sysetmatically bad model, MME is not better than a single good model. • Is a sophisticated technique better than a simple composite? When the signal to noise ratio is small, superensemble tends to be unstable due to overfitting. On the other hand, superensemble can be better than a simple composite where the signal to noise ratio ( potential predictability is high) and number of model is not large.

21. Summary • Due to the unpredictable noise, the most accurate deterministic forecast is a perfect signal. (ensemble mean of perfect model) • To obtain perfect signal, we have to - reduce noise in the forecast - correct forecast signal (systematic error correction) MME is a reasonable approach to the perfect signal • Composite based MME has some dependency on the combination of models : need to exclude bad models. • Complex MME (Superensemble) have a problem of overfitting in the case of low signal to noise ratio and short historical record. • Generally, Composite based MME is more feasible and skillful. The composite of calibrated forecast is more beneficial.

22. Probabilistic Seasonal Forecastsand the Economic value

23. Forecast value Resolution → Value of the forecast : accuracy & utility Forecast is valuable when it is used by decision maker and has some benefits. Forecast Information $, ¥, ₩,  Decision making Forecast value = Accuracy x Utility Accuracy Utility Resolution → Resolution →

24. Choice of forecast system Forecast system Forecast system Forecast system Forecast system Forecast system Choice : the most Valuable forecast system The form and properties should be matched by a particular situation of user Decision maker • Climatological probability of event : Pc • Cost-Loss ratio of user : C/L • Flexibility of interpretation by user : Pt

25. Probability distribution of summer mean precipitation □Distribution of total ensemble members OBS Single model Multi model

26. Ensemble PDF of particular year A B Climatological PDF C 0 Xc -Xc Probability formulation A Probability of ABOVE normal B Probability of NORMAL C Probability of BELOW normal

27. Model 1 Model 2 Model 3 MME1 Number of occurrence Value Multi model ensemble probability • • • MME1 Probability Collecting all normalized ensemble members (5 model , 45 samples) μ MME3 Probability Change the ensemble mean with MME3 in deterministic forecast μ : ensemble mean μ *: corrected ensemble mean μ μ*

28. Reliability Diagram (Above normal) (a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N) Obs. probability Fcst probability Fcst probability Single model MME1 MME3

29. Reliability Diagram (Below normal) (a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N) Obs. probability Fcst probability Fcst probability Single model MME1 MME3

30. Economic Value of Prediction System CONTINGENCY TABLE OF C/L Forecast/action ECONOMIC VAULE : V Yes No Yes Observation No C: Total cost r=C/Lp Lu: Unprotectable loss o: the climatetological Lp: Protected against frequency of the event Economic Value as a Function of Cost/Loss Ratio (GCPS) Global East Asia West US Australia Economic Value

31. Economic value of deterministic forecast (Above normal) (a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N) C/L Single model MME1 MME2 MME3

32. Economic value of Probabilistic forecast (Above normal) (a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N) Single model Black line: MME3 in deterministic forecast MME1 MME3

33. Economic value of forecast (above normal, C/L=0.1) Deterministic Probabilistic

34. Generalized application of forecast • Application to the simple decision system : Y or N – 1bit problem. • Probabilistic forecast is converted deterministic forecast using pt • Benefit of probabilistic forecast cannot be maximized. • Application to the generalized decision system : Action function & Contingency map • Action function • Contingency map • Action using Det. Forecast : F(T0) • Action using Prob. Forecast : ∫p(T)F(T)dT Net benefit Temperature (T) Observation Action function : F(T) Probability forecast p(T) Deterministic forecast (T0) Forecast Forecast utilization covering wide range of problem