Outline • Introduction • Model and Data • Results • ● Deterministic skills • ● Probabilistic skills • 4. Summary The Assessment of Seasonal Forecast Skills from the Canadian HFP Q. Teng1, S. Kharin1, F. Zwiers1 and X. Zhang2 1Canadian Centre for Climate Modelling and Analysis, Meteorological Service of Canada 2CCRM, Climate Research Branch, Meteorological Service of Canada
multi-model ensemble prediction ► initial conditions; ensemble prediction ► model error multi-model prediction ► deterministic measures: e.g., mean square error (MSE); root mean square error (RMSE); root mean square skill score (RMSSS); anomaly correlation coefficient (ACC) etc. ► probabilistic measures: e.g., Brier score (BS); Brier skill score (BSS); attributes (or reliability) diagram; relative operating characteristic (ROC) and its skill score etc. Introduction ●Basic premise: Slow variations in lower-boundary forcing (e.g., SST; sea-ice cover and temperature; land surface). ●Main sources of error: ●Forecast verification:
●Verification datasets: NCEP/NCAR (Kalnay et al. 1996) and ERA-40 (Simmons and Gibson 2000) reanalyses fields ●Variables: Z500 ,T700 & T2mfor MAM, JJA, SON, DJF; (1969-1995) Model and Data ●Model Output
GCM2 SEF GEM GCM3 Bias -- Z500 (DJF)
NAM NEX TR GL ERA-40 purple: GCM2 green: SEF yellow: GEM red: GCM3 RMSE -- Z500 NCEP North America (NAM): 20° - 80°N, 150° - 45°W; Northern Extra-tropics (NEX): 30° - 87.5°N, 180°E - 180°W; Tropics (TR): 30°S - 30°N, 180°E - 180°W; Globe (GL): 87.5°S - 87.5°N, 180°E - 180°W. Canada (CA): 40° - 87.5°N, 150° - 45°W; Pacific-North-America (PNA): 20° - 80°N, 180° - 60°W; Northern Hemisphere (NH): 0° - 87.5°N, 180°E - 180°W;
NAM NEX TR GL Z500 GCM2; SEF; GEM; GCM3 T700 T2m RMSE
is the MSE of probability forecast; where is the forecast probability; is the event variable: Brier Score (BS) and Brier Skill Score (BSS) ●BS ●BSS
Where Brier Skill Score (BSS) -- Cont’d ●BSS ●Three methods to estimate P (Kharin and Zwiers 2003):
BSS -- Z500 North America BNNNAN MAM JJA SON DJF
BNNNAN MAM JJA SON DJF BSS -- Z500 Tropics
BSS -- Time Series (DJF Tropics) Years The year refers to the Dec. of the corresponding winter.
Where is the number of probability bins; denotes the total number of forecast/event pairs; represents the number of times each probability forecast is used; is the observed sample relative frequency; is the observed overall relative frequency or sample climatology. Decomposition of BS Following Murphy (1973), the BS can be expressed as: reliability resolution uncertainty
Attributes Diagram -- T2m Tropics (DJF) Count Gaussian Improved
ROC -- Z500 North America (DJF) Count Gaussian Improved
ROCSS Z500T700 T2m
Summary ●The identity of the best single model varies; ●The multi-model ensemble is superior to the single model ensembles; ●The raw Gaussian fit method is generally better than the count method, while the statistically improved technique may not always be beneficial.
Acknowledgments Juan-Sebastian Fontecilla, Normand Gagnon, Fouad Majaess and Steve Lambert.