1 / 23

Assessing the skill of decadal predictions Reidun Gangst ø, Andreas P. Weigel, Mark A. Liniger

Assessing the skill of decadal predictions Reidun Gangst ø, Andreas P. Weigel, Mark A. Liniger. EMS Annual Meeting, Berlin, 13 September 2011. Outline. The ENSEMBLES decadal predictions Impact of drift correction on skill Is there any skill apart from the trend ?

pepper
Download Presentation

Assessing the skill of decadal predictions Reidun Gangst ø, Andreas P. Weigel, Mark A. Liniger

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing the skill of decadal predictionsReidun Gangstø, Andreas P. Weigel, Mark A. Liniger EMS Annual Meeting, Berlin, 13 September 2011

  2. Outline • The ENSEMBLES decadal predictions • Impact of drift correction on skill • Is there any skill apart from the trend? • Impact of cross-validation on skill • Evaluating skill with the Jackknife bias corrector • Summary and conclusions

  3. Global average T2 temperature(ENSEMBLES decadal predictions vs ERA-40/Interim re-analysis data) ECMWF UKMO T2 (°C) IFM-GEOMAR CERFACS T2 (°C) Hindcast year Hindcast year

  4. Problem: sample size too small (8-9) to obtain robust bias estimates for each year separately Drift correction methods: • Subtracting the lead-time dependent bias (CONV) • Fitting a 3rd degree polynomial fit to the lead-time dependent bias (FIT) Example of drift evolutionwith lead-time Crosses: CONV solid lines: FIT T2 temperature (°C) Lead-time (year) The drift correction is done in a leave-one-out cross-validation mode

  5. Global mean T2, after drift correction ECMWF UKMO T2 (°C) Multi-model IFM-GEOMAR CERFACS T2 (°C) Hindcast year Hindcast year

  6. Correlation after drift correction(in cross-validation) Correlation, FIT (T2 mean over years 1-5) Mean of grid point-wise correlation Latitude Correlation Lead-time (year) Longitude

  7. Removing the model trend 1-5 y 1-5 y All lead-times 1-10 y 1-5 y T2 temperature (°C) 6-10 y 6-10 y 6-10 y Year

  8. Removing the observed trend 1-5 y 1-5 y 1-5 y - 6-10 y 6-10 y 6-10 y -

  9. Correlation after detrending (drift correction with CONV, in cross-validation) Correlation, model trend removed (yrs 1-5) Mean of grid point-wise correlation Latitude Correlation Lead-time (year) Longitude Why is the skill predominantly negative???

  10. Cross-validation … then correlate with observations 1960, 1965, 1970, …

  11. Toy model: bias from cross-validation Correlation as measured Prescribed correlation: 0 Number of experiments: 10’000 Var.fcst / Var.obsv 1:12 Drift-correction (method: CONV) in cross-validation

  12. Illustration of cross-validation bias,example: CONV drift correction Forecasts Obsv. Not bias corrected NO CORRELATION

  13. Illustration of cross-validation biasexample: CONV drift correction Forecasts Obsv. Not bias corrected Bias corrected NO CORRELATION

  14. Illustration of cross-validation biasexample: CONV drift correction Forecasts Obsv. Not bias corrected Bias corrected NO CORRELATION

  15. Illustration of cross-validation biasexample: CONV drift correction Forecasts Obsv. Not bias corrected Bias corrected NO CORRELATION

  16. Illustration of cross-validation biasexample: CONV drift correction Forecasts Obsv. Not bias corrected Bias corrected NEGATIVE CORRELATION NO CORRELATION

  17. Consequences for verification • Estimates of actual prediction skill of decadal forecasts problematic because: • Issues of data situation in hindcasts (e.g. ocean data before 1980s) • small sample size induces bias in cross-validation procedure • It may be better to look at potential predictability, i.e. the skill we would have with an infinite number of training data, and assuming that there are no limitations in data quality

  18. Jackknifing as a pragmatic solution • Empirical approach frequently used to quantify sample size related biases • Related to bootstrapping • The idea is that the estimator is computed from the full sample, then recomputed n times, leaving a different observation out each time • Reference: B. Efron (1982). The Jackknife, the Bootstrap and other resampling plans. J.W. Arrowsmith, Ltd., Bristol, England, 92 pp.

  19. Toy model: bias from cross-validation Jackknife estimate Correlation as measured Prescribed correlation: 0 Number of experiments: 10’000 Var.fcst / Var.obsv 1:12 Drift-correction (method: CONV) in cross-validation

  20. Local correlation after drift correction,with the Jackknife bias corrector (JK) applied Correlation, CONV, with CV Correlation with JK (T2 mean over years 1-5) Latitude Mean of grid point-wise correlation Correlation Longitude Lead-time (year)

  21. Local correlation after detrending, with the Jackknife bias corrector applied Correlation CONV, with CV Correlation with JK, model trend removed (yrs 1-5) Latitude Mean of grid point-wise correlation Correlation Longitude Lead-time (year)

  22. Difference in correlation between the detrending methods Uncertainties related to the choice of detrending method are of the same order of magnitude as remaining fluctuations

  23. Summary and conclusions • Predicted near-surface temperature from the ENSEMBLES decadal model forecasts are compared to ERA-40/Interim re-analysis data • Drift correction: • Reduction of noise by fitting suitable polynomial through annual bias estimates • Verification: • Unbiased estimate of forecasts problematic due to small sample sizes • It may be more useful to focus on potential predictability (e.g. Jackknife method) • Trend: • By far most of the skill is related to reproduction of linear trend • Skill of predicting remaining (interannual) fluctuations close to zero • Exact quantification difficult due to uncertainties in detrending methods

More Related