Exploring sample size issues for 6-10 day forecasts using ECMWF’s reforecast data set

1. Exploring sample size issues for 6-10 day forecasts using ECMWF’s reforecast data set • Model: 2005 version of ECMWF model; T255 resolution. • Initial Conditions: 15 members, ERA-40 analysis + singular vectors • Dates of reforecasts: 1982-2001, Once-weekly reforecasts from 01 Sep - 01 Dec, 14 total. So, 20*14 ensemble reforecasts = 280 samples. • Data sent to NOAA / ESRL : T2M and precipitation ensemble over most of North America, excluding Alaska. Saved on 1-degree lat / lon grid. Forecasts to 10 days lead. Tom Hamill and Jeff Whitaker, NOAA/ESRL Data courtesy of Renate Hagedorn & ECMWF

2. What we did • Considered 6-10 day forecasts of T2m and precipitation (longest-possible lead from this data set). • Relevance to weeks 2, 3, 4 forecast? Your guess is as good as mine (I think probably some relevance, less for week 4 than week 2). • Experiments: • N-member reforecast, N members real time (N=1, 3, 5, 7, 9, 11, 13, 15) • N-member reforecast, 15 members real time • Use established statistical calibration procedures

3. Observation locations for2-meter temperature calibration Uses stations from NCAR’s DS472.0 database that have more than 96% of the yearly records available, and overlap with the domain that ECMWF sent us. (Note: precipitation calibration based on NARR over CONUS)

4. T2m calibration procedure: “NGR”“Non-homogeneous Gaussian Regression” • Reference: Gneiting et al., MWR, 133, p. 1098 • Predictors: ensemble mean and ensemble spread • Output: mean, spread of calibrated normal distribution • Advantage: leverages possible spread/skill relationship appropriately. Large spread/skill relationship, c ≈ 0.0, d ≈1.0. Small, d ≈ 0.0 • Disadvantage: iterative method, slow…no reason to bother (relative to using simple linear regression) if there’s little or no spread/skill relationship.

5. Also: T2m calibration procedure:linear regression • Predictors: ensemble mean of lowest sigma-layer temp • Output: predicted mean and standard deviation where  is determined by and y denotes the observations and S the training sample size

6. Training data forNon-homogeneous Gaussian Regression(all cross validated) • 01 Sep: 01 Sep, 08 Sep, 15 Sep • 08 Sep: 01 Sep, 08 Sep, 15 Sep, 22 Sep • 15 Sep: 01 Sep, 08 Sep, 15 Sep, 22 Sep, 29 Sep • 17 Nov: 03 Nov, 10 Nov, 17 Nov, 24 Nov, 01 Dec • 24 Nov: 10 Nov, 17 Nov, 24 Nov, 01 Dec • 01 Dec: 17 Nov, 24 Nov, 01 Dec Use a centered training data set for weeks 3 - 12, uncentered for weeks 1, 2, 13, and 14

7. T2m results, same ensemble size reforecast as real-time forecast Notes: Some sensitivity to ensemble size; more members clearly better, most of benefit by 11 members. (2) Linear regression slightly better for small ensemble size, NGR slightly better for large ensemble size

8. T2m results, smaller reforecast than 15-mbr real-time forecast Notes: NGR line replicated from previous plot for sake of comparison. Linear regression from with coefficients developed from 3-member reforecastand applied to15 members real time provides almost all of the benefit of full 15-member reforecast. (2)

9. Precipitation forecast calibration: logistic regression Given predictors x1, … , xN (such as the ensemble- mean), find regression coefficients 0, 1, …, N for the equation This generates an S-shaped curve (here for one predictor) Probability Predictor value

10. Precipitation calibrationtraining procedure • Cross-validate (for example, 1983 forecasts use 1982, 1984-2001). • Use all fall season data together, unlike temperature (1 Sep forecasts use 1 Sep - 1 Dec training data). [seasonal biases assumed less important than training sample size] • Sole predictor: (ens. mean precip)0.5

11. Increasing logistic regressionsample size by compositing data from different locations Big dot: location to perform logistic regression. Small dots: grid points with similar observed climatologies, used to augment training sample at big dot’s location. Constrained so that the analog composite locations can’t be too near to each other. Sub-optimal (what if forecast climatologies differ? What if forecast/observed correlations differ? These not accounted for in choosing analog locations.)

12. Precipitation calibration example

13. Reliability diagrams 15-member, from raw ECMWF ensemble 15-member reforecast / 15-member real-time calibrated

14. Precipitation Brier skill scores Again, fewer members are needed in reforecast, as long as real-time forecast is larger. Most of the benefit achieved with 5-7 members in the reforecast (larger than the 3 members with temperature calibration)

15. Comments from Renate Hagedorn, ECMWF • “The results itself are pretty much consistent with my results on the importance of the number of ensemble members in the training data set. I've also seen that 5 members are already quite sufficient and increasing the number to 15 doesn't give much benefit. In contrast to that, the number of years seems to be more important. Since increasing the reforecasts from 5 to 15 members is obviously very expensive (and doesn't seem to be justified very much) we'll go for a 5-member reforecast ensemble in our new system.” • “Why you don't use ECMWF monthly forecast / reforecast data if you are interested in week 2,3,4 aspects? This could help with the problem/question of relevance of the 6-10 day forecasts.”

16. Reconfiguration of CFS?(intended as a starting point for discussion) • Real-time: • Planned : 4x/day to 9 months (=36 months/day) • Reconfigured : 2x/day, 10 members out to 1 month, then single member to 9 months (2*(10+8) = 36 months/day) • Reforecasts • Planned: 1 run/day to 9 months (9 months/day) • Reconfigured: • 10-member ensemble to 1 month every 2nd day (alternating 00Z, 12Z) = 5 months/day • 1 member extending for 2-9 months every other day = 4 months/day • Total = 5 + 4 = 9 months/day

17. Conclusions • Assuming 15-member real-time forecast: • 3-member reforecast sufficient for calibration of 6-10 day temperatures • 5-7 member reforecast sufficient for calibration of 6-10 day precipitation • Relevance to calibration of weeks 2, 3, and 4? (perhaps could explore ECMWF’s monthly data set for greater relevance). • Reconfiguration/supplementation of CFS to improve sub-seasonal forecasts should be discussed.