Automated Homogenization of Monthly Temperature Series via Pairwise Comparisons

1. Automated Homogenization of Monthly Temperature Series via Pairwise Comparisons Matthew Menne and Claude Williams NOAA/National Climatic Data Center Asheville, North Carolina USA

2. Outline • Motivation: The United States Historical Climatology Network (U.S. HCN) • Overview of the “pairwise” homogenization algorithm • Some examples • Impact of inhomogeneities on U.S. temperature trends • A word about GHCN-Daily

3. U.S. Climate Network Historical Climatology Network Cooperative Observer

4. U.S. HCN -- Version 1Monthly Data • 1221 stations selected to comprise the HCN in mid-1980s • Monthly dataset originally released in 1987 • Addressed the following: • Time of observation bias (Karl et al. 1986; Vose et al. 2003) • Station History Changes (Karl and Williams 1987) • Optimized reference series based on station history archives • Urbanization (Karl et al. 1988) • LiG to MMTS instrument change (Quayle et al. 1991)

5. U.S. HCN -- Version 2Monthly Data • 1218 stations in a re-defined network • Addresses • Time of observation bias (Karl et al. 1986; Vose et al. 2003) • Station history (documented) and undocumented changes (Menne and Williams, Journal of Climate, in review) • Automated pairwise comparison of series

6. Station SitingExample of ratings assigned by Watts based upon NOAA/NCDC criteria Class 1 - Flat & horizontal ground. Sensors located at least 100 meters from artificial heating Class 2 - Same as Class 1, except no artificial heating sources within 30 meters. Class 3 - Same as Class 2, except no artificial heating sources within 10 meters. Siting Classification based upon standards for NOAA’s U.S. Climate Reference Network ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/documentation/program/X030FullDocumentD0.pdf Class 5 - Temperature sensor located next to/above an artificial heating source Class 4 - Artificial heating sources <10 meters. www.surfacestations.org

7. Station SitingExample of ratings assigned by Watts based upon NOAA/NCDC criteria Siting Classification based upon standards for NOAA’s U.S. Climate Reference Network ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/documentation/program/X030FullDocumentD0.pdf Class 5 - Temperature sensor located next to/above an artificial heating source Class 4 - Artificial heating sources <10 meters.

8. Why Pairwise? • Avoid problems associated with reference series, e.g., • Difficulties in ensuring homogeneity • Mix of record lengths in climate series • All temperature series can be evaluated

9. Pairwise Comparison of Series • Jones et al. (1986) • Informal examination of paired temperature series • Cassinus and Mestre (2004) • Optimal segmentation of paired difference series • Series causing the change point can be traced more directly

10. Basic Steps • Form combinations of pairwise difference series • Apply undocumented changepoint tests to the difference series • “Unconfound” the identified changepoints • Conflate changepoint dates • Undocumented changepoints attributed to date of metadata event, or • To most common changepoint date • Calculate multiple pairwise estimates of step change amplitude for each target changepoint

11. Step 1: Formation of difference series • All series are paired the with most highly correlated neighboring series • First difference correlation used to minimize impact of step changes and trends on correlation

12. Simulations • Simulated 1000 groups of 21 correlated red noise series (n=1200) • Imposed between 0 and 10 changepoints at random locations and of random magnitude (average = 5)

13. Simulated temperature series with random shifts caused by station moves/site changes (Annual Averages) σ (°C) • Series in red treated as the target in subsequent figures • All shifts are considered to be undocumented • True “climate” trend in all simulated series is zero

14. Target series (lower panel) and differences with neighbors Case 7 unadjusted

15. Step 2: Breakpoint Testing Multiple breaks resolved via a semi-hierarchical splitting algorithm (Hawkins, 1976; Menne and Williams 2005) SNHT (Alexandersson,1986) –- “TPR-0” Change in mean with no trend “TPR-1” (Wang 2003) Change in mean within constant trend “TPR-2” (Lund and Reeves 2002) Change in mean and/or change in trend

16. Step 2: Breakpoint Testing • Use SNHT (TPR-0) with Bayesian Information Criterion (S(q)) verification of changepoint

17. Step 2: Changepoint model identification q = 1 q = 4 q = 4 q = 2 q = 4 q = 3 q = 5 q = 4

18. Why worry about local trends? • Determine impact of land use changes (e.g., urbanization) • Trend changes get confounded with step changes (especially at annual resolutions)

20. Step 3: Attribute Cause of Shifts • Date by date find station whose target-neighbor difference series has failed Ho the most • Subtract one from tally of total number of shifts on corresponding date from each neighbor-target difference series • Iterate for all dates and difference series

23. Step 4: Conflation of Changepoint Dates

24. Step 4: Conflation of Changepoint Dates • Estimate magnitude of changepoint • Assign cluster of changepoint dates within “uncertainty window” • to a single event in the target station’s history or, • to most common changepoint date if undocumented

25. |010|011|012|013|014|015|016|017|018|019|01a|01b|01c|01d|01e|01f|01g|01h|01i|01j|01k| 1940 6 474 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 4|---|---|---|---| 1940 7 475 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---|---| 1940 8 476 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 3|---|---|---|---| 1940 9 477 |---| 3|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1940 11 479 |---|---|---|---|---| 6|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 1 481 |---| 4|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 2 482 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---| 1941 3 483 |---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 4 484 |---|---|---| 2|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 6 486 |---|---| 3| 3|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 11 491 |---|---|---|---|---|---|---|---| 4|---|---|---|---|---|---|---|---|---|---|---|---| 1941 12 492 |---|---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---| 1942 2 494 |---|---|---|---|---|---|---|---| 3|---|---|---|---|---|---|---|---|---|---|---|---| 1942 4 496 |---|---| 4|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1942 7 499 |---|---|---|---|---|---|---|---|---|---|---|---| 5|---|---|---|---|---|---|---|---| 1943 11 515 |---|---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---| 1944 2 518 |---|---|---|---|---|---|---|---|---|---|---|---| 5|---|---|---|---|---|---|---|---| 1944 3 519 |---|---|---|---|---|---| 2|---|---|---|---|---| 3|---|---|---|---|---|---|---|---| 1944 4 520 |---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---| 1944 12 528 |---|---|---|---|---|---| 4| 5|---|---|---|---|---|---|---|---|---|---|---|---|---| 1945 1 529 |---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---| 1945 4 532 |---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1945 11 539 |---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---| 1946 2 542 |---|---|---|---|---|---|---|---|---|---|---|---|---| 3|---|---|---|---|---|---|---| 1946 8 548 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---| 1948 7 571 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---| 1948 8 572 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---| 1949 6 582 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 9|---|---|---|---|

26. |010|011|012|013|014|015|016|017|018|019|01a|01b|01c|01d|01e|01f|01g|01h|01i|01j|01k| 1940 6 474 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 9|---|---|---|---| 1940 11 479 |---|---|---|---|---| 6|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 1 481 |---| 7|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 2 482 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---| 1941 4 484 |---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 6 486 |---|---| 3| 7|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1941 11 491 |---|---|---|---|---|---|---|---| 9|---|---|---|---|---|---|---|---|---|---|---|---| 1942 4 496 |---|---| 4|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1942 7 499 |---|---|---|---|---|---|---|---|---|---|---|---| 5|---|---|---|---|---|---|---|---| 1943 11 515 |---|---|---|---|---|---|---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---| 1944 2 518 |---|---|---|---|---|---|---|---|---|---|---|---| 10|---|---|---|---|---|---|---|---| 1944 12 528 |---|---|---|---|---|---| 7| 8|---|---|---|---|---|---|---|---|---|---|---|---|---| 1945 4 532 |---|---| 2|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 1946 2 542 |---|---|---|---|---|---|---|---|---|---|---|---|---| 5|---|---|---|---|---|---|---| 1946 8 548 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 2|---|---|---| 1948 7 571 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 4|---|---|---| 1949 6 582 |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| 9|---|---|---|---|

27. Step 5: Estimation of Step Change • Use remaining metadata • Step-change magnitude calculated according to model appropriate for each target-neighbor changepoint or as a simple difference in means • Median of step estimates is used as adjustment; significance evaluated by estimating the 5th (median > 0) or 95th (median < 0) of pairwise estimates.

28. Simulated temperature series with random shifts caused by station moves/site changes (Annual Averages) σ (°C) • Series in red treated as the target in subsequent figures • All shifts are considered to be undocumented • True “climate” trend in all simulated series is zero

29. Target series and differences with neighbors before adjustment for undocumented shifts Case 7 unadjusted

30. Target series and differences with neighbors after adjustment for undocumented shifts

31. Simulated temperature series following adjustment by pairwise algorithm (Annual Averages) σ (°C) • Original Target Series in Red • Adjusted Target Series in Green • Adjusted Neighbor Series in Black

32. Diagnostic • For the target example and its nine neighbors, 34 of 43 changepoints were detected and attributed to the correct series. • Of the 9 changepoints not accounted for • 6 are under ±0.3σ • 2 are under ±0.5 σ • 1 was equal to 0.696σ (but was preceded by an unidentified shift of -0.451 10 months earlier)

33. Simulations • Simulated 1000 groups of 21 correlated red noise series (n=1200) • “Monthly Case 1”: Imposed between 0 and 10 changepoints at random locations and of random magnitude (average = 5) • “Monthly Case 2”: As in case 1, except with random unrepresentative (“local”) trends (from 0.001σ/month up to about 0.18σ/month)

34. Algorithm Results for “Step Change Only” Case

35. Algorithm Results for “Steps and Local Trends” Case

41. Impact of Adjustments on Trends U.S. annual and seasonal temperature trends (°C dec-1) 1895 to 2006

50. How to conceive of the difference series?