ESPON 2013 Programme Workshop Managing Time Series and Estimating Missing Values 6 May 2010

1. Outlier Detection and the Estimation of Missing ValuesMartin Charlton and Paul HarrisNational Centre for GeocomputationNational University of Ireland MaynoothMaynooth, Co Kildare, IRELAND ESPON 2013 Programme Workshop Managing Time Series and Estimating Missing Values 6 May 2010 Luxembourg

2. Outline • Time Series • ESPON DB data issues • Detecting exceptional values • Estimation of missing values • Case study

3. 1: Time Series

4. What is a time series? • A variable which is measured sequentially in time at fixed sampling intervals is known as a time series • The behaviour of such series can be modelled • The main features of time series are trend and (sometimes) seasonal variation • Observations which are close together in time tend to be correlated

5. Air Passengers 1949-1960 A time plot of the number of air passengers per month between January 1949 and December 1960 in the USA reveals a rising trend There is also a seasonal pattern of travel within each year. More people travel in the summer than the winter.

6. Aggregating the series annually reveals the rising trend, and the boxplot shows that more people travel in the summer months.

7. Forecasting: 1 There are many modelling and forecasting techniques. Here we use the Holt Winters procedure to model the series behaviour… The fit is quite promising

8. Forecasting: 2 And if the growth of the US air traffic during the first 4 years of the 1960s follows the pattern of the previous 12… the forecast is for some 800 million passengers by 1965

9. Models • There are a wide variety of different models, including • Basic stochastic models (like Holt Winters) • Stationary models (AR, MA, ARMA) • Non-stationary models (ARIMA, ARCH) • Spectral analysis (based on the Fourier transform) • Multivariate models (two or more series are involved)

10. 2: ESPON DB Data Issues

11. We might normalise these by the population to reach a comparable ‘per capita’ figure Some typical data… household income The NUTS2 regions in Austria are the Länder – here we have short time series concerning disposable income of private households from 1995 to 2007. Each series has only 13 elements

12. Short series… • We should be aware that there is an interaction between the amount of data available and what can be done with it • Paas, Kusk, Schlitte and Võrk’s 2007 analysis of income convergence in selected countries of the EU using NUTS3 data had this to say:

13. George Box, 1976, Science and Statistics • Models include not just the analytical tools that others might use, but those which we use to examine the data for outliers and estimating values • ‘Wrong’ for Box includes models that fail to encapsulate the process under investigation

14. ESPON Tigers • Long time series tend to be for large areal units, such as countries, or major administrative regions – the MAUP may well also be a tiger • Smaller regions… • shorter series • incomplete series • a long time period between elements (decennial censuses) in the case of very small units

15. 3: Detecting Exceptional Values

16. Exceptional values • Two types: • Logical errors (e.g. negative unemployment rate) • Statistical outlier (e.g. unusually high unemployment rate) • Identification methods • Logical errors: mechanical (& statistical) techniques • Statistical outliers: statistical techniques

17. Types of outliers

18. Our approach • There is no single ‘best’ detection technique, so… • Apply a selection of outlier detection methods, which are simple and robust • Flag an observation if it is a likely outlier according to each technique • Build up the weight of evidence for the likelihood of an value being statistically exceptional • Suggest what type of outlier it is likely to be • aspatial, spatial, temporal, relationship, a mixture • Consult an expert of the data to decide on the appropriate cause of action

19. Issues • Temporal outliers • The time series are often too short to apply a ‘standard’ technique reliably • So... Parallel time series are treated as additional variables (there will be a high positive correlation between series from different years) • Then... Apply an aspatial/spatial/relationship detection technique • That is... We add the spatial component which is then treated either implicitly or explicitly • Modifiable Areal Unit Problem MAUP • Identify exceptional values at the finest spatial resolution

20. Weight of evidence • If we apply a range of techniques, then we can build up the weight of evidence for the likelihood of an observation being exceptional • Observations which are exceptional on most or all of the tests are those which we would select for further investigation • Here’s an example showing three observations…

22. 4: Estimating Missing Data

23. Data estimation techniques • There is an enormous range of possibilities • Choice depends on • Data type, size, dimensionality, and properties • Objective – prediction or prediction uncertainty accuracy • Model complexity • We can estimate missing values using... • Averaging • Regression (with or without autocorrelation, global and local) • Inverse distance weighting • Regression Kriging • Co-Kriging • Bayesian Markov Chain Monte Carlo methods

24. 5: Case studyIdentifying NUTS regions with exceptional time-series values

25. Unemployment at NUTS 23 2000-2007 • A dataset for NUTS23 regions was obtained from UMS-RIATE • For each year there are counts of • Economically active population • Unemployed, economically active population • Shapefile created from NUTS2/NUTS3 shapefiles in Mapkit • Analysis undertaken in R

26. Eight ‘unemployment rate’ variables for 2000 to 2007 Rate = [Unemployed/Economically active] 790 x 8 observations at NUTS 2/3 level Some island data removed

27. Data post-processing • Logical input errors • Original data checked • There appear to be none, appear to be a few exceptional values • Assessing outlier detection methods • 320 values randomly picked (~5% of the data) • These are in 271 regions • Values doubled and then randomly redistributed among the 320 positions in the data • These observations are assumed to be outlying in some way (but we cannot guarantee this)

28. Effect ofoutliers? Merely looking at some maps doesn’t help in easily identifying the regions with exceptional values

29. Interseries correlations Those plots about the main diagonal are highly correlated. The effect of the randomly introduced values is clearer on the more distant plots (these are also ‘distant’ in time)

30. Detection Techniques for comparison • Simple time-series approach (TS) – outlined in FIR: we have used a simplified version • Principal Components Analysis (PCA) • GWPrincipal Components Analysis (GWPCA) • The PCA based methods allow us to consider more than simply pairs of time series simultaneously

31. We’ll compare the various methods

32. Time Series method (TS) • For each of the 790 regions, index TS is calculated at each of 8 time observations (using the 8-observation data set): • TS = [observation – mean]2/[variance] • Assuming Gaussian errors, a time observation is taken as outlying if TS > 3.84 (95% level) • In this study, we simply find outliers according to boxplot statistics • An indicator variable is then set at any region for which at least one time observation is outlying

33. Principal Components Analysis (PCA) • Principal Components Analysis is a technique which transforms m correlated variables into m new variables which are have a correlation of zero • All of the variance in the original m variables is retained during the transformation • Values of the new variables are known as scores – we can use these for identifying exceptional values

34. Geographically Weighted PCA • PCA is a global transformation but it ignores the spatial arrangement of the NUTS regions • With GWPCA we obtain local transformations by applying geographical weighting – this gives us a set of components for each NUTS region • We can use the scores from these local transformations to identify exceptional values

35. PCA for the unemployment series The series are highly correlated, so the first component accounts for the majority of the variance

36. Using PCA and GWPCA • Examine the residual component data (those with small variances) • Use boxplot statistics to define outlying values • In this case, a significant result indicates one or more outlying time observations in a NUTS region • GWPCA will also indicate a spatial ‘outlyingness’ in the data

37. The various techniques are compared on the next slides

38. (a) TS method compared with PCA The TS method appears to be less discriminating than the global PCA method

39. (b) TS compared with GWPCA The GWPCA method would appear to be very discriminating in identifying potentially exceptional regions

40. (c) PCA compared with GWPCA The global PCA is slightly less discriminating than the GW PCA

41. Results for the 271 randomised sites • Sites not identified as outlying – 21.4% • Outlying by at least one method – 78.6% • Outlying by one method only – 55.3% • Outlying by two methods – 18.8% • Outlying by all three methods – 4.8%

42. Identification by method: • TS (75.6%) • PCA (22.5%) • GWPCA (8.8%) • False positives at 519 un-affected sites: • TS (29.5%) • PCA (2.3%) • GWPCA (1.3%) • These results endorse the “weight of evidence” approach to the identification of exceptional values…

43. Acknowledgements • We are disappointed that Eyjafjallajökull decided to send some ash to Ireland • We are deeply grateful to Claude for presenting this work – some of it is not easy • We also acknowledge statistical advice from Professor Chris Brunsdon, Professor of Geographic Information at the University of Leicester

44. Thank You!