Towards the 2011 UK Census Editing Strategy

1. Towards the 2011 UK Census Editing Strategy Heather Wagstaff and Steven Rogers Methodology Directorate Office for National Statistics, U.K.

2. Overview The presentation is structured as follows: • Overview of edit & imputation in UK Census • CANCEIS: fitness for 2011 UK Census • Development of 2011 Census Edit Strategy • Summary

3. Overview of UK Census Edit & Imputation • EDIS hard coded leading to flexibility problems: • UK countries all had slightly differing requirements; • 1999 Rehearsal Data too late for system testing; • problems during live running: • late changes to question set not tested; • complex filter questions not followed by number of respondents.

4. Statistical Modernisation Programme • Main Focus: to deliver standard statistical infrastructure, methodologies and tools; • Main Aim: to apply recognised standards and practices in highly efficient way. • CANCEIS endorsed as corporate edit and imputation tool: • implement where data are mainly nominal; • implemented on household surveys; and Civil Registration • now endorsed for 2011 Census.

5. CANCEIS • CANadian Census Edit and Imputation System • generalised edit and imputation system • specify edits in decision logic tables • nearest neighbour imputation methodology • computationally efficient • simultaneous imputation of numeric and categorical variables

6. CANCEIS - ensuring fitness for 2011 Census • 2001 UK Census • processed about 27 million forms in relation to circa. 60 million people. • Demonstrate evidence of robustness for 2011 Census: • provide proof of concept; • replicate 2001 Census Editing Strategy; • recover statistical properties of data.

7. CANCEIS - ensuring fitness for 2011 Census • Stage 1: Provide proof of concept • purpose: to access whether CANCEIS will produce complete and consistent census data • UK Census data processed by Administration Areas • convert edit rules to DLT’s • apply CANCEIS to census data • replicate in SAS to QA edit process • outcome: CANCEIS produced complete and consistent dataset in under 2 hours.

8. CANCEIS - ensuring fitness for 2011 Census • Stage 2: Replicate 2001 Census Editing Strategy • purpose: to assess range of functionality

9. CANCEIS - ensuring fitness for 2011 Census • Stage 3: Recover statistical properties of the data • purpose: to assess whether CANCEIS contained an imputation process of acceptable quality; • micro-simulation environment; • 170K households and 400K individuals; • stochastic process - apply CANCEIS in multiple runs; • measure distributional and predictive accuracy.

10. CANCEIS - ensuring fitness for 2011 Census • Step 3: Recover statistical properties of the data

11. Towards the 2011 Census Edit Strategy • Develop in two parts integral to Census Quality Strategy: • Part 1: Specification of comprehensive and cohesive edit rules • Part 2: Research imputation methodology including • partitioning person variables • large household sizes • Communal Establishments (collectives) • differing area types

12. Towards the 2011 Census Edit Strategy • Example of importance of single cohesive set of edit rules: • dependence of filter rules on 100% accuracy of date of birth at data capture.

13. Towards the 2011 Census Edit Strategy • Example of methodology: partitioning person variables: • 2001 Census question set partitioned in 6 main topics • Labour Market contained 4 subsets • 2011 question set currently unendorsed

14. Concluding Remarks • Benefits of applying a generalised system (CANCEIS) in the 2011 UK Census include: • significant cost savings and efficiency gains (inc. time to process large datasets); • flexibility and transparency; • allow time and resource to address difficult methodological issues.