General Register Office for Scotland 2006 Census Test – Evaluation Methodology

1. General Register Office for Scotland2006 Census Test – Evaluation Methodology Paul Fensom, GROS Bernard Baffour, UoS Royal Statistical Society 29 April 2008

2. Presentation Outline • Brief overview of the Scottish Census Test • Preliminary Results and Critique • Further analysis • Concluding remarks

3. Objectives • Trial different enumeration methods on areas known to specifically contain ‘special’ population groups • Complete census of households in selected areas • Two Design Variables: Postal Delivery and Income

4. Block 1 Block 2 Block 3 Postout Delivery Postout Delivery Postout Delivery Income Income Income Income Income Income Postout Delivery Postout Delivery Postout Delivery No Income No Income No Income No Income No Income No Income Glasgow North Glasgow South West Dunbartonshire Block 4 Block 5 Postout Delivery Postout Delivery Income Income Income Income Postout Delivery Postout Delivery No Income No Income No Income No Income Lochaber Breadalbane Block design of the 2006 Census Test Design Variables

5. Initial Results – Response Rates • Postout 46%; Hand Delivery 53% households that had their forms hand delivered by enumerators tended to have a higher response rate • Income 48%; No Income 44% income question did not detrimentally affect household response rate

6. Critique • Lack of inference due to the non-randomness in the selection of areas • too much focus on ‘extreme’ areas • design not equipped to make national conclusions • Introduction of intra-ED correlations • too much similarity in sub-areas • lack of randomisation of households

7. Further Evaluation • Logistic Regression Analysis • ‘Event’ (response) v ‘Non-Event’ (non-response) • Makes use of non-responding households • Scottish Index of Multiple Deprivation • analyse effects of clustering

8. Methodology • The normal logistic procedure assumes data sampled randomly; • But this is not entirely valid for complex sample designs; • Evidence of cluster effects in Scottish Census Test; • Use of PROC SURVEYLOGISTIC.

9. Scottish Index of Multiple Deprivation (SIMD) • There is a link between undercount and deprivation; • Deprivation was used in the Design stage, so it follow that it needed to be included in analysis; • Deprivation is a defining characteristic in logistic modeling of the results.

10. Clustering • The design applied same treatments to neighbouring households. • Dependence due to clustering needs to be accounted for • A ‘cluster’ variable created based on SIMD quintile and treatment effect to group ‘similar’ households.

11. Results • Distribution of households by Deprivation Quintile

12. Results • The odds ratios under both methods are the same BUT different standard errors.

13. Results – ‘best’ selected logistic model

14. Conclusion • Results remain pretty much unchanged: • evidence that the income question did not have a significant effect on the household response. • the enumeration methodology had a significant effect on whether or not a household responded.

15. Remarks • The Test has shown difficult challenges ahead for achieving high response in 2011 Census; • Non-response schemes should be dynamic to take account of these difficulties.