Standard measures and variables

1. Standard measures and variables Paul Lambert, University of Stirling Presentation to the Scottish Civil Society Data Partnership Project (S-CSDP), Webinar 3 on ‘Dealing with data: Using standard measures and variables and linking together datasets’ www.thinkdata.org.uk, 10 Mar 2016

2. The importance of standard measures and variables “No man is an island” (Donne, 1624) In social research, we can draw upon a vast array of previous operationalisations of measures, and take advice from influential organisations/researchers on the optimal ways of constructing measures (i.e. ‘standards’) • Scientific importance: Maximise replicability, consistency, reliability, validity and prospective impact of research results • Pragmatic importance: Save time and energy by drawing on existing resources about important measures • see ‘CSDP workshop 1’ for notes on measures about socio-economic circumstances; voluntarism; protected characteristics S-CSDP, 10 Mar 2016

3. (a) Data on standards: UK NSI’s • ONS guidance: https://www.ons.gov.uk/methodology/programmesandservices/harmonisationprogramme • Scottish government information: http://www.gov.scot/Topics/Statistics/About/Methodology/Classifications • ADLS and P-ADLS: http://www.adls.ac.uk/padls/ From CODE briefing ‘How has ethnic diversity changed in Scotland?’ http://www.ethnicity.ac.uk/medialibrary/briefings/dynamicsofdiversity/code-census-briefing-scotland_v2.pdf S-CSDP, 10 Mar 2016

4. (a) Data on standards: Secondary surveys • ...any measure you’re thinking of will probably have been used in a previous large-scale social survey [e.g. ‘Survey question bank’ at UKDS] • …it’s nearly always better to re-use a measure/format that was piloted and chosen by experts, than invent a new one • …secondary surveys also exhibit good practice in documentation and metadata provision e.g. BHPS documentation online, at https://www.iser.essex.ac.uk/bhps/documentation S-CSDP, 10 Mar 2016

5. (a) Data on standards: Cross-national standards • Academic literature with recommended approaches • NSI’s and cross-national agencies with recommendations for comparisons, e.g. https://data.oecd.org/ Hoffmeyer-Zlotnik, J. H. P., & Warner, U. (2014). Harmonising Demographic and Socio-Economic Variables for Cross-National Comparative Survey Research. Berlin: Springer. S-CSDP, 10 Mar 2016

6. (a) Data on standards: Academic research advice on standards • Methodologists tend to argue: • (e.g. Bulmer 2010; Dale 2006) • Use an existing standard unless you have a compelling reason not to • Use sensitivity analysis to operationalise, compare and document a few plausible measures • Provide crystal clear information on the standards used • …but many academics ‘do their own thing’ in research and disregard standards… S-CSDP, 10 Mar 2016

7. (b) Using standards: ‘Data management’ issues Operationalising many alternative socio-economic measures makes sense – but linking the data and metadata is not easy! Ideally, the construction of standard measures should be… • Clearly documented (e.g. with command ‘syntax’) • Consistent with published recommendations • Linked to published metadata (e.g. using an ‘index file’ or ‘translation matrix’) • In some situations, a standard measure isn’t plausible, but may be adapted (& should be documented) (e.g. due to sparse representations in key categories) S-CSDP, 10 Mar 2016

8. (b) Using standards: Data analytical issues • ‘Equivalence’ considerations • ‘Measurement equivalence’ = trust the measure intrinsically • ‘Meaning’ or ‘functional’ equivalence = relative meaning, within the national/temporal/sample context (e.g. use ‘arithmetic standardisation’) • Contextual considerations • Are there other important correlated factors? • Interaction terms? Example: Highest educational qualification is a particularly difficult concept to analyse appropriately because of its strong relation to birth cohort and gender S-CSDP, 10 Mar 2016

9. Summary: Attention to standards is worthwhile… • Time-saving, improve quality, lessen risks of errors • ..all for the cost of a small amount of work in checking and trying to use relevant recommendations References cited • Bulmer, M., Gibbs, J., & Hyman, L. (Eds.). (2010). Social Measurement through Social Surveys: An Applied Approach. Aldershot: Ashgate. • Dale, A. (2006). Quality Issues with Survey Research. International Journal of Social Research Methodology, 9(2), 143-158. • Hoffmeyer-Zlotnik, J. H. P., & Warner, U. (2014). Harmonising Demographic and Socio-Economic Variables for Cross-National Comparative Survey Research. Berlin: Springer. S-CSDP, 10 Mar 2016