Validation in statistics canada health surveys
1 / 29

Validation in Statistics Canada Health Surveys - PowerPoint PPT Presentation

  • Uploaded on

Validation in Statistics Canada Health Surveys. Presentation to RRFSS Workshop June 20, 2007 Vincent Dale. Outline. Statistics Canada quality assurance framework Ensuring data accuracy Past validation projects Future projects Future directions. Quality Assurance Framework.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Validation in Statistics Canada Health Surveys' - alaula

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Validation in statistics canada health surveys

Validation in Statistics Canada Health Surveys

Presentation to RRFSS Workshop

June 20, 2007

Vincent Dale


  • Statistics Canada quality assurance framework

  • Ensuring data accuracy

  • Past validation projects

  • Future projects

  • Future directions

Statistics canada quality assurance framework
Statistics CanadaQuality Assurance Framework

  • Trade-offs between aspects of quality

  • These are actively managed through a variety of processes, including:

    • User and stakeholder feedback mechanisms

    • Program review

    • Data analysis and dissemination

    • Standards and documentation (concepts, variables, classifications)

Ensuring data accuracy
Ensuring Data Accuracy

  • Questionnaire development

    • Wherever possible, validated questionnaire modules are used

      • Sometimes modified for use in population-based survey

      • Sometimes not as valid as advertised

  • Questionnaire testing

    • STC policy requires testing of all new questionnaires

      • Cognitive interviews and focus groups

  • Coherence versus accuracy

    • Sometimes better to keep measure stable even if imperfect

Ensuring data accuracy1
Ensuring Data Accuracy

  • Sampling error

    • error attributed to studying a fraction of a population rather than carrying out a census

  • Non-sampling error

    • coverage errors

    • response errors

    • non-response errors

    • processing errors

    • estimation errors

    • analysis errors

Ensuring data accuracy2
Ensuring Data Accuracy

  • Explosion of health survey data

    • More data, more often for smaller levels of geography

  • Increasing attention paid to validity

    • Health measures

    • Administrative data

    • Complimentary surveys

What is validity
What is validity?

  • Face validity

  • Internal validity

    • construct validity

  • External validity

    • Criterion

    • Sensitivity, specificity, predictive value

Health care utilisation
Health Care Utilisation

  • Data linkage of CCHS responses with BC administrative health records

  • Supplemented with analysis of:

    • Respondent interpretation and formulation of responses

    • Interviewer behaviour and training

    • Patterns in response changes, edits and timing of response entry

Contacts with health professionals
Contacts with Health Professionals

  • Results of linkage:

    • Compared to provincial health records:

      • Most CCHS respondents (58%) reported fewer primary care physician contacts

      • On average, CCHS respondents reported 1.7 fewer primary care physician contacts

      • Older CCHS respondents and respondents with better self-perceived health tended to report fewer contacts

      • Younger respondents and respondents with poorer self-perceived health tended to report more contacts

Contacts with health professionals1
Contacts with Health Professionals

  • Recommendations from study:

    • Revise wording of specific questions to minimize misinterpretation

    • Facilitate consistent interviewer probing techniques

    • Improved edits and CAPI/CATI application navigation for interviewers to facilitate changes to previously-answered questions

Evaluation of coverage of linked cchs and hospital inpatient records
Evaluation of coverage of linked CCHS and hospital inpatient records

  • Probabilistic linkage used to identify CCHS 1.1 respondents (excluding Québec) hospitalized over a 14-month period

    • Health person-oriented information database (HPOI) is a virtual census of hospital admissions and used as the standard

  • Survey weights applied to the 8230 CCHS records which were found in the HPOI database

Evaluation of coverage of linked cchs and hospital inpatient records2
Evaluation of coverage of linked CCHS and hospital inpatient records

  • Under-reporting rates similar between women and men

    • Lower among Manitoba residents (69.2%)

    • Higher among individuals aged 12-74 (86.1%) than those aged 75+ (70.3%)

  • Under-reporting is an essential prerequisite to further analyses based on the CCHS – HPOI linked data

    • Use of the linked file could lead to bias due depending on province/territory of residence and age

Cchs measured height weight
CCHS Measured Height & Weight records

  • In 2005, height / weight were measured for a sub sample of CCHS Cycle 3.1 participants (n=4567)

    • Weight: mean difference between measured and self-reported weight of 2.1 kg (2.5 kg for women)

    • Height: mean difference between measured and self-reported height of -0.7 cm (-1.0 cm for men)

    • BMI: mean difference between measured and self-reported BMI was 1.1

Cchs mode effect study
CCHS Mode Effect Study records

  • Potential differences associated with two methods of collection used in CCHS

    • CAPI: computer assisted personal interview

    • CATI: computer assisted telephone interview

  • Used a split-panel design with a unique sample frame

    • secondary sampling units randomly assigned to CAPI or CATI.

    • Fully integrated as part of CCHS cycle 2.1

    • 11 sites selected to provide a good representation of each region in Canada

Cchs mode effect study1
CCHS Mode Effect Study records

  • Important differences observed for obesity rates

    • CAPI = 17.9%; CATI = 13.2%

  • Physical activity index – inactive persons

    • CAPI = 42.3%; CATI = 34.4%

  • Statistically significant differences for contact with medical doctors and unmet health care needs

  • No significant differences observed in the vast majority of health indicators

Cchs mode effect study2
CCHS Mode Effect Study records

  • Overall results show that cycles 1.1 and 2.1 are largely comparable despite an increase in CATI collection for Cycle 2.1 (2003)

  • Results led to a decision to measure exact height and weight for a sub-sample of respondents in cycle 3.1 (2005)

  • Led to improved standardization of interviewer procedures across the two collection modes

Scale reliability factor analysis
Scale Reliability - Factor Analysis records

  • Construct validity / scale reliability:

    • Cronbach’s Alpha calculated for scales used in CCHS questionnaire

    • Results could be published in user guide

      • What are standards?

      • Some researchers feel that scores should be above 0.8

Cchs depression module
CCHS Depression Module records

  • Currently, CIDI Short form for Major Depression (CIDI-SF) is used in CCHS

    • Also used in NPHS and several regional and provincial surveys

  • Some problems with its use in CCHS

    • Has not been validated against International Classification of Disease (ICD)

    • Evaluates 12-month prevalence, not necessarily current treatment need

    • Does not evaluate some items related to clinical significance

  • Patient Health Questionnaire (PHQ) identified as potential CIDI-SF replacement

Cchs depression module1
CCHS Depression Module records

  • Primary goals of potential validation study:

    • Determine the validity of the CIDI-SF and PHQ in relation to a gold standard diagnostic interview (SCAN – Schedules for Clinical Assessment in Neuropsychiatry)

    • Identify optimal scoring procedures for the PHQ in Canadian population-based studies

Cchs depression module2
CCHS Depression Module records

  • Samples of n=200 subjects to be drawn in two sites (English and French)

    • Supplemented with n=100 subjects selected from psychiatric outpatient settings to increase the number of positive cases of major depression

  • Each participant to be administered:

    • 1) Standard demographic module

    • 2) PHQ-9

    • 3) Module to distinguish between clinical depression and bereavement

    • 4) SIDI-SF

    • 5) Set of modules to assess consequences of construct in terms of quality of life

Cchs depression module3
CCHS Depression Module records

  • Sensitivity and specificity of the CIDI-SF and PHQ to be measured using the SCAN as a gold standard

  • Ordinal CIDI-SF ratings to be correlated with PHQ ordinal ratings using Spearman correlation coefficient

  • Test of construct validity of PHQ to be performed using exploratory factor analysis

  • Internal consistency of scales and subscales to be assessed using Cronbach’s alpha

  • Test-retest reliability of PHQ and CIDI-SF and inter-rater reliability of the SCAN will be evaluated for 50 respondents

Cchs depression module4
CCHS Depression Module records

  • The estimated cost for the project exceeded $200,000

  • Due to our inability to secure external funding and the lack of available budget and personnel internally, there are no concrete plans to proceed with study

Directions forward
Directions Forward records

  • Focus on accuracy, interpretability and coherence

  • Trade-offs between aspects of data quality

    • Improved timeliness, accessibility and relevance

  • How good is “good enough”?

  • Partnerships

    • Are there areas where CCHS, RRFSS and others can collaborate ?

  • Contact information
    Contact Information records

    Vincent Dale

    Survey Manager, Canadian Community Health Survey


    Sylvain Tremblay

    Content Manager, Canadian Community Health Survey