Validation in statistics canada health surveys
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

Validation in Statistics Canada Health Surveys PowerPoint PPT Presentation


  • 124 Views
  • Uploaded on
  • Presentation posted in: General

Validation in Statistics Canada Health Surveys. Presentation to RRFSS Workshop June 20, 2007 Vincent Dale. Outline. Statistics Canada quality assurance framework Ensuring data accuracy Past validation projects Future projects Future directions. Quality Assurance Framework.

Download Presentation

Validation in Statistics Canada Health Surveys

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Validation in statistics canada health surveys

Validation in Statistics Canada Health Surveys

Presentation to RRFSS Workshop

June 20, 2007

Vincent Dale


Outline

Outline

  • Statistics Canada quality assurance framework

  • Ensuring data accuracy

  • Past validation projects

  • Future projects

  • Future directions


Quality assurance framework

Quality Assurance Framework


Statistics canada quality assurance framework

Statistics CanadaQuality Assurance Framework

  • Trade-offs between aspects of quality

  • These are actively managed through a variety of processes, including:

    • User and stakeholder feedback mechanisms

    • Program review

    • Data analysis and dissemination

    • Standards and documentation (concepts, variables, classifications)


Ensuring data accuracy

Ensuring Data Accuracy

  • Questionnaire development

    • Wherever possible, validated questionnaire modules are used

      • Sometimes modified for use in population-based survey

      • Sometimes not as valid as advertised

  • Questionnaire testing

    • STC policy requires testing of all new questionnaires

      • Cognitive interviews and focus groups

  • Coherence versus accuracy

    • Sometimes better to keep measure stable even if imperfect


Ensuring data accuracy1

Ensuring Data Accuracy

  • Sampling error

    • error attributed to studying a fraction of a population rather than carrying out a census

  • Non-sampling error

    • coverage errors

    • response errors

    • non-response errors

    • processing errors

    • estimation errors

    • analysis errors


Ensuring data accuracy2

Ensuring Data Accuracy

  • Explosion of health survey data

    • More data, more often for smaller levels of geography

  • Increasing attention paid to validity

    • Health measures

    • Administrative data

    • Complimentary surveys


What is validity

What is validity?

  • Face validity

  • Internal validity

    • construct validity

  • External validity

    • Criterion

    • Sensitivity, specificity, predictive value


Past cchs validation projects

Past CCHS Validation Projects


Health care utilisation

Health Care Utilisation

  • Data linkage of CCHS responses with BC administrative health records

  • Supplemented with analysis of:

    • Respondent interpretation and formulation of responses

    • Interviewer behaviour and training

    • Patterns in response changes, edits and timing of response entry


Contacts with health professionals

Contacts with Health Professionals

  • Results of linkage:

    • Compared to provincial health records:

      • Most CCHS respondents (58%) reported fewer primary care physician contacts

      • On average, CCHS respondents reported 1.7 fewer primary care physician contacts

      • Older CCHS respondents and respondents with better self-perceived health tended to report fewer contacts

      • Younger respondents and respondents with poorer self-perceived health tended to report more contacts


Contacts with health professionals1

Contacts with Health Professionals

  • Recommendations from study:

    • Revise wording of specific questions to minimize misinterpretation

    • Facilitate consistent interviewer probing techniques

    • Improved edits and CAPI/CATI application navigation for interviewers to facilitate changes to previously-answered questions


Evaluation of coverage of linked cchs and hospital inpatient records

Evaluation of coverage of linked CCHS and hospital inpatient records

  • Probabilistic linkage used to identify CCHS 1.1 respondents (excluding Québec) hospitalized over a 14-month period

    • Health person-oriented information database (HPOI) is a virtual census of hospital admissions and used as the standard

  • Survey weights applied to the 8230 CCHS records which were found in the HPOI database


Evaluation of coverage of linked cchs and hospital inpatient records1

Evaluation of coverage of linked CCHS and hospital inpatient records


Evaluation of coverage of linked cchs and hospital inpatient records2

Evaluation of coverage of linked CCHS and hospital inpatient records

  • Under-reporting rates similar between women and men

    • Lower among Manitoba residents (69.2%)

    • Higher among individuals aged 12-74 (86.1%) than those aged 75+ (70.3%)

  • Under-reporting is an essential prerequisite to further analyses based on the CCHS – HPOI linked data

    • Use of the linked file could lead to bias due depending on province/territory of residence and age


Cchs measured height weight

CCHS Measured Height & Weight

  • In 2005, height / weight were measured for a sub sample of CCHS Cycle 3.1 participants (n=4567)

    • Weight: mean difference between measured and self-reported weight of 2.1 kg (2.5 kg for women)

    • Height: mean difference between measured and self-reported height of -0.7 cm (-1.0 cm for men)

    • BMI: mean difference between measured and self-reported BMI was 1.1


Cchs measured height weight1

CCHS Measured Height & Weight


Cchs mode effect study

CCHS Mode Effect Study

  • Potential differences associated with two methods of collection used in CCHS

    • CAPI: computer assisted personal interview

    • CATI: computer assisted telephone interview

  • Used a split-panel design with a unique sample frame

    • secondary sampling units randomly assigned to CAPI or CATI.

    • Fully integrated as part of CCHS cycle 2.1

    • 11 sites selected to provide a good representation of each region in Canada


Cchs mode effect study1

CCHS Mode Effect Study

  • Important differences observed for obesity rates

    • CAPI = 17.9%; CATI = 13.2%

  • Physical activity index – inactive persons

    • CAPI = 42.3%; CATI = 34.4%

  • Statistically significant differences for contact with medical doctors and unmet health care needs

  • No significant differences observed in the vast majority of health indicators


Cchs mode effect study2

CCHS Mode Effect Study

  • Overall results show that cycles 1.1 and 2.1 are largely comparable despite an increase in CATI collection for Cycle 2.1 (2003)

  • Results led to a decision to measure exact height and weight for a sub-sample of respondents in cycle 3.1 (2005)

  • Led to improved standardization of interviewer procedures across the two collection modes


Future validation projects

Future Validation Projects


Scale reliability factor analysis

Scale Reliability - Factor Analysis

  • Construct validity / scale reliability:

    • Cronbach’s Alpha calculated for scales used in CCHS questionnaire

    • Results could be published in user guide

      • What are standards?

      • Some researchers feel that scores should be above 0.8


Cchs depression module

CCHS Depression Module

  • Currently, CIDI Short form for Major Depression (CIDI-SF) is used in CCHS

    • Also used in NPHS and several regional and provincial surveys

  • Some problems with its use in CCHS

    • Has not been validated against International Classification of Disease (ICD)

    • Evaluates 12-month prevalence, not necessarily current treatment need

    • Does not evaluate some items related to clinical significance

  • Patient Health Questionnaire (PHQ) identified as potential CIDI-SF replacement


Cchs depression module1

CCHS Depression Module

  • Primary goals of potential validation study:

    • Determine the validity of the CIDI-SF and PHQ in relation to a gold standard diagnostic interview (SCAN – Schedules for Clinical Assessment in Neuropsychiatry)

    • Identify optimal scoring procedures for the PHQ in Canadian population-based studies


Cchs depression module2

CCHS Depression Module

  • Samples of n=200 subjects to be drawn in two sites (English and French)

    • Supplemented with n=100 subjects selected from psychiatric outpatient settings to increase the number of positive cases of major depression

  • Each participant to be administered:

    • 1) Standard demographic module

    • 2) PHQ-9

    • 3) Module to distinguish between clinical depression and bereavement

    • 4) SIDI-SF

    • 5) Set of modules to assess consequences of construct in terms of quality of life


Cchs depression module3

CCHS Depression Module

  • Sensitivity and specificity of the CIDI-SF and PHQ to be measured using the SCAN as a gold standard

  • Ordinal CIDI-SF ratings to be correlated with PHQ ordinal ratings using Spearman correlation coefficient

  • Test of construct validity of PHQ to be performed using exploratory factor analysis

  • Internal consistency of scales and subscales to be assessed using Cronbach’s alpha

  • Test-retest reliability of PHQ and CIDI-SF and inter-rater reliability of the SCAN will be evaluated for 50 respondents


Cchs depression module4

CCHS Depression Module

  • The estimated cost for the project exceeded $200,000

  • Due to our inability to secure external funding and the lack of available budget and personnel internally, there are no concrete plans to proceed with study


Directions forward

Directions Forward

  • Focus on accuracy, interpretability and coherence

  • Trade-offs between aspects of data quality

    • Improved timeliness, accessibility and relevance

  • How good is “good enough”?

  • Partnerships

    • Are there areas where CCHS, RRFSS and others can collaborate ?


  • Contact information

    Contact Information

    Vincent Dale

    Survey Manager, Canadian Community Health Survey

    613-951-4265

    Sylvain Tremblay

    Content Manager, Canadian Community Health Survey

    613-951-2528


  • Login