Missing data in randomized control trials l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 82

Missing Data in Randomized Control Trials PowerPoint PPT Presentation


  • 144 Views
  • Uploaded on
  • Presentation posted in: General

Missing Data in Randomized Control Trials. John W. Graham The Prevention Research Center and Department of Biobehavioral Health Penn State University. IES/NCER Summer Research Training Institute, July 2008. [email protected] Sessions in Four Parts. (1) Introduction: Missing Data Theory

Download Presentation

Missing Data in Randomized Control Trials

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Missing data in randomized control trials l.jpg

Missing Data in Randomized Control Trials

John W. Graham

The Prevention Research Center

and

Department of Biobehavioral Health

Penn State University

IES/NCER Summer Research Training Institute, July 2008

[email protected]


Sessions in four parts l.jpg

Sessions in Four Parts

  • (1) Introduction: Missing Data Theory

  • (2) Attrition: Bias and Lost Power

  • (3) A brief analysis demonstration

    • Multiple Imputation with

      • NORM and

      • Proc MI

  • (4) Hands-on Intro to Multiple Imputation


Recent papers l.jpg

Recent Papers

  • Graham, J. W., Cumsille, P. E.,& Elek-Fisk,E. (2003).Methods for handling missing data. In J. A. Schinka & W. F. Velicer (Eds.). Research Methods in Psychology (pp. 87_114). Volume 2 of Handbook of Psychology (I. B. Weiner, Editor-in-Chief). New York: John Wiley & Sons.

  • Graham, J. W., (2009, in press).Missing data analysis: making it work in the real world. Annual Review of Psychology, 60.

  • Collins, L. M., Schafer, J. L.,& Kam, C. M.(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351.

  • Schafer, J. L.,& Graham,J. W.(2002).Missing data: our view of the state of the art. Psychological Methods, 7, 147-177.


Part 1 a brief introduction to analysis with missing data l.jpg

Part 1:A Brief Introduction toAnalysis with Missing Data


Problem with missing data l.jpg

Problem with Missing Data

  • Analysis procedures were designed for complete data. . .


Solution 1 l.jpg

Solution 1

  • Design new model-based procedures

  • Missing Data + Parameter Estimation in One Step

  • Full Information Maximum Likelihood (FIML)SEM and Other Latent Variable Programs(Amos, LISREL, Mplus, Mx, LTA)


Solution 2 l.jpg

Solution 2

  • Data based procedures

    • e.g., Multiple Imputation (MI)

  • Two Steps

    • Step 1: Deal with the missing data

      • (e.g., replace missing values with plausible values

      • Produce a product

    • Step 2: Analyze the product as if there were no missing data


Slide8 l.jpg

FAQ

  • Aren't you somehow helping yourself with imputation?. . .


No missing data imputation l.jpg

NO. Missing data imputation . . .

  • does NOT give you something for nothing

  • DOES let you make use of all data you have

    . . .


Slide10 l.jpg

FAQ

  • Is the imputed value what the person would have given?


No when we impute a value l.jpg

NO. When we impute a value . .

  • We do not impute for the sake of the value itself

  • We impute to preserve important characteristics of the whole data set

    . . .


We want l.jpg

We want . . .

  • unbiased parameter estimation

    • e.g., b-weights

  • Good estimate of variability

    • e.g., standard errors

  • best statistical power


Causes of missingness l.jpg

Causes of Missingness

  • Ignorable

    • MCAR: Missing Completely At Random

    • MAR: Missing At Random

  • Non-Ignorable

    • MNAR: Missing Not At Random


Mcar missing completely at random l.jpg

MCAR(Missing Completely At Random)

  • MCAR 1: Cause of missingness completely random process (like coin flip)

  • MCAR 2:

    • Cause uncorrelated with variables of interest

    • Example: parents move

  • No bias if cause omitted


Mar missing at random l.jpg

MAR (Missing At Random)

  • Missingness may be related to measured variables

  • But no residual relationship with unmeasured variables

    • Example: reading speed

  • No bias if you control for measured variables


Mnar missing not at random l.jpg

MNAR (Missing Not At Random)

  • Even after controlling for measured variables ...

  • Residual relationship with unmeasured variables

  • Example: drug use reason for absence


Mnar causes l.jpg

MNAR Causes

  • The recommended methods assume missingness is MAR

  • But what if the cause of missingness is not MAR?

  • Should these methods be used when MAR assumptions not met?

    . . .


Yes these methods work l.jpg

YES! These Methods Work!

  • Suggested methods work better than “old” methods

  • Multiple causes of missingness

    • Only small part of missingness may be MNAR

  • Suggested methods usually work very well


Methods old vs mar vs mnar l.jpg

Methods:"Old" vs MAR vs MNAR

  • MAR methods (MI and ML)

    • are ALWAYS at least as good as,

    • usually better than "old" methods (e.g., listwise deletion)

  • Methods designed to handle MNAR missingness are NOT always better than MAR methods


Analysis old and new l.jpg

Analysis: Old and New


Old procedures analyze complete cases listwise deletion l.jpg

Old Procedures: Analyze Complete Cases(listwise deletion)

  • may produce bias

  • you always lose some power

    • (because you are throwing away data)

  • reasonable if you lose only 5% of cases

  • often lose substantial power


Analyze complete cases listwise deletion l.jpg

Analyze Complete Cases(listwise deletion)

  • 1 1 1 1

  • 0 1 1 1

  • 1 0 1 1

  • 1 1 0 1

  • 1 1 1 0

  • very common situation

  • only 20% (4 of 20) data points missing

  • but discard 80% of the cases


  • Other old procedures l.jpg

    Other "Old" Procedures

    • Pairwise deletion

      • May be of occasional use for preliminary analyses

    • Mean substitution

      • Never use it

    • Regression-based single imputation

      • generally not recommended ... except ...


    Recommended model based procedures l.jpg

    Recommended Model-Based Procedures

    • Multiple Group SEM (Structural Equation Modeling)

    • LatentTransitionAnalysis (Collins et al.)

      • A latent class procedure


    Recommended model based procedures25 l.jpg

    Recommended Model-Based Procedures

    • Raw Data Maximum Likelihood SEMaka Full Information Maximum Likelihood (FIML)

      • Amos (James Arbuckle)

      • LISREL 8.5+ (Jöreskog & Sörbom)

      • Mplus (Bengt Muthén)

      • Mx (Michael Neale)


    Amos 7 mx mplus lisrel 8 8 l.jpg

    Amos 7, Mx, Mplus, LISREL 8.8

    • Structural Equation Modeling (SEM) Programs

    • In Single Analysis ...

      • Good Estimation

      • Reasonable standard errors

      • Windows Graphical Interface


    Limitation with model based procedures l.jpg

    Limitation with Model-Based Procedures

    • That particular model must be what you want


    Recommended data based procedures l.jpg

    Recommended Data-Based Procedures

    EM Algorithm (ML parameter estimation)

    • Norm-Cat-Mix, EMcov, SAS, SPSS

      Multiple Imputation

    • NORM, Cat, Mix, Pan (Joe Schafer)

    • SAS Proc MI

    • LISREL 8.5+

    • Amos 7


    Em algorithm l.jpg

    EM Algorithm

    • Expectation - Maximization

      Alternate between

      E-step: predict missing data

      M-step: estimate parameters

    • Excellent (ML) parameter estimates

    • But no standard errors

      • must use bootstrap

      • or multiple imputation


    Multiple imputation l.jpg

    Multiple Imputation

    • Problem with Single Imputation:Too Little Variability

      • Because of Error Variance

      • Because covariance matrix is only one estimate


    Too little error variance l.jpg

    Too Little Error Variance

    • Imputed value lies on regression line


    Imputed values on regression line l.jpg

    Imputed Values on Regression Line


    Restore error l.jpg

    Restore Error . . .

    • Add random normal residual


    Regression line only one estimate l.jpg

    Regression Line only One Estimate


    Covariance matrix regression line only one estimate l.jpg

    Covariance Matrix (Regression Line) only One Estimate

    • Obtain multiple plausible estimates of the covariance matrix

    • ideally draw multiple covariance matrices from population

    • Approximate this with

      • Bootstrap

      • Data Augmentation (Norm)

      • MCMC (SAS 8.2, 9)


    Data augmentation l.jpg

    Data Augmentation

    • stochastic version of EM

    • EM

      • E (expectation) step: predict missing data

      • M (maximization) step: estimate parameters

    • Data Augmentation

      • I (imputation) step: simulate missing data

      • P (posterior) step: simulate parameters


    Data augmentation37 l.jpg

    Data Augmentation

    • Parameters from consecutive steps ...

      • too related

      • i.e., not enough variability

    • after 50 or 100 steps of DA ...

      covariance matrices are like random draws from the population


    Multiple imputation allows l.jpg

    Multiple Imputation Allows:

    • Unbiased Estimation

    • Good standard errors

      • provided number of imputations (m) is large enough

      • too few imputations  reduced power with small effect sizes


    Slide39 l.jpg

    ρ

    From Graham, J.W., Olchowski, A.E., & Gilreath, T.D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206-213.


    Part 2 attrition bias and loss of power l.jpg

    Part 2Attrition: Bias and Loss of Power


    Relevant papers l.jpg

    Relevant Papers

    • Graham, J.W., (in press).Missing data analysis: making it work in the real world. Annual Review of Psychology, 60.

    • Collins, L. M., Schafer, J. L.,& Kam, C. M.(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351.

    • Hedeker, D.,& Gibbons,R.D.(1997).Application of random-effects pattern-mixture models for missing data in longitudinal studies, Psychological Methods, 2, 64-78.

    • Graham, J.W.,& Collins, L.M. (2008). Using Modern Missing Data Methods with Auxiliary Variables to Mitigate the Effects of Attrition on Statistical Power. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request)

    • Graham, J.W.,Palen, L.A., et al. (2008). Attrition: MAR & MNAR missingness, and estimation bias. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request)


    What if the cause of missingness is mnar l.jpg

    What if the cause of missingness is MNAR?

    Problems with this statement

    • MAR & MNAR are widely misunderstood concepts

    • I argue that the cause of missingness is never purely MNAR

    • The cause of missingness is virtually never purely MAR either.


    Mar vs mnar l.jpg

    MAR vs MNAR

    • "Pure" MCAR, MAR, MNAR never occur in field research

    • Each requires untenable assumptions

      • e.g., that all possible correlations and partial correlations are r = 0


    Mar vs mnar44 l.jpg

    MAR vs MNAR

    • Better to think of MAR and MNAR asforming a continuum

    • MAR vs MNAR NOT even the dimension of interest


    Mar vs mnar what is the dimension of interest l.jpg

    MAR vs MNAR: What IS the Dimension of Interest?

    • How much estimation bias?

      • when cause of missingness cannot be included in the model


    Bottom line l.jpg

    Bottom Line ...

    • All missing data situations are partly MAR and partly MNAR

    • Sometimes it matters ...

      • bias affects statistical conclusions

    • Often it does not matter

      • bias has tolerably little effect on statistical conclusions

        (Collins, Schafer, & Kam, Psych Methods, 2001)


    Methods old vs mar vs mnar47 l.jpg

    Methods:"Old" vs MAR vs MNAR

    • MAR methods (MI and ML)

      • are ALWAYS at least as good as,

      • usually better than "old" methods (e.g., listwise deletion)

    • Methods designed to handle MNAR missingness are NOT always better than MAR methods


    Yardstick for measuring bias l.jpg

    Yardstick for Measuring Bias

    Standardized Bias =

    (average parameter est) – (population value)

    -------------------------------------------------------- X 100

    Standard Error (SE)

    • |bias| < 40 considered small enough to be tolerable


    A little background for collins schafer kam 2001 csk l.jpg

    A little background for Collins, Schafer, & Kam (2001; CSK)

    • Example model of interest: X  Y

      X = Program (prog vs control)

      Y = Cigarette Smoking

      Z = Cause of missingness: say, Rebelliousness (or smoking itself)

    • Factors to be considered:

      • % Missing (e.g., % attrition)

      • rYZ

      • rZR


    Slide50 l.jpg

    rYZ

    • Correlation between

      • cause of missingness (Z)

        • e.g., rebelliousness (or smoking itself)

      • and the variable of interest (Y)

        • e.g., Cigarette Smoking


    R z r l.jpg

    rZR

    • Correlation between

      • cause of missingness (Z)

        • e.g., rebelliousness (or smoking itself)

      • and missingness on variable of interest

        • e.g., Missingness on the Smoking variable

    • Missingness on Smoking (often designated: R or RY)

      • Dichotomous variable:

        R = 1: Smoking variable not missing

        R = 0: Smoking variable missing


    Csk study design partial l.jpg

    CSK Study Design (partial)

    • Simulations manipulated

      • amount of missingness (25% vs 50%)

      • rZY (r = .40, r = .90)

      • rZR held constant

        • r = .45 with 50% missing

        • (applies to "MNAR-Linear" missingness)


    Csk results partial mnar missingness l.jpg

    CSK Results (partial)(MNAR Missingness)

    • 25% missing, rYZ = .40 ... no problem

    • 25% missing, rYZ = .90 ... no problem

    • 50% missing, rYZ = .40 ... no problem

    • 50% missing, rYZ = .90 ... problem

      * "no problem" = bias does not interfere with inference

    These Results apply to the regression coefficient for X  Y with "MNAR-Linear" missingness (see CSK, 2001, Table 2)


    But even csk results too conservative l.jpg

    But Even CSK ResultsToo Conservative

    • Not considered by CSK: rZR

    • In their simulation rZR = .45

    • Even with 50% missing and rYZ = .90

      • bias can be acceptably small

    • Graham et al. (2008):

      • Bias acceptably small(standardized bias < 40) as long as rZR < .24


    R z r 24 very plausible l.jpg

    rZR < .24 Very Plausible

    (estimated)

    Study rZR

    ______________

    HealthWise(Caldwell, Smith, et al., 2004).106

    AAPT(Hansen & Graham, 1991).093

    Botvin1.044

    Botvin2.078

    Botvin3.104

    • All of these yield standardized bias < 10


    Csk and follow up simulations l.jpg

    CSK and Follow-up Simulations

    • Results very promising

    • Suggest that even MNAR biases are often tolerably small

    • But these simulations still too narrow


    Beginnings of a taxonomy of attrition l.jpg

    Beginnings of a Taxonomy of Attrition

    Causes of Attrition on Y (main DV)

    • Case 1: not Program (P), not Y, not PY interaction

    • Case 2: P only

    • Case 3: Y only . . . (CSK scenario)

    • Case 4: P and Y only


    Beginnings of a taxonomy of attrition58 l.jpg

    Beginnings of a Taxonomy of Attrition

    Causes of Attrition on Y (main DV)

    • Case 5: PY interaction only

    • Case 6: P + PY interaction

    • Case 7: Y + PY interaction

    • Case 8: P, Y, and PY interaction


    Taxonomy of attrition l.jpg

    Taxonomy of Attrition

    • Cases 1-4

      • often little or no problem

    • Cases 5-8

      • Jury still out (more research needed)

      • Very likely not as much of a problem as previously though

      • Use diagnostics to shed light


    Use of missing data diagnostics l.jpg

    Use of Missing Data Diagnostics

    • Diagnostics based on pretest data not much help

      • Hard to predict missing distal outcomes from differences on pretest scores

    • Longitudinal Diagnostics can be much more helpful


    Hedeker gibbons 1997 l.jpg

    Hedeker & Gibbons (1997)

    • Plot main DV over time for four groups:

      • for Program and Control

      • for those with and without last wave of data

    • Much can be learned


    Empirical examples l.jpg

    Empirical Examples

    • Hedeker & Gibbons (1997)

      • Drug treatment of psychiatric patients

    • Hansen & Graham (1991)

      • Adolescent Alcohol Prevention Trial (AAPT)

      • Alcohol, smoking, other drug prevention among normal adolescents (7th – 11th grade)


    Empirical example used by hedeker gibbons 1997 l.jpg

    Empirical Example Used by Hedeker & Gibbons (1997)

    • IV: Drug Treatment vs. Placebo Control

    • DV: Inpatient Multidimensional Psychiatric Scale (IMPS)

      • 1 = normal

      • 2 = borderline mentally ill

      • 3 = mildly ill

      • 4 = moderately ill

      • 5 = markedly ill

      • 6 = severely ill

      • 7 = among the most extremely ill


    From hedeker gibbons 1997 l.jpg

    From Hedeker & Gibbons (1997)

    Placebo Control

    IMPS

    low = better outcomes

    Drug Treatment

    Weeks of Treatment


    Longitudinal diagnostics hedeker gibbons example l.jpg

    Longitudinal DiagnosticsHedeker & Gibbons Example

    • Treatment

      • droppers do BETTER than stayers

    • Control

      • droppers do WORSE than stayers

    • Example of Program X DV interaction

    • But in this case, pattern would lead to suppression bias

      • Not as bad for internal validity in presence of significant program effect


    Aapt hansen graham 1991 l.jpg

    AAPT (Hansen & Graham, 1991)

    • IV: Normative Education Program vs Information Only Control

    • DV: Cigarette Smoking (3-item scale)

      • Measured at one-year intervals

      • 7th grade – 11th grade


    Slide67 l.jpg

    AAPT

    Control

    Program

    Control

    Cigarette Smoking

    (high = more smoking; arbitrary scale)

    Program

    th

    th

    th

    th

    th


    Longitudinal diagnostics aapt example l.jpg

    Longitudinal DiagnosticsAAPT Example

    • Treatment

      • droppers do WORSE than stayers

        • little steeper increase

    • Control

      • droppers do WORSE than stayers

        • little steeper increase

    • Little evidence for Prog X DV interaction

    • Very likely MAR methods allow good conclusions (CSK scenario holds)


    Use of auxiliary variables l.jpg

    Use of Auxiliary Variables

    • Reduces attrition bias

    • Restores some power lost due to attrition


    What is an auxiliary variable l.jpg

    What Is an Auxiliary Variable?

    • A variable correlated with the variables in your model

      • but not part of the model

      • not necessarily related to missingness

      • used to "help" with missing data estimation

    • Best auxiliary variables:

      • same variable as main DV, but measured at waves not used in analysis model


    Model of interest l.jpg

    Model of Interest


    Benefit of auxiliary variables l.jpg

    Benefit of Auxiliary Variables

    • Example from Graham & Collins (2008)

      X Y Z

      1 1 1 500 complete cases

      1 0 1500 cases missing Y

    • X, Y variables in the model (Y sometimes missing)

    • Z is auxiliary variable


    Benefit of auxiliary variables73 l.jpg

    Benefit of Auxiliary Variables

    • Effective sample size (N')

      • Analysis involving N cases, with auxiliary variable(s)

      • gives statistical power equivalent to N' complete cases without auxiliary variables


    Benefit of auxiliary variables74 l.jpg

    Benefit of Auxiliary Variables

    • It matters how highly Y and Z (the auxiliary variable) are correlated

    • For exampleincrease

      • rYZ = .40 N = 500 gives power of N' = 542( 8%)

      • rYZ = .60 N = 500 gives power of N' = 608 (22%)

      • rYZ = .80 N = 500 gives power of N' = 733(47%)

      • rYZ = .90 N = 500 gives power of N' = 839(68%)


    Effective sample size by r yz l.jpg

    Effective Sample Size by rYZ

    Effective

    Sample

    Size

    rYZ


    Conclusions l.jpg

    Conclusions

    • Attrition CAN be bad for internal validity

    • But often it's NOT nearly as bad as often feared

    • Don't rush to conclusions, even with rather substantial attrition

    • Examine evidence (especially longitudinal diagnostics) before drawing conclusions

    • Use MI and ML missing data procedures!

    • Use good auxiliary variables to minimize impact of attrition


    Part 3 illustration of missing data analysis multiple imputation with norm and proc mi l.jpg

    Part 3:Illustration of Missing Data Analysis: Multiple Imputation with NORM and Proc MI


    Multiple imputation basic steps l.jpg

    Multiple Imputation:Basic Steps

    • Impute

    • Analyze

    • Combine results


    Imputation and analysis l.jpg

    Imputation and Analysis

    • Impute 40 datasets

      • a missing value gets a different imputed value in each dataset

    • Analyze each data set with USUAL procedures

      • e.g., SAS, SPSS, LISREL, EQS, STATA

    • Save parameter estimates and SE’s


    Combine the results parameter estimates to report l.jpg

    Combine the ResultsParameter Estimates to Report

    • Average of estimate (b-weight) over 40 imputed datasets


    Combine the results standard errors to report l.jpg

    Combine the ResultsStandard Errors to Report

    Weighted sum of:

    • “within imputation” variance

      average squared standard error

      • usual kind of variability

    • “between imputation” variance

      sample variance of parameter estimates over 40 datasets

      • variability due to missing data


    Materials for spss regression l.jpg

    Materials for SPSS Regression

    Starting place

    http://methodology.psu.edu

    • downloads (you will need to get a free user ID to download all our free software)

      missing data software

      Joe Schafer's Missing Data Programs

      John Graham's Additional NORM Utilities

      http://mcgee.hhdev.psu.edu/missing/index.html


  • Login