Causal Inference in Instructional Research

1. Causal Inference in Instructional Research

2. Discussion Focus • Research goal of understanding (not forecasting) • Quantitative inquiry (not qualitative) • A common sense view of causation (no review of different definitions) • Available ER courses and resources (not other departments or colleges)

3. Causation: One Definition • A causal relationship exists if “the manipulation of a cause will result in the manipulation of an effect” (Cook and Campbell, 1979, p. 36). • See Cook and Campbell for a comparison of different philosophical positions on causation.

4. Logic of Causal Inference • To infer that the correlation between two variables (or the difference in mean outcomes for two groups) is in part due to a causal effect, one must: • “rule out alternative explanations (or rival hypotheses),” aka • “eliminate all threats to internal validity,” aka • “isolate the effect of one variable on the other, controlling for all other possible causes.”

5. Role of Instrument Reliability and Validity in Causal Inference • Construct validity required to ensure that an empirical relationship reflects the putative cause and effect • Reasonable instrument reliability required to minimize the threat to statistical validity • ER courses: e.g., EDF 5432

6. Overview of Study Designs for Causal Inference • Experimental studies: Researcher controls the treatment intervention and the random assignment of subjects to treatments. • Quasi-experimental studies: Researcher controls the treatment intervention, but can’t assign subjects randomly. Statistical control may be used. • Non-experimental (aka correlational) studies: Researcher controls only the selection of variables to be used for statistical control.

7. Experimental Studies (I) • Payoff of random assignment: Control the threat of “non-equivalent groups.” • Remaining threats to internal validity due primarily to focused inequities and to differential mortality. • Reasons why an experiment may not be possible or desirable include, e.g., impossibility of manipulation (e.g., gender), ethical concerns (e.g., exposure to disease), and need for quick policy decisions.

8. Experimental Studies (II): Related ER Courses • Research Design (EDF 5481) • Statistical Analyses w/ ANOVA and ANCOVA • EDF 5402 (classical) • EDF 5401 (regression approach) • EDF 5406 (multiple outcomes with MANOVA) • Alternative analyses (e.g., nonparametric with EDF 5410) • Note validity of causal inference not determined by analysis method

9. Quasi-Experimental Studies (I) • Introduces a new threat compared to experiments – nonequivalent groups • A range of possible nonequivalent (‘ne’) group designs – from • Posttest-only with ne groups • One group pretest-posttest design • Control group design with pretest and posttest • Interrupted time series design

10. Quasi-Experimental Studies (II): Related ER Courses • Research design (EDF 5481) • Statistical analysis: • ANOVA and ANCOVA designs with increased attention to statistical control for nonequivalent groups (EDF 5401 and 5402) • Time series analysis (Stat department)

11. Non-experimental Studies (I) • Bivariate correlation wo/ control • “Correlation does not imply causation” (spurious correlation) • Necessary but not sufficient condition for causation • ER courses: EDF 5400, 5410 • Bivariate correlation w/ control (e.g., partial correlation)

12. Non-experimental Studies (II): Statistical Control with Multiple Regression • Results in valid causal inference when correctly specified • Threat to validity of causal inference is incorrect model specification (e.g., leaving out important variables) • There is no empirical proof of correctness of a model • Must rely on theoretical justification of model

13. Non-experimental Studies (III): Statistical Control with Multiple Regression (Continued) • Some limitations: • Only estimates the direct causal effects (i.e., no mediation) • No test of model consistency with data (i.e., can’t falsify a model) • ER courses: EDF 5401 (linear models)

14. Non-experimental Studies (IV): Structural Equation Modeling (SEM) • Hypothesize a causal model (see next slide). The hypothesized direct effects imply indirect effects. • Test the fit of the model to data. • If fit not acceptable (i.e., if the model is falsified), consider possible model revisions. • Once acceptable fit found, describe direct, indirect, and total causal effects.

15. Path Diagram for Hypothesized Model z 1 g 11 Aptitude Motivation b 31 ( y ) ( x ) 1 1 g 31 Achievement z 3 f ( y ) b 12 3 21 g b 22 32 SES Study Habits ( x ) ( y ) 2 2 z 2

16. Non-experimental Studies (V): Structural Equation Modeling (Cont.) • Primary threat to validity of causal inference – incorrect model specification • No empirical test for correctness of model • When the model fits the data, it can be concluded that the “model is consistent with the data,” not that the model has been proven true • ER course: EDF 5401 (EDF 5409 changing to 100% HLM)

17. A COE Resource on Methods of Inquiry • Go to College of Education web site and click on “Inquiry Skills” • Help needed on revision and maintenance of site

18. Some Concluding Comments • Strength of causal inference is a continuum, not a dichotomy • Causal inference is possible with experimental, quasi-experimental, and non-experimental (correlational) studies • The strength of causal inference: • Depends on study and analysis logic allowing for the elimination of alternative explanations, • Does not depend on the analysis method