Conceptualizing Intervention Fidelity: Implications for Measurement, Design, and Analysis. Chris S. Hulleman, Ph.D. Implementation: What to Consider At Different Stages in the Research Process Panel presentation for the Institute for Education Sciences Annual Grantee Meeting September 7, 2011.
Chris S. Hulleman, Ph.D.
Implementation: What to Consider At Different Stages in the Research Process
Panel presentation for the Institute for Education Sciences Annual Grantee Meeting
September 7, 2011
Fidelity: How faithful was the implemented intervention (tTx) to the intended intervention (TTx)?
Infidelity: TTx – tTx
Implementation Assessment Continuum
What happened as the intervention was implemented?
A priori model
How much, and with what quality, were the core intervention components implemented?
Most assessments include both
Rubin’s Causal Model:
True causal effect of X is (YiTx – YiC)
RCT is best approximation
Tx – C = average causal effect
Examines the difference between implementedcausal components in the Tx and C
This difference is the achieved relative strength (ARS) of the intervention
Theoretical relative strength = TTx – TC
Achieved relative strength = tTx – tC
Index of fidelity
(1) Essentialor corecomponents (activities, processes, structures)
(2) Necessary, but not unique,activities, processes and structures (supporting the essential components of Tx)
(3) Best practices
(4) Ordinary featuresof the setting (shared with the control group)
Statistical Conclusion Validity
Effect Size Impact of Reading First on Reading Outcomes = .05
Adapted from Gamse et al. (2008) and Moss et al. (2008)
*Averaged over 1st, 2nd, and 3rd grades (Gamse et al., 2008).
And Special Thanks to My Collaborators:
David S. Cordray
Chris Hulleman is an assistant professor at James Madison University with joint appointments in Graduate Psychology and the Center for Assessment and Research Studies. Chris also co-directs the Motivation Research Institute at James Madison. He received his PhD in social/personality psychology from the University of Wisconsin-Madison in 2007, and then spent two years as an Institute for Education Sciences Research Fellow in Vanderbilt University’s Peabody College of Education. In 2009, he won the Pintrich Outstanding Dissertation Award from Division 15 (Educational Psychology) of the American Psychological Association. He teaches courses in graduate statistics and research methods, and serves as the assessment liaison for the Division of Student Affairs. His motivation research focuses on motivation in academic, sport, work, and family settings. His methodological interests include developing guidelines for translating laboratory research into the field, and developing indices of intervention fidelity. As a Research Affiliate for the National Center on Performance Incentives, Chris is involved in several randomized field experiments of teacher pay-for-performance programs in K-12 settings. His scholarship has been published in journals such as Science, Psychological Bulletin, Journal of Research on Educational Effectiveness, Journal of Educational Psychology, and Phi Delta Kappan.
Department of Graduate Psychology
James Madison University
Achieved Relative Strength (ttx) = 0.15
(85)-(70) = 15
Expected Relative Strength = TTx - TC = (0.40-0.15) = 0.25