1 / 49

Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis

Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis. David S. Cordray & Chris Hulleman Vanderbilt University Presentation for the IES Research Conference Washington, DC June 9, 2009. Overview . Fidelity and Achieved Relative Strength:

dympna
Download Presentation

Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Assessing Intervention Fidelity in RCTs: Models, Methods and Modes of Analysis David S. Cordray & Chris Hulleman Vanderbilt University Presentation for the IES Research Conference Washington, DC June 9, 2009

  2. Overview • Fidelity and Achieved Relative Strength: • Definitions, distinctions and illustrations • Conceptual foundation for assessing fidelity in RCTs • Achieved relative strength, a special case in RCTs • Modes of analysis: • Approaches and challenges • Chris Hulleman -- Assessing implementation fidelity and achieved relative strength indices: The single core component case • Questions and discussion

  3. Distinguishing Implementation Assessment from the Assessment of Implementation Fidelity • Two ends on a continuum of intervention implementation/fidelity: • A purelydescriptive model: • Answering the question “What transpired as the intervention was put in place (implemented). • Based on a priori intervention model, with explicit expectations about implementation of program components: • Fidelity is the extent to which the realized intervention (tTx) is faithful to the pre-stated intervention model (TTx ) • Infidelity = TTx – tTx • Most implementation fidelity assessments involve descriptive and model-based approaches.

  4. Dimensions Intervention Fidelity • Aside from agreement at the extremes, little consensus on what is meant by the term “intervention fidelity”. • Most frequent definitions: • True Fidelity = Adherence or compliance: • Program components are delivered/used/received, as prescribed • With a stated criteria for success or full adherence • The specification of these criteria is relatively rare • Intervention Exposure: • Amount of program content, processes, activities delivered/received by all participants (aka, receipt, responsiveness) • This notion is most prevalent • Intervention Differentiation: • The unique features of the intervention are distinguishable from other programs, including the control condition • A unique application within RCTs

  5. Linking Intervention Fidelity Assessment to Contemporary Models of Causality Rubin’s Causal Model: True causal effect of X is (YiTx – YiC) RCT methodology is the best approximation to this true effect In RCTs, the difference between conditions, on average, is the causal effect Fidelity assessment within RCTs entails examining the difference between causal components in the intervention and control conditions. Differencing causal conditions can be characterized as achieved relative strength of the contrast. Achieved Relative Strength (ARS) = tTx – tC ARS is a default index of fidelity

  6. Treatment Strength Outcome .45 .40 .35 .30 .25 .20 .15 .10 .05 .00 TTx 100 90 85 80 75 70 65 60 55 50 Infidelity t tx Achieved Relative Strength =.15 (85)-(70) = 15 tC “Infidelity” TC ExpectedRelative Strength = (0.40-0.15) = 0.25

  7. Why is this Important? • Statistical Conclusion validity • Unreliability of Treatment Implementation: Variations across participants in the delivery receipt of the causal variable (e.g., treatment). Increases error and reduces the size of the effect; decreases chances of detecting covariation. • Resulting in a reduction in statistical power or the need for a larger study….

  8. The Effects Structural Infidelity on Power .60 Fidelity .80 1.0

  9. Influence of Infidelity on Study-size 1.0 .80 Fidelity .60

  10. If That Isn’t Enough…. • Construct Validity: • Which is the cause? (TTx - TC) or (tTx – tC) • Poor implementation: essential elements of the treatment are incompletely implemented. • Contamination: The essential elements of the treatment group are found in the control condition (to varying degrees). • Pre-existing similarities between T and C on intervention components. • External validity – generalization is about (tTx - tC) • This difference needs to be known for proper generalization and future specification of the intervention components

  11. Infidelity Augmentation of Control PD= Professional Development Asmt=Formative Assessment Diff Inst= Differentiated Instruction So what is the cause? …The achieved relative difference in conditions across components

  12. TTX’TTx Infidelity t tx tC “Infidelity” Positive Infidelity .45 .40 .35 .30 .25 .20 .15 .10 .05 .00 Intervention Exposure 100 90 85 80 75 70 65 60 55 50 True Fidelity Intervention Differentiation Achieved Relative Strength =.15 TC Tx Contamination Augmentation of C Intervention Exposure Treatment Strength Outcome Review: Concepts and Definitions

  13. Some Sources and Types of Infidelity • If delivery or receipt could be dichotomized (yes or no): • Simple fidelity involves compliers; • Simple infidelity involves “No shows” and cross-overs. • Structural flaws in implementing the intervention: • Missing or incomplete resources, processes • External constraints (e.g. snow days) • Incomplete delivery of core intervention components • Implementer failures or incomplete delivery

  14. A Tutoring Program: Variation in Exposure 4-5 tutoring sessions per week, 25 minutes each, 11weeks Expectations: 44-55 sessions Random Assignment of Students Time  Cycle 1 47.7 16-56 Cycle 2 33.1 12-42 Cycle 3 31.6 16-44 Average Sessions Delivered Range

  15. Average Number of Tutoring Sessions per Tutor Individual Tutors Variation in Exposure: Tutor Effects The other fidelity question: How faithful to the tutoring model is each tutor?

  16. In Practice…. • Identify core components in the intervention group • e.g., via a Model of Change • Establish bench marks (if possible) for TTX and TC • Measure core components to derive tTx and tC • e.g., via a “Logic model” based on Model of Change • Measurement (deriving indicators) • Converted to Achieved Relative Strength and implementation fidelity scales • Incorporated into the analysis of effects

  17. What do we measure? What are the options? (1) Essential or core components (activities, processes); (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of T); and (3) Ordinary features of the setting (shared with the control group) • Focus on 1 and 2.

  18. Fidelity Assessment Starts With a Model or Framework for the Intervention From: Gamse et al. 2008

  19. Core Reading Components for Local Reading First Programs Design and Implementation of Research-Based Reading Programs Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application 1)Teacher use of instructional strategies and content based on five essential components of reading instruction 2) Use of assessments to diagnose student needs and measure progress 3) Classroom organization and supplemental services and materials that support five essential components Teacher professional development in the use of materials and instructional approaches After Gamse et al. 2008

  20. From Major Components to Indicators… Indicators Major Components Sub-components Facets Scheduled block? Block Instructional Time Actual Time Reported time Instructional Material Reading Instruction Instructional Activities/Strategies Support for Struggling Readers Assessment Professional Development

  21. Reading First Implementation: Specifying Components and Operationalization Adapted from Moss et al. 2008

  22. Reading First Implementation: Some Results Adapted from Moss et al. 2008

  23. So What Do I Do With All This Data? • Start with: • Scale construction, aggregation over facets, sub-components, components • Use as: • Descriptive analyses • Explanatory (AKA exploratory) analyses • There are a lot of options • In this section we describe a hierarchy of analyses, higher to lower levels of causal inference • Caveat: Except for descriptive analyses, most approaches are relative new and not fully tested.

  24. Hierarchy of Approaches to Analysis: • ITT (Intent-to-treat) estimates (e.g., ES) plus: • an index of true fidelity: • ES=.50 Fidelity = 96% • an index of Achieved Relative Strength (ARS). • Hulleman’s initial analysis: ES=0.45, ARS=0.92. • LATE (Local Average Treatment Effect): • If treatment receipt/delivery can be meaningfully dichotomized and there is experimentally induced receipt or non-receipt of treatment: • adjust ITT estimate by T and C treatment receipt rates. • Simple model can be extended to an Instrumental Variable Analysis (see Bloom’s 2005 book). • ITT retains causal status; LATE can approximate causal statements.

  25. More on Fidelity to Outcome Linkages • TOT (Treatment-on-Treated) • Simple: ITT estimate adjusted for compliance rate in Tx, no randomization. • Two-level linear production function, modeling the effects of implementation factors in Tx and modeling factors affecting C in separate Level 2 equations. • Regression-based model, exchanging implementation fidelity scales for treatment exposure variable.

  26. Descriptive Analyses • Fidelity is often examined in the intervention group, only. • Dose-response relationship • Partition intervention sites into “high” and “low” implementation fidelity: • My review of some ATOD prevention studies, the ESHIGH =0.13 to 0.18 ESLOW =0.00 to 0.03

  27. Some Challenges • Interventions are rarely clear; • Measurement involves novel constructs; • How should components be weighted? If at all. • Fidelity assessment occurs at multiple levels; • Fidelity indicators are used in 2nd and 3rd levels of HLM models, few degrees of freedom; • There is uncertainty about the psychometric properties of fidelity indicators; and • Functional form of fidelity and outcome measures is not always known. • But, despite these challenges, Chris Hulleman has a dandy example…

  28. Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

  29. The Theory of Change INTEREST MANIPULATED RELEVANCE PERCEIVED UTILITY VALUE PERFORMANCE Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)

  30. Methods(Hulleman & Cordray, 2009)

  31. Motivational Outcome ? g = 0.05 (p = .67)

  32. Fidelity Measurement and Achieved Relative Strength • Simple intervention – one core component • Intervention fidelity: • Exposure: “quality of participant responsiveness” • Rated on scale from 0 (none) to 3 (high) • 2 independent raters, 88% agreement

  33. Exposure

  34. Indexing Fidelity Absolute • Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx) Average • Mean levels of observed fidelity (tTx) Binary • Yes/No treatment receipt based on fidelity scores • Requires selection of cut-off value

  35. Fidelity Indices

  36. Indexing Fidelity as Achieved Relative Strength Intervention Strength = Treatment – Control Achieved Relative Strength (ARS) Index • Standardized difference in fidelity index across Tx and C • Based on Hedges’ g (Hedges, 2007) • Corrected for clustering in the classroom (ICC’s from .01 to .08) • See Hulleman & Cordray (2009)

  37. Average ARS Index Group Difference Sample Size Adjustment Clustering Adjustment Where, = mean for group 1 (tTx ) = mean for group 2 (tC) ST = pooled within groups standard deviation nTx = treatment sample size nC = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size

  38. Absolute and Binary ARS Indices Group Difference Sample Size Adjustment Clustering Adjustment Where, pTx = proportion for the treatment group (tTx ) pC = proportion for the control group (tC) nTx = treatment sample size nC = control sample size n = average cluster size p = Intra-class correlation (ICC) N = total sample size

  39. Average ARS Index Fidelity Achieved Relative Strength = 1.32 Treatment Strength 100 66 33 0 3 2 1 0 TTx Infidelity t tx (0.74)-(0.04) = 0.70 tC Infidelity TC

  40. Achieved Relative Strength Indices

  41. Linking Achieved Relative Strength to Outcomes

  42. Sources of Infidelity in the Classroom Student behaviors were nested within teacher behaviors • Teacher dosage • Frequency of student exposure Student and teacher behaviors were used to predict treatment fidelity (i.e., quality of responsiveness/exposure).

  43. Sources of Infidelity: Multi-level Analyses Part I: Baseline Analyses • Identified the amount of residual variability in fidelity due to students and teachers. • Du to missing data, we estimated a 2-level model (153 students, 6 teachers) Student: Yij = b0j + b1j(TREATMENT)ij+ rij, Teacher: b0j = γ00 + u0j, b1j = γ10 + u10j

  44. Sources of Infidelity: Multi-level Analyses Part II: Explanatory Analyses • Predicted residual variability in fidelity (quality of responsiveness) with frequency of responsiveness and teacher dosage Student: Yij = b0j + b1(TREATMENT)ij+ b2(RESPONSE FREQUENCY)ij+ rij Teacher: b0j = γ00 + u0j b1j = γ10 + b10(TEACHER DOSAGE)j+ u10j b2j = γ20 + b20(TEACHER DOSAGE)j+ u20j

  45. Sources of Infidelity: Multi-level Analyses * p < .001.

  46. Case Summary The motivational intervention was more effective in the lab (g = 0.45) than field (g = 0.05). Using 3 indices of fidelity and, in turn, achieved relative treatment strength, revealed that: Classroom fidelity < Lab fidelity Achieved relative strength was about 1 SD less in the classroom than the laboratory Differences in achieved relative strength = differences motivational outcome, especially in the lab. Sources of fidelity: teacher (not student) factors

  47. Key Points and Issues • Identifying and measuring, at a minimum, should include model-based core and necessary components • Collaborations among researchers and practitioners (e.g., developers and implementers) is essential for specifying: • Intervention models • Core and essential components • Benchmarks for TTx (e.g., an educationally meaningful dose; what level of X is needed to instigate change) • Tolerable adaptation

  48. Key Points and Issues • Fidelity assessment serves two roles: • Average causal difference between conditions; and • Using fidelity measures to assess the effects of variation in implementation on outcomes. • Post-experimental (re)specification of the intervention

  49. Thank You Questions and Discussion

More Related