Achieved Relative Intervention Strength: Models and Methods

1. Achieved Relative Intervention Strength: Models and Methods Chris S. Hulleman David S. Cordray Presentation for the SREE Research Conference Washington, DC March 5, 2010

2. Overview • Conceptual Framework • Definitions and Importance • Indexing Fidelity as Achieved Relative Strength (ARS) • Three examples • Lab and Field Experiments • Reading First • Practical Considerations and Challenges • Questions and discussion

3. Definitions and Implications Fidelity The extent to which the implemented Tx (tTx) was faithful to the intended Tx (TTx) Measure core intervention components Achieved Relative Strength (ARS) The difference between implementedcausal components in the Tx and C tTx – tC ARS is a default index of fidelity Implications Infidelity reduces construct, external, and statistical conclusion validity

4. Treatment Strength Outcome .45 .40 .35 .30 .25 .20 .15 .10 .05 .00 TTx 100 90 85 80 75 70 65 60 55 50 Infidelity t tx Achieved Relative Strength = 0.15 (85)-(70) = 15 tC “Infidelity” TC ExpectedRelative Strength = TTx - TC = (0.40-0.15) = 0.25

5. Indexing Fidelity as Achieved Relative Strength Intervention Strength = Treatment – Control Achieved Relative Strength (ARS) Index • Standardized difference in fidelity index across Tx and C • Based on Hedges’ g (Hedges, 2007) • Corrected for clustering in the classroom (ICC’s from .01 to .08) • See Hulleman & Cordray (2009)

6. Indexing Fidelity Average • Mean levels of observed fidelity (tTx) Absolute • Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx) Binary • Yes/No treatment receipt based on fidelity scores • Requires selection of cut-off value

7. Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention Examples 1 and 2

8. The Theory of Change INTEREST MANIPULATED RELEVANCE PERCEIVED UTILITY VALUE PERFORMANCE Fidelity Measure: Quality of participant responsiveness (0 to 3 scale) Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)

9. Achieved Relative Strength Indices

10. Average ARS Index Fidelity Achieved Relative Strength = 1.32 Treatment Strength 100 66 33 0 3 2 1 0 TTx Infidelity t tx (0.74)-(0.04) = 0.70 tC Infidelity TC

11. Assessing Implementation Fidelity in a Large-Scale Policy Intervention: The Case of Reading First Example 3

12. In Education, Intervention Models are Multi-faceted (from Gamse et al., 2008) Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application 1)Teacher use of instructional strategies and content based on five essential components of reading instruction 2) Use of assessments to diagnose student needs and measure progress 3) Classroom organization and supplemental services and materials that support five essential components Teacher professional development in the use of materials and instructional approaches

13. From Major Components to Indicators… Indicators Major Components Sub-components Facets Scheduled block? Block Instructional Time Actual Time Reported time Reading Instruction Instructional Material Instructional Activities/Strategies Support for Struggling Readers Assessment Professional Development

14. Reading First Implementation: Specifying Components and Operationalization Adapted from Moss et al. 2008

15. Reading First Implementation: Some Results Adapted from Gamse et al. (2008) and Moss et al. (2008)

16. Linking Fidelity to Outcomes

17. ARS: How Big is Big Enough? *Averaged over 1st, 2nd, and 3rd grades (Gamse et al., 2008).

18. What Do I Do With Fidelity Indices? Start with: • Scale construction, aggregation over model sub-components and components Use as: • Descriptive analyses • Causal analyses (Intent-to-Treat: ITT) • Explanatory (AKA exploratory) analyses • E.g., LATE, Instrumental variables, TOT Except for descriptive analyses, most approaches are relative new and not fully tested

19. In Practice…. • Identify core intervention components • e.g., via a Model of Change • Establish bench marks for TTX and TC • Measurement • Determine indicators of core components • Derive tTx and tC • Develop scales • Convert to ARS • Incorporate into intervention analyses • Multi-level analyses (Justice, Mashburn, Pence, & Wiggins, 2008)

20. Some Challenges Intervention models • Often unclear • Scripted vs. Unscripted Measurement • Novel constructs • Multiple levels • Aggregation (within and across levels) Analyses • Weighting of components • Uncertainty about psychometric properties • Functional form not always known

21. Summary of Key Points • Identify and measure core components • Fidelity assessment serves two roles: • Average causal difference between conditions • Using fidelity measures to assess the effects of variation in implementation on outcomes • Post-experimental (re)specification of the intervention • ARS: How much is enough? • Need more data!

22. Thank You Questions and Discussion