Achieved relative intervention strength models and methods
Download
1 / 22

Achieved Relative Intervention Strength: Models and Methods - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

Achieved Relative Intervention Strength: Models and Methods. Chris S. Hulleman. David S. Cordray. Presentation for the SREE Research Conference Washington, DC March 5, 2010. Overview . Conceptual Framework Definitions and Importance Indexing Fidelity as Achieved Relative Strength (ARS)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Achieved Relative Intervention Strength: Models and Methods' - habib


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Achieved relative intervention strength models and methods l.jpg

Achieved Relative Intervention Strength: Models and Methods

Chris S. Hulleman

David S. Cordray

Presentation for the SREE Research Conference

Washington, DC

March 5, 2010


Overview l.jpg
Overview

  • Conceptual Framework

    • Definitions and Importance

    • Indexing Fidelity as Achieved Relative Strength (ARS)

  • Three examples

    • Lab and Field Experiments

    • Reading First

  • Practical Considerations and Challenges

  • Questions and discussion


Definitions and implications l.jpg
Definitions and Implications

Fidelity

The extent to which the implemented Tx (tTx) was faithful to the intended Tx (TTx)

Measure core intervention components

Achieved Relative Strength (ARS)

The difference between implementedcausal components in the Tx and C

tTx – tC

ARS is a default index of fidelity

Implications

Infidelity reduces construct, external, and statistical conclusion validity


Slide4 l.jpg

Treatment Strength

Outcome

.45

.40

.35

.30

.25

.20

.15

.10

.05

.00

TTx

100

90

85

80

75

70

65

60

55

50

Infidelity

t tx

Achieved Relative Strength = 0.15

(85)-(70) = 15

tC

“Infidelity”

TC

ExpectedRelative Strength = TTx - TC = (0.40-0.15) = 0.25


Indexing fidelity as achieved relative strength l.jpg
Indexing Fidelity as Achieved Relative Strength

Intervention Strength = Treatment – Control

Achieved Relative Strength (ARS) Index

  • Standardized difference in fidelity index across Tx and C

  • Based on Hedges’ g (Hedges, 2007)

  • Corrected for clustering in the classroom (ICC’s from .01 to .08)

  • See Hulleman & Cordray (2009)


Indexing fidelity l.jpg
Indexing Fidelity

Average

  • Mean levels of observed fidelity (tTx)

    Absolute

  • Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx)

    Binary

  • Yes/No treatment receipt based on fidelity scores

  • Requires selection of cut-off value


Slide7 l.jpg

Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

Examples 1 and 2


Slide8 l.jpg

The Theory of Change Classrooms: The Case of a Motivation Intervention

INTEREST

MANIPULATED RELEVANCE

PERCEIVED UTILITY VALUE

PERFORMANCE

Fidelity Measure:

Quality of participant responsiveness (0 to 3 scale)

Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)


Achieved relative strength indices l.jpg
Achieved Relative Strength Indices Classrooms: The Case of a Motivation Intervention


Slide10 l.jpg

Average ARS Index Classrooms: The Case of a Motivation Intervention

Fidelity

Achieved Relative Strength = 1.32

Treatment Strength

100

66

33

0

3

2

1

0

TTx

Infidelity

t tx

(0.74)-(0.04) = 0.70

tC

Infidelity

TC


Assessing implementation fidelity in a large scale policy intervention the case of reading first l.jpg

Assessing Implementation Fidelity in a Large-Scale Policy Intervention: The Case of Reading First

Example 3


Slide12 l.jpg

In Education, Intervention Models are Multi-faceted Intervention: The Case of Reading First(from Gamse et al., 2008)

Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application

1)Teacher use of instructional strategies and content based on five essential components of reading instruction

2) Use of assessments to diagnose student needs and measure progress

3) Classroom organization and supplemental services and materials that support five essential components

Teacher professional development in the use of materials and instructional approaches


From major components to indicators l.jpg
From Major Components to Indicators… Intervention: The Case of Reading First

Indicators

Major Components

Sub-components

Facets

Scheduled block?

Block

Instructional Time

Actual Time

Reported time

Reading Instruction

Instructional Material

Instructional Activities/Strategies

Support for Struggling Readers

Assessment

Professional Development


Reading first implementation specifying components and operationalization l.jpg
Reading First Implementation: Specifying Components and Operationalization

Adapted from Moss et al. 2008


Reading first implementation some results l.jpg
Reading First Implementation: Some Results Operationalization

Adapted from Gamse et al. (2008) and Moss et al. (2008)


Linking fidelity to outcomes l.jpg
Linking Fidelity to Outcomes Operationalization


Ars how big is big enough l.jpg
ARS: How Big is Big Enough? Operationalization

*Averaged over 1st, 2nd, and 3rd grades (Gamse et al., 2008).


What do i do with fidelity indices l.jpg
What Do I Do With Fidelity Indices? Operationalization

Start with:

  • Scale construction, aggregation over model sub-components and components

    Use as:

  • Descriptive analyses

  • Causal analyses (Intent-to-Treat: ITT)

  • Explanatory (AKA exploratory) analyses

    • E.g., LATE, Instrumental variables, TOT

      Except for descriptive analyses, most approaches are relative new and not fully tested


In practice l.jpg
In Practice…. Operationalization

  • Identify core intervention components

    • e.g., via a Model of Change

  • Establish bench marks for TTX and TC

  • Measurement

    • Determine indicators of core components

    • Derive tTx and tC

    • Develop scales

    • Convert to ARS

  • Incorporate into intervention analyses

    • Multi-level analyses (Justice, Mashburn, Pence, & Wiggins, 2008)


Some challenges l.jpg
Some Challenges Operationalization

Intervention models

  • Often unclear

  • Scripted vs. Unscripted

    Measurement

  • Novel constructs

  • Multiple levels

  • Aggregation (within and across levels)

    Analyses

  • Weighting of components

  • Uncertainty about psychometric properties

  • Functional form not always known


Summary of key points l.jpg
Summary of Key Points Operationalization

  • Identify and measure core components

  • Fidelity assessment serves two roles:

    • Average causal difference between conditions

    • Using fidelity measures to assess the effects of variation in implementation on outcomes

  • Post-experimental (re)specification of the intervention

  • ARS: How much is enough?

    • Need more data!


Slide22 l.jpg

Thank You Operationalization

Questions and Discussion