achieved relative intervention strength models and methods
Download
Skip this Video
Download Presentation
Achieved Relative Intervention Strength: Models and Methods

Loading in 2 Seconds...

play fullscreen
1 / 22

Achieved Relative Intervention Strength: Models and Methods - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

Achieved Relative Intervention Strength: Models and Methods. Chris S. Hulleman. David S. Cordray. Presentation for the SREE Research Conference Washington, DC March 5, 2010. Overview . Conceptual Framework Definitions and Importance Indexing Fidelity as Achieved Relative Strength (ARS)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Achieved Relative Intervention Strength: Models and Methods' - habib


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
achieved relative intervention strength models and methods

Achieved Relative Intervention Strength: Models and Methods

Chris S. Hulleman

David S. Cordray

Presentation for the SREE Research Conference

Washington, DC

March 5, 2010

overview
Overview
  • Conceptual Framework
    • Definitions and Importance
    • Indexing Fidelity as Achieved Relative Strength (ARS)
  • Three examples
    • Lab and Field Experiments
    • Reading First
  • Practical Considerations and Challenges
  • Questions and discussion
definitions and implications
Definitions and Implications

Fidelity

The extent to which the implemented Tx (tTx) was faithful to the intended Tx (TTx)

Measure core intervention components

Achieved Relative Strength (ARS)

The difference between implementedcausal components in the Tx and C

tTx – tC

ARS is a default index of fidelity

Implications

Infidelity reduces construct, external, and statistical conclusion validity

slide4

Treatment Strength

Outcome

.45

.40

.35

.30

.25

.20

.15

.10

.05

.00

TTx

100

90

85

80

75

70

65

60

55

50

Infidelity

t tx

Achieved Relative Strength = 0.15

(85)-(70) = 15

tC

“Infidelity”

TC

ExpectedRelative Strength = TTx - TC = (0.40-0.15) = 0.25

indexing fidelity as achieved relative strength
Indexing Fidelity as Achieved Relative Strength

Intervention Strength = Treatment – Control

Achieved Relative Strength (ARS) Index

  • Standardized difference in fidelity index across Tx and C
  • Based on Hedges’ g (Hedges, 2007)
  • Corrected for clustering in the classroom (ICC’s from .01 to .08)
  • See Hulleman & Cordray (2009)
indexing fidelity
Indexing Fidelity

Average

  • Mean levels of observed fidelity (tTx)

Absolute

  • Compare observed fidelity (tTx) to absolute or maximum level of fidelity (TTx)

Binary

  • Yes/No treatment receipt based on fidelity scores
  • Requires selection of cut-off value
slide7

Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

Examples 1 and 2

slide8

The Theory of Change

INTEREST

MANIPULATED RELEVANCE

PERCEIVED UTILITY VALUE

PERFORMANCE

Fidelity Measure:

Quality of participant responsiveness (0 to 3 scale)

Model Adapted from: Eccles et al. (1983); Hulleman et al. (2009)

slide10

Average ARS Index

Fidelity

Achieved Relative Strength = 1.32

Treatment Strength

100

66

33

0

3

2

1

0

TTx

Infidelity

t tx

(0.74)-(0.04) = 0.70

tC

Infidelity

TC

assessing implementation fidelity in a large scale policy intervention the case of reading first

Assessing Implementation Fidelity in a Large-Scale Policy Intervention: The Case of Reading First

Example 3

slide12

In Education, Intervention Models are Multi-faceted (from Gamse et al., 2008)

Use of research-based reading programs, instructional materials, and assessment, as articulated in the LEA/school application

1)Teacher use of instructional strategies and content based on five essential components of reading instruction

2) Use of assessments to diagnose student needs and measure progress

3) Classroom organization and supplemental services and materials that support five essential components

Teacher professional development in the use of materials and instructional approaches

from major components to indicators
From Major Components to Indicators…

Indicators

Major Components

Sub-components

Facets

Scheduled block?

Block

Instructional Time

Actual Time

Reported time

Reading Instruction

Instructional Material

Instructional Activities/Strategies

Support for Struggling Readers

Assessment

Professional Development

reading first implementation specifying components and operationalization
Reading First Implementation: Specifying Components and Operationalization

Adapted from Moss et al. 2008

reading first implementation some results
Reading First Implementation: Some Results

Adapted from Gamse et al. (2008) and Moss et al. (2008)

ars how big is big enough
ARS: How Big is Big Enough?

*Averaged over 1st, 2nd, and 3rd grades (Gamse et al., 2008).

what do i do with fidelity indices
What Do I Do With Fidelity Indices?

Start with:

  • Scale construction, aggregation over model sub-components and components

Use as:

  • Descriptive analyses
  • Causal analyses (Intent-to-Treat: ITT)
  • Explanatory (AKA exploratory) analyses
    • E.g., LATE, Instrumental variables, TOT

Except for descriptive analyses, most approaches are relative new and not fully tested

in practice
In Practice….
  • Identify core intervention components
    • e.g., via a Model of Change
  • Establish bench marks for TTX and TC
  • Measurement
    • Determine indicators of core components
    • Derive tTx and tC
    • Develop scales
    • Convert to ARS
  • Incorporate into intervention analyses
    • Multi-level analyses (Justice, Mashburn, Pence, & Wiggins, 2008)
some challenges
Some Challenges

Intervention models

  • Often unclear
  • Scripted vs. Unscripted

Measurement

  • Novel constructs
  • Multiple levels
  • Aggregation (within and across levels)

Analyses

  • Weighting of components
  • Uncertainty about psychometric properties
  • Functional form not always known
summary of key points
Summary of Key Points
  • Identify and measure core components
  • Fidelity assessment serves two roles:
    • Average causal difference between conditions
    • Using fidelity measures to assess the effects of variation in implementation on outcomes
  • Post-experimental (re)specification of the intervention
  • ARS: How much is enough?
    • Need more data!
slide22

Thank You

Questions and Discussion

ad