Emr 6550 experimental and quasi experimental designs
This presentation is the property of its rightful owner.
Sponsored Links
1 / 31

EMR 6550: Experimental and Quasi-Experimental Designs PowerPoint PPT Presentation


  • 172 Views
  • Uploaded on
  • Presentation posted in: General

EMR 6550: Experimental and Quasi-Experimental Designs. Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013. Agenda. Quasi-experimental designs that use both control groups and pretests Interrupted time-series designs Design and power problems.

Download Presentation

EMR 6550: Experimental and Quasi-Experimental Designs

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Emr 6550 experimental and quasi experimental designs

EMR 6550:Experimental and Quasi-Experimental Designs

Dr. Chris L. S. Coryn

Kristin A. Hobson

Fall 2013


Agenda

Agenda

Quasi-experimental designs that use both control groups and pretests

Interrupted time-series designs

Design and power problems


Designs that use both control groups and pretests

Designs that Use Both Control Groups and Pretests


Untreated control group design with dependent pretest and posttest s amples

Untreated Control Group Design with Dependent Pretest and Posttest Samples

A selection bias is always present, but the pretest observation allows for determining the magnitude and direction of bias


Outcome pattern 1

Outcome Pattern 1

  • Both groups grow apart at different average rates in the same direction

Treatment

Control

  • This pattern is consistent with treatment effects and can sometimes be causally interpreted, but it is subject to numerous threats, especially selection-maturation


Outcome pattern 2

Outcome Pattern 2

  • Spontaneous growth only occurs in the treatment group

Treatment

Control

  • Not a lot of reliance can be placed on this pattern as the reasons why spontaneous growth only occurred in the treatment group must be explained (e.g., selection-maturation)


Outcome pattern 3

Outcome Pattern 3

  • Initial pretest differences favoring the treatment group diminish over time

Treatment

Control

  • Same internal validity threats as outcome patterns #1 and #2 except that selection-maturation threats are less plausible


Outcome pattern 4

Outcome Pattern 4

Control

  • Initial pretest differences favoring the control group diminish over time

Treatment

  • Subject to numerous validity threats (e.g., selection-instrumentation, selection-history), but generally can be causally interpreted


Outcome pattern 5

Outcome Pattern 5

  • Outcomes that crossover in the direction of relationships

Treatment

Control

  • Most amenable to causal interpretation and most threats cannot plausibly explain this pattern


Modeling selection bias

Modeling Selection Bias

  • Simple matching and stratifying

    • Overt biases with respect to measured variables/characteristics

  • Instrumental variable analysis

    • Statistical modeling of covariates believed to explain selection biases

  • Hidden bias analysis

    • Difference with respect to unmeasured variables/characteristics

    • Sensitivity analysis (how much hidden bias would need to be present to explain observed differences)

  • Propensity score analysis

    • Predicted probabilities of group membership

    • Propensities then used for matching or as covariate


Effect decay functions

Effect-Decay Functions

Immediate Effect, No Decay

Delayed Effect

Large

Large

Response

Response

Small

Small

ProgramOnset

ProgramTermination

ProgramOnset

ProgramTermination

Time

Time

Immediate Effect, Rapid Decay

Early Effect, Slow Decay

Large

Large

Response

Response

Small

Small

ProgramOnset

ProgramTermination

ProgramOnset

ProgramTermination

Time

Time


Untreated control group design with dependent pretest and posttest samples using a double pretest

Untreated Control Group Design with Dependent Pretest and Posttest Samples Using a Double Pretest

  • Permits assessment of selection-maturation on the assumption that the rates between O1 and O2 will continue between O2 and O3

  • Testable only on the control group


Emr 6550 experimental and quasi experimental designs

Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Switching Replications

  • A strong design and only a pattern of historical changes that mimics the time sequence of the treatment introductions can serve as an alternate explanation

  • The addition of treatment removal (X) can strengthen cause-effect claims


Emr 6550 experimental and quasi experimental designs

Untreated Control Group Design with Dependent Pretest and Posttest Samples Using Reversed Treatment Control Group

  • Interpretation of this design depends on producing two effects with opposite signs

  • Adding a control is useful

  • Ethically, often difficult to use a reversed treatment


Interrupted time series designs

Interrupted Time-Series Designs


Interuppted time series

Interuppted Time-Series

  • A large series of observations made on the same variable consecutively over time

    • Observations can be made on the same units (e.g., people) or on constantly changing units (e.g., populations)

  • Must know the exact point at which a treatment or intervention occurred (i.e., the interruption)

  • Interrupted time-series designs are powerful cause-probing designs when experimental designs cannot be used and when a time series is feasible


Types of effects

Types of Effects

Form of the effect (slope or intercept)

Permanence of the effect (continuous or discontinuous)

Immediacy of the effect (immediate or delayed)


Analytic considerations

Analytic Considerations

  • Independence of observations

    • (Most) statistical analyses assume observations are independent (one observation is independent of another)

    • In interrupted time-series, observations are autocorrelated (related to prior observations or lags)

    • Requires a large number of observations to estimate autocorrelation

  • Seasonality

    • Observations that coincide with seasonal patterns

    • Seasonality effects must be modeled and removed from a time-series before assessing treatment impact


Simple interrupted time series design

Simple Interrupted Time-Series Design

The basic interrupted time-series design requires one treatment group with many observations before and after a treatment


Change in intercept

Change in Intercept

Intervention

Change in intercept


Change in slope

Change in Slope

Intervention

Change in slope


Weak and delayed effects

Weak and Delayed Effects

Intervention

Impact begins


Validity threats

Validity Threats

  • With most interrupted time-series designs, the major validity threat is history

    • Events that occur at the same time as the treatment was introduced

  • Instrumentation is also often a threat

    • Over long time periods, methods of data collection may change, how variables are defined and/or measured may change

  • Selection is sometimes a threat

    • If group membership changes abruptly


Additional designs

Additional Designs

(1) nonequivalent control group, (2) nonequivalent dependent variable, and (3) removed treatment


Nonequivalent control group

Nonequivalent Control Group

Intervention

Treatment group

Control group


Nonequivalent dependent variable

Nonequivalent Dependent Variable

Intervention

Nonequivalent dependent variable

Dependent variable


Removed treatment

Removed Treatment

Introduction

Removal

Treatment period


Design and power problems

Design and Power Problems


Problem 1

Problem #1

  • A school administrator wants to know whether students in his district are scoring better or worse than the national norm of 500 on the SAT

  • He decides that a difference of 20-25 points or more from this normative value would be important to detect

  • He anticipates that the standard deviation of scores in his district is about 80 points

    • Determine the number of students necessary for power at 95% to detect a difference of 20 and 25 points

    • Graph both

    • Diagram the design of the study


Problem 2

Problem #2

  • Patients suffering from allergies are nonrandomly assigned to a treatment and placebo condition and asked to rate their comfort level on a scale of 0 to 100

  • The expected standard deviation is 20 and a difference of 10-20 is expected (treatment = 50-60 and placebo = 40)

    • Determine the number of patients necessary for power at 95% to detect a difference of 10 and 20 points

    • Graph both

    • Diagram the design of the study


Problem 3

Problem #3

  • The cure rate for two current treatments are 10% and 60%, respectively

  • The alternative treatments are expected to increase the cure rate by 10%

    • Determine the number of patients necessary for power at 95% to detect a difference of 10% for both scenarios

    • Graph both

    • Diagram the design of the studies


  • Login