comments on tradeoffs and issues l.
Skip this Video
Loading SlideShow in 5 Seconds..
Comments on Tradeoffs and Issues PowerPoint Presentation
Download Presentation
Comments on Tradeoffs and Issues

Loading in 2 Seconds...

play fullscreen
1 / 13

Comments on Tradeoffs and Issues - PowerPoint PPT Presentation

  • Uploaded on

Comments on Tradeoffs and Issues William R. Shadish University of California, Merced Key Tradeoffs Causal vs noncausal questions Bias control vs context-specific feasibility Internal vs External Validity Clarifying Some Issues About the Choice of Design

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Comments on Tradeoffs and Issues

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
comments on tradeoffs and issues

Comments on Tradeoffs and Issues

William R. Shadish

University of California, Merced

key tradeoffs
Key Tradeoffs
  • Causal vs noncausal questions
  • Bias control vs context-specific feasibility
  • Internal vs External Validity
clarifying some issues about the choice of design
Clarifying Some Issues About the Choice of Design
  • There are many good reasons not to do randomized designs, and not to do experiments at all.
    • When it is not possible (treatment already started, impossibility of getting control)
    • When it is unethical
    • When it is not necessary (the counterfactual is known, PKU screening example)
    • When we are not asking a causal question
  • But let’s not reject experiments for the wrong reasons
terminological confusion
Terminological Confusion
  • Randomized trials (not equal to) efficacy trial (RT efficacy, RT effectiveness)
  • Effectiveness research (not equal to) translational research
    • Effectiveness: Does the intervention work under field conditions (an RCT is perfectly appropriate because the question is still about cause and effect)
    • Translational: How can we get practitioners to adopt and sustain, patients to accept and sustain (RCT’s largely irrelevant; more relevant are process and implementation studies, cost analysis, needs assessment, ethnography, etc).
are we asking causal questions
Are We Asking Causal Questions?
  • The question should drive the method. Both efficacy and effectiveness are causal questions. But many translational research questions are not.
  • Randomization most helpful for cause and effect questions—effectiveness or efficacy.
  • Quasi-experiments are also helpful (RD, ITS, etc.)
  • But if the question is not causal, the method should be different. For example
    • What is the need for treatment?
    • How do stakeholders value treatments and outcomes?
    • How are resources distributed?
    • How is treatment implemented?
    • Costs of treatment?
questionable reasons to reject randomized experiments
Questionable Reasons to Reject Randomized Experiments
  • Reasons that often apply to many nonrandomized experiments, as well:
    • Value of examining treatment implementation
    • Low power when number of units is small (remember Henry Feldman’s low power for the 3 group quasi-exp; it would have been lower with selection modeling).
    • Nesting of units in groups
    • The intervention is more than just the program, but includes staff, context etc.
    • We don’t know which component of tmt worked
  • The constant treatment assumption: Randomized trials do not require constant treatments, especially not effectiveness trials. (Angrist et al. JASA)
questionable reasons continued
Questionable Reasons, continued
  • Randomized trials may or may not be more expensive, more time consuming, or less powerful than other studies, depending on what the other study design is. For example:
    • Power ITS > RE > QE with selection model?
    • Can you do ITS with archival data or do you have to gather your own data anew? This will affect both time and expense.
  • The cost of making an inferential error.
  • But in general, more money buys you better data no matter what the design is.
using multiple design elements in quasi experiments
Using Multiple Design Elements in Quasi-Experiments
  • Quasi-experiments are not a set of fixed designs
  • Rather, each quasi-experiment should be an aggregation of many design elements that each are thoughtfully chosen to compensate for the possible threats to causal inference
  • Moreover, good quasi-experiments use multiple designs to address the same questions (e.g., Jim Dwyer on Agent Orange)
an example reynolds and west
An Example: Reynolds and West
  • Arizona’s “Ask for the Sale” campaign to sell lottery tickets.
  • Nonequivalent control group (stores agreeing to participate vs not)
  • Four pretests and four posttests to assess trends and possible regression artifacts
  • three nonequivalent dependent variables (treatment increased ticket sales but not sales of gas, cigarettes, or grocery items).
  • Examined some stores in which treatment was removed and repeated, or was initiated later
role of meta analysis
Role of Meta-Analysis
  • The question of generalizability of effects from RCT is an empirical question, not a fact.
    • Results from nonrepresentative samples may indeed generalize to representative samples
  • There are lots of ways to look at generalizability, of which meta-analysis of experiments (randomized or quasi) is one.
    • Look at large numbers of RCTs and QEs
    • Examine variability of effects over different kinds of units, persons, settings, and outcomes
    • Use response surface modeling to predict effects of ideal study (e.g., most clinically representative)
meta analysis continued
Meta-Analysis, continued
  • Publication bias
    • We have technologies to study that.
    • It’s when we don’t do meta-analysis that we have this problem.
summary of agreements
Summary of Agreements
  • The question should drive the method
  • Translational research involves more than just causal questions, and so requires many different kinds of design.
  • Even when effects are at issues, randomized experiments are sometimes not possible or ethical, and nonrandomized experiments must be considered.
  • In the hierarchy of nonrandomized designs, time series and regression discontinuity are generally preferred on theoretical grounds, when practical.
summary of agreements13
Summary of Agreements
  • We have made a lot of progress on addressing problems in both randomized and nonrandomized experiments (things I’ve not heard mentioned here), for example:
    • Incomplete treatment implementation (Angrist)
    • Missing data
    • Selection bias and propensity scores
  • Triangulating using different designs is a good thing (e.g., time series on archival data with surveys on other dv’s; or RCT on proximal outcomes with observational study on long-term outcome).