Comments on tradeoffs and issues
1 / 13

Comments on Tradeoffs and Issues - PowerPoint PPT Presentation

  • Uploaded on

Comments on Tradeoffs and Issues William R. Shadish University of California, Merced Key Tradeoffs Causal vs noncausal questions Bias control vs context-specific feasibility Internal vs External Validity Clarifying Some Issues About the Choice of Design

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Comments on Tradeoffs and Issues' - oshin

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Comments on tradeoffs and issues l.jpg

Comments on Tradeoffs and Issues

William R. Shadish

University of California, Merced

Key tradeoffs l.jpg
Key Tradeoffs

  • Causal vs noncausal questions

  • Bias control vs context-specific feasibility

  • Internal vs External Validity

Clarifying some issues about the choice of design l.jpg
Clarifying Some Issues About the Choice of Design

  • There are many good reasons not to do randomized designs, and not to do experiments at all.

    • When it is not possible (treatment already started, impossibility of getting control)

    • When it is unethical

    • When it is not necessary (the counterfactual is known, PKU screening example)

    • When we are not asking a causal question

  • But let’s not reject experiments for the wrong reasons

Terminological confusion l.jpg
Terminological Confusion

  • Randomized trials (not equal to) efficacy trial (RT efficacy, RT effectiveness)

  • Effectiveness research (not equal to) translational research

    • Effectiveness: Does the intervention work under field conditions (an RCT is perfectly appropriate because the question is still about cause and effect)

    • Translational: How can we get practitioners to adopt and sustain, patients to accept and sustain (RCT’s largely irrelevant; more relevant are process and implementation studies, cost analysis, needs assessment, ethnography, etc).

Are we asking causal questions l.jpg
Are We Asking Causal Questions?

  • The question should drive the method. Both efficacy and effectiveness are causal questions. But many translational research questions are not.

  • Randomization most helpful for cause and effect questions—effectiveness or efficacy.

  • Quasi-experiments are also helpful (RD, ITS, etc.)

  • But if the question is not causal, the method should be different. For example

    • What is the need for treatment?

    • How do stakeholders value treatments and outcomes?

    • How are resources distributed?

    • How is treatment implemented?

    • Costs of treatment?

Questionable reasons to reject randomized experiments l.jpg
Questionable Reasons to Reject Randomized Experiments

  • Reasons that often apply to many nonrandomized experiments, as well:

    • Value of examining treatment implementation

    • Low power when number of units is small (remember Henry Feldman’s low power for the 3 group quasi-exp; it would have been lower with selection modeling).

    • Nesting of units in groups

    • The intervention is more than just the program, but includes staff, context etc.

    • We don’t know which component of tmt worked

  • The constant treatment assumption: Randomized trials do not require constant treatments, especially not effectiveness trials. (Angrist et al. JASA)

Questionable reasons continued l.jpg
Questionable Reasons, continued

  • Randomized trials may or may not be more expensive, more time consuming, or less powerful than other studies, depending on what the other study design is. For example:

    • Power ITS > RE > QE with selection model?

    • Can you do ITS with archival data or do you have to gather your own data anew? This will affect both time and expense.

  • The cost of making an inferential error.

  • But in general, more money buys you better data no matter what the design is.

Using multiple design elements in quasi experiments l.jpg
Using Multiple Design Elements in Quasi-Experiments

  • Quasi-experiments are not a set of fixed designs

  • Rather, each quasi-experiment should be an aggregation of many design elements that each are thoughtfully chosen to compensate for the possible threats to causal inference

  • Moreover, good quasi-experiments use multiple designs to address the same questions (e.g., Jim Dwyer on Agent Orange)

An example reynolds and west l.jpg
An Example: Reynolds and West

  • Arizona’s “Ask for the Sale” campaign to sell lottery tickets.

  • Nonequivalent control group (stores agreeing to participate vs not)

  • Four pretests and four posttests to assess trends and possible regression artifacts

  • three nonequivalent dependent variables (treatment increased ticket sales but not sales of gas, cigarettes, or grocery items).

  • Examined some stores in which treatment was removed and repeated, or was initiated later

Role of meta analysis l.jpg
Role of Meta-Analysis

  • The question of generalizability of effects from RCT is an empirical question, not a fact.

    • Results from nonrepresentative samples may indeed generalize to representative samples

  • There are lots of ways to look at generalizability, of which meta-analysis of experiments (randomized or quasi) is one.

    • Look at large numbers of RCTs and QEs

    • Examine variability of effects over different kinds of units, persons, settings, and outcomes

    • Use response surface modeling to predict effects of ideal study (e.g., most clinically representative)

Meta analysis continued l.jpg
Meta-Analysis, continued

  • Publication bias

    • We have technologies to study that.

    • It’s when we don’t do meta-analysis that we have this problem.

Summary of agreements l.jpg
Summary of Agreements

  • The question should drive the method

  • Translational research involves more than just causal questions, and so requires many different kinds of design.

  • Even when effects are at issues, randomized experiments are sometimes not possible or ethical, and nonrandomized experiments must be considered.

  • In the hierarchy of nonrandomized designs, time series and regression discontinuity are generally preferred on theoretical grounds, when practical.

Summary of agreements13 l.jpg
Summary of Agreements

  • We have made a lot of progress on addressing problems in both randomized and nonrandomized experiments (things I’ve not heard mentioned here), for example:

    • Incomplete treatment implementation (Angrist)

    • Missing data

    • Selection bias and propensity scores

  • Triangulating using different designs is a good thing (e.g., time series on archival data with surveys on other dv’s; or RCT on proximal outcomes with observational study on long-term outcome).