Comments on Tradeoffs and Issues

1. Comments on Tradeoffs and Issues William R. Shadish University of California, Merced

2. Key Tradeoffs • Causal vs noncausal questions • Bias control vs context-specific feasibility • Internal vs External Validity

3. Clarifying Some Issues About the Choice of Design • There are many good reasons not to do randomized designs, and not to do experiments at all. • When it is not possible (treatment already started, impossibility of getting control) • When it is unethical • When it is not necessary (the counterfactual is known, PKU screening example) • When we are not asking a causal question • But let’s not reject experiments for the wrong reasons

4. Terminological Confusion • Randomized trials (not equal to) efficacy trial (RT efficacy, RT effectiveness) • Effectiveness research (not equal to) translational research • Effectiveness: Does the intervention work under field conditions (an RCT is perfectly appropriate because the question is still about cause and effect) • Translational: How can we get practitioners to adopt and sustain, patients to accept and sustain (RCT’s largely irrelevant; more relevant are process and implementation studies, cost analysis, needs assessment, ethnography, etc).

5. Are We Asking Causal Questions? • The question should drive the method. Both efficacy and effectiveness are causal questions. But many translational research questions are not. • Randomization most helpful for cause and effect questions—effectiveness or efficacy. • Quasi-experiments are also helpful (RD, ITS, etc.) • But if the question is not causal, the method should be different. For example • What is the need for treatment? • How do stakeholders value treatments and outcomes? • How are resources distributed? • How is treatment implemented? • Costs of treatment?

6. Questionable Reasons to Reject Randomized Experiments • Reasons that often apply to many nonrandomized experiments, as well: • Value of examining treatment implementation • Low power when number of units is small (remember Henry Feldman’s low power for the 3 group quasi-exp; it would have been lower with selection modeling). • Nesting of units in groups • The intervention is more than just the program, but includes staff, context etc. • We don’t know which component of tmt worked • The constant treatment assumption: Randomized trials do not require constant treatments, especially not effectiveness trials. (Angrist et al. JASA)

7. Questionable Reasons, continued • Randomized trials may or may not be more expensive, more time consuming, or less powerful than other studies, depending on what the other study design is. For example: • Power ITS > RE > QE with selection model? • Can you do ITS with archival data or do you have to gather your own data anew? This will affect both time and expense. • The cost of making an inferential error. • But in general, more money buys you better data no matter what the design is.

8. Using Multiple Design Elements in Quasi-Experiments • Quasi-experiments are not a set of fixed designs • Rather, each quasi-experiment should be an aggregation of many design elements that each are thoughtfully chosen to compensate for the possible threats to causal inference • Moreover, good quasi-experiments use multiple designs to address the same questions (e.g., Jim Dwyer on Agent Orange)

9. An Example: Reynolds and West • Arizona’s “Ask for the Sale” campaign to sell lottery tickets. • Nonequivalent control group (stores agreeing to participate vs not) • Four pretests and four posttests to assess trends and possible regression artifacts • three nonequivalent dependent variables (treatment increased ticket sales but not sales of gas, cigarettes, or grocery items). • Examined some stores in which treatment was removed and repeated, or was initiated later

10. Role of Meta-Analysis • The question of generalizability of effects from RCT is an empirical question, not a fact. • Results from nonrepresentative samples may indeed generalize to representative samples • There are lots of ways to look at generalizability, of which meta-analysis of experiments (randomized or quasi) is one. • Look at large numbers of RCTs and QEs • Examine variability of effects over different kinds of units, persons, settings, and outcomes • Use response surface modeling to predict effects of ideal study (e.g., most clinically representative)

11. Meta-Analysis, continued • Publication bias • We have technologies to study that. • It’s when we don’t do meta-analysis that we have this problem.

12. Summary of Agreements • The question should drive the method • Translational research involves more than just causal questions, and so requires many different kinds of design. • Even when effects are at issues, randomized experiments are sometimes not possible or ethical, and nonrandomized experiments must be considered. • In the hierarchy of nonrandomized designs, time series and regression discontinuity are generally preferred on theoretical grounds, when practical.

13. Summary of Agreements • We have made a lot of progress on addressing problems in both randomized and nonrandomized experiments (things I’ve not heard mentioned here), for example: • Incomplete treatment implementation (Angrist) • Missing data • Selection bias and propensity scores • Triangulating using different designs is a good thing (e.g., time series on archival data with surveys on other dv’s; or RCT on proximal outcomes with observational study on long-term outcome).