Sequential Experimentation: A Different Approach to Prevention Research

Sequential Experimentation: A Different Approach to Prevention Research Annual Meeting of the Society for Prevention Research Washington, DC June, 2003

Linda M. Collins The Methodology Center and Department of Human Development & Family Studies Penn State Susan A. Murphy Department of Statistics and Institute for Social Research University of Michigan Vijay Nair Departments of Statistics and Industrial & Operations Engineering University of Michigan

Overview • Why we are proposing a change in how we conduct prevention research • Introduction to sequential experimentation • Advantages we see to sequential experimentation • Considerations and open questions

Two broad scientific issues facing prevention science

How can we improve our understanding of the mechanisms underlying prevention? • In early days of prevention, the field struggled to establish THAT prevention worked • Now, the time is right to begin understanding HOW, WHY and FOR WHOM prevention works • This will allow us to fine-tune, even optimize, interventions

How can we hasten the building of a coherent body of knowledge about prevention? • The field has moved forward in important ways – but slowly • Solid, replicable results are mixed with unreplicable and contradictory findings • We need to speed up the accumulation of consistent, replicable findings

We argue that a new approach is needed in order to address these issues. But what is the difficulty with the current approach?

Two prevention program evaluation research agendas • Answering the bottom-line question of whether a program is effective • Informing theory and design of prevention programs

Two prevention program evaluation research agendas, one approach • We try to address both agendas with large, expensive prevention trials involving • lots of subjects • power for detecting treatment effects • lots of variables • post hoc analyses with the hope of informing theory and figuring out how the program worked (or why it didn’t work)

Two prevention research program evaluation research agendas, one approach • This approach tries to serve both agendas simultaneously, but is it the BEST way to serve either one?

How can we best inform theory? • Post-hoc or secondary analyses can be very misleading • e.g. post-hoc “dosage received” analyses • Best way to inform theory: series of randomized experiments

How can we best answer questions about program effectiveness? • Even sound post-hoc analyses cannot help improve the prevention trial that generated the data • Best way to maximize program effectiveness: obtain data to inform the prevention trial BEFORE it is conducted

We propose a different approach: sequential experimentation

A different approach: Sequential experimentation • Emerged from agriculture and engineering • Philosophy: • scientific research is an iterative process of deduction and induction • constant interplay among theory, experimentation, data analysis

A different approach: Sequential experimentation • In engineering • theory suggests many possible treatment combinations that would require a prototype • not practical to build ALL prototypes – each one is resource-intensive

A different approach: Sequential experimentation • In prevention • Theory suggests many candidate components and combinations of components that could make up a preventive intervention • Not practical to bring EVERY possible combination of components to a prevention trial – each trial is resource-intensive

The objective of sequential experimentation • Sort through the space of all possible treatment combinations and dosages in an efficient way, to select programs and dosages most likely to be successful

Sequential experimentation: A sequence of three phases • Screening of candidate program components • Refining of program components • Confirming the effectiveness of a refined program

Screening phase • Purpose: to sort through candidate components to decide which are effective and should be investigated further • Starting point: candidate intervention components suggested by theory • e.g. normative education, resistance training, parental component • Examine each via a series of randomized controlled experiments

Screening phase • Components identified as effective in the screening phase are tentatively selected for inclusion in the intervention. • Next, go to refining phase.

Refining phase • Purpose: identify interactions, investigate blocking variables and putative moderators, refine dosage • e.g.: • examine whether normative education and parental component interact • examine effects of gender • determine optimal number of resistance training lessons • Starting point: components identified as effective in the screening phase

Refining phase • Series of randomized experiments on components that passed the screening phase • Depending on results, may go back to screening, or on to confirming phase

Confirming phase • Purpose: construct and test preventive intervention • Starting point: active program components, appropriate dosages, and important moderators identified during screening and refining phases

Confirming phase • Develop a refined intervention, consisting of only active, effective components, delivered in appropriate amounts • A full randomized prevention trial testing the refined intervention

A few key points about sequential experimentation • Theory plays a critically important role, particularly in the screening and refining phases • Selection of candidate components • Informing the designs used

A few key points about sequential experimentation • Extensive use made of designs such as balanced fractional factorials and response surfaces • These designs use theory to inform choices about where to concentrate resources • Much more efficient than full factorial designs • Number of cells and sample size requirements can be greatly reduced, WHILE MAINTAINING STATISTICAL POWER

A few key points about sequential experimentation • The screening and refining phases are NOT pilot studies • They are randomized, controlled studies

A few key points about sequential experimentation • Type II errors (overlooking active prevention component) may be worse here than Type I errors (mistakenly selecting inactive component) • Type I error rate greater than usual p<.05 may be used

Advantages we see to the sequential experimentation approach • Screening, refining phases address “how, why, and for whom” questions

Advantages we see to the sequential experimentation approach • Results of screening and refining phases will be replicable and will lead to a coherent body of knowledge

Advantages we see to the sequential experimentation approach • Programs that go to prevention trials have been refined; the idea is to produce larger effect sizes • increased statistical power • increased public health benefits

Advantages we see to the sequential experimentation approach • It may allow for more creative risk-taking • it is possible to try a brand-new approach in the screening phase without the investment of an entire prevention trial

Advantages we see to the sequential experimentation approach • Less likely to run into unanticipated problems • Thus less likely to be presented with the familiar dilemma: • Midway through the prevention trial we realize the program needs revision, but we don’t want to change the program and thereby make the evaluation difficult

Considerations and open questions • In most cases, the screening and refining phases can occur in a five-year funding cycle • The confirming phase can occur in a second five-year funding cycle • The support of NIH program staff will be needed

Considerations and open questions • Hierarchical data • we are working on this • Steady stream of research subjects needed • series of experiments may take place over several years • possibility of cohort effects

Considerations and open questions • You can think of many others!

Sequential Experimentation: A Different Approach to Prevention Research