Study Eligibility Criteria

1. Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov

2. Systematic Review Process Overview

3. Learning Objectives • To understand the role of study eligibility criteria (inclusion/exclusion) in framing a systematic review • To know when and how to set study eligibility criteria • To understand the effect of study eligibility criteria on interpretation of a review

4. Study Eligibility Criteria • Determine which studies will be included in the analysis • Function the same in systematic reviews as in primary research • Should reflect the analytic framework and key questions • Are powerful tools for widening or narrowing the scope of a review • Provide information to determine whether reviews can be compared or combined

5. Some Sample Criteria • Population type (adult, community-dwelling females) • Intervention (screening tool to identify depression risk) • Country (United States only) • Setting (hospital-based studies only) • Study population size (N > 200) • Study design (randomized controlled trials)

6. Using Broad Criteria • Can be as broad as “all related studies” • Can be helpful for exploring “what is known” • May result in too much literature to feasibly review • Could pull in disparate literature that cannot be compared

7. Using Narrow Criteria • Can be helpful in culling homogenous literature • Can reduce size of the literature to a manageable scope • Could be appropriate for a narrow research question • May reduce applicability of the review and sometimes risks introducing bias

8. Refining Criteria • Choosing how broad or how narrow to set criteria requires a balance of obtaining adequate information to answer a given question without obscuring the results with irrelevant literature. • Review teams should work together to find this balance. • The overarching goal is to minimize bias related to which studies are selected.

9. Bias in This Context • Distortion of the estimate of effect that comes from how studies are selected for inclusion • Affects the applicability, or “external validity,” of the review itself

10. Examples of Bias in This Context • Inappropriate eligibilty criteria may limit applicability of the review. • Example: the use of studies of twin pregnancies in a review of preterm labor management for low-risk women • Eligibility criteria may result in the inclusion of more of a certain study type that either overestimates or underestimates effectiveness.

11. Selecting Criteria • Review study goals • Assess analytic framework and key questions • Set criteria before beginning abstract review

12. Tie Criteria to PICOTS • Population — condition, disease severity and stage, comorbidities, patient demographics • Intervention — dosage, frequency, method of administration • Comparator — placebo, usual care, or active control • Outcome — health outcomes, morbidity, mortality, quality of life • Timing — Duration of followup • Setting — Primary, specialty, inpatient, cointerventions

13. Other Considerations for Setting Criteria • What study designs should be included? • Include studies in foreign languages? • Include studies conducted in other countries but published in English? • Include “grey” or “fugitive” literature? • Include year of publication?

14. Types of Studies To Be Included • Limit to randomized controlled trials? • Include observational studies? • If so, what specific types? • What is the value of a case series? • How is the definition of case series operationalized?

15. Observational Studies (I) • There are four main types of observational studies: • Cohorts (with comparisons) • Case controls • Case series • Registries/databases

16. Observational Studies (II) • Well-conducted observational studies can address issues of applicability and the need for longer-term outcomes if they: • include more representative patient populations, • have relevant comparators, and • report more meaningful clinical outcomes over longer time frames. • Observational studies may be a better source of information about harms.

17. Non-English–Language Study Reports • Positive findings may be more likely to be published in high-profile English-language journals. • Therefore, to include only English-language journals may overestimate the positive effect of an intervention. • Empirically, the bias associated with limiting a review to English-language reports has been shown to be small. Gregoire G, et al. J Clin Epidemiol 1995;48:159-63; Moher D, et al. J Clin Epidemiol 2000;53:964-72.

18. Grey or “Fugitive “Literature • Material that may be difficult to identify and retrieve: • Government reports and regulatory sites (Center for Drug Evaluation and Research/U.S. Food and Drug Administration) • Pharmacoepidemiologic databases, including postmarketing surveillance • Book chapters • Conference proceedings • Published dissertations • Scientific information packets (Agency for Healthcare Research and Quality) • Rationale for a search: • Because journals may publish positive or statistically significant results, finding grey literature of unpublished nonsignificant or null results may indicate the presence of publication bias.

19. Year of Publication • Literature searches typically have date parameters (i.e., earliest or latest date of publication). • Date parameters are particularly important when there has been a change in policy, practice, or formulation that makes older studies less applicable.

20. Using the Criteria • The rest of this module contains a number of questions and some sample criteria. • Using your newly acquired knowledge of selection criteria and the guidelines described in the previous slides, work through the thought exercises and examples that follow.

21. Exercise 1 What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorder who are entering adulthood? • Before seeing the key questions, consider the categories of criteria that you will want to apply.

22. Exercise 1: Basic PICOTS Questions • What is the relevant population? • What is the intervention of interest? • To what exposure is the intervention being compared? • What outcomes are relevant? • What short-term or long-term outcomes should be considered? • In what setting would the results be applicable? PICOTS = population, intervention, comparators, outcomes, timing, and setting

23. Exercise 1: PICOTS

24. Exercise 1: What Would You Do With . . . • A study that included “individuals over 10 years of age”? • A paper about an intervention for individuals with a range of developmental disabilities, not only autism spectrum disorder (ASD)? • A study of children with Asperger’s syndrome but not other forms of ASD?

25. Exercise 1: No “Right” Answers • There are no “right” answers to these questions. • The team should consider how selection criteria may bias the review or affect applicability. • The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review.

26. Exercise 2: Selecting Criteria for a Narrow Research Question What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth?

27. Exercise 2: Implications of a Question That Requires Narrow Criteria • Efficacy: randomized controlled trials only • Low risk: no previous preterm birth • Yield: 11 studies

28. Exercise 3:Selecting Criteria for a Broad Research Question What is the best management strategy for overactive bladder (OAB) among women? • consideration: • OAB is a fairly difficult condition to define. • The study of OAB is often combined with other types of urological conditions, such as stress incontinence or prostate issues, when studied. • OAB is treated pharmacologically, behaviorally, and surgically and with complementary and alternative interventions — each area includes very different types of studies.

29. Exercise 3: Impact of Study-Size Criteria on Excluded Studies • The research team required a minimum sample size N ≥ 50 in each study at study start. • Implications: • Total excluded for N < 50: 79 • Excluded for N < 20: 36 • Excluded for N 20 to 29: 23 • Excluded for N 30 to 39: 8 • Excluded for N 40 to 49: 12

30. Exercise 3:Impact of Gender on Study Selection Criteria • Studies had to include at least 75 percent women. • This decision was based on expert opinion and the size and scope of the literature. • Based on this criterion, 40 studies were excluded. • Another 27 studies would have been excluded had the review been limited to studies of only women.

31. Example 1: A Maternal-Fetal Surgery Technical Brief • Case series with N ≥ 2 were included. • Only 3 of 169 studies were randomized controlled trials; 122 were case series. • Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups.

32. Example 2:Systematic Review of Cesarean Delivery • A systematic review on outcomes of cesarean delivery on maternal request. • Conducted for the National Institutes of Health-Office of Medical Applications of Research (NIH-OMAR) State-of-the-Science Conference

33. Example 2: The Challenge in the Systematic Review on Cesarean Delivery • No evidence on outcomes of cesarean delivery on maternal request (CDMR) versus other modes of delivery • Urgent need for actionable evidence • Need to recognize and account for confounders 

34. Example 2: The Solution in the Systematic Review on Cesarean Delivery • Expand search to include proxies • Weight rungs of evidence to account for confounding • Highest rung: trials of breech delivery, but only for maternal outcomes • Lower rung: planned cesarean versus planned vaginal delivery • Lowest rung: comparisons of maternal and neonatal outcomes from actual modes of delivery

35. Key Messages • Eligibility criteria: • Specify the scope of the literature to be included • Should be tied to the analytic framework, key questions, and PICOTS • Can reduce bias and support applicability, but run the risk of introducing bias if not chosen appropriately. • Must be applied consistently among individuals on the review team • Need to be recorded throughout the project and in the report for transparency and replicability PICOTS = population, intervention, comparator, outcome, timing, and setting

36. References • Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91. • Gregoire G, Derderain F, Le L. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63. • McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228-31. • McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology. New York: Oxford University Press; 2006. p. 99-124. • Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000; 53:964-72.

37. Author • This presentation was prepared by Melissa L. McPheeters, Ph.D., M.P.H., a member of the Vanderbilt University Evidence-based Practice Center. • The information in this module is currently not included in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_principles1.pdf).