Towards Evidence-Based Discovery Informatics Tools for Synthesis Guest Speaker : Tim Cary

1. Towards Evidence-Based DiscoveryInformatics Tools for Synthesis Guest Speaker : Tim Cary Catherine Blake School of Information and Library Science University of North Carolina at Chapel Hill http://www.ils.unc.edu/~cablake cablake@email.unc.edu

2. Systematic Review Process • Formulate the problem • Locate and select studies • Assess quality of studies • Collect data • Analyze and present results • Interpret results • Improve and update review 28 months from initial idea to publication Increased demand due to evidence-based medicine

3. Guesswork guided by scientifically trained intuition Rescher (1978) Manual Synthesis Select Verify Extract Analyze

4. Cochrane - RevMan • Review Manager (RevMan) is the software used for preparing and maintaining Cochrane reviews. • You can use RevMan for protocols and full reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically. • Source: http://www.cc-ims.net/RevMan

5. Cochrane - GRADEpro • GRADEpro (GRADEprofiler) is the software used to create Summary of Findings (SoF) tables in Cochrane systematic reviews. It can retrieve data of the systematic review and meta-analyses from a Review Manager 5 file, combine these data with user-entered data, and then export a Summary of Findings table ready for import into Review Manager 5. It performs many of the calculations necessary to present the key results of systematic reviews in a table format and guides users through the process of grading the quality of the evidence using the GRADE approach. • Source: http://www.cc-ims.net/gradepro

6. Reporting Guidelines • CONSORT - reporting of RCTs • PRISMA (formerly QUOROM) [PDF document] - preferred reporting items for systematic reviews and meta-analyses • STROBE - reporting of observational studies in epidemiology • EQUATOR Network - collection of reporting guidelines • Source: http://www.cochrane.org/index_authors_researchers.htm

7. Selection Step • Typical information retrieval framing • Input: MEDLINE • Output: Articles included in previous studies • Goal: identify weighting schemes that identify only articles included in a traditional analysis • Examples • Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing Workload in Systematic Review Preparation Using Automated Citation Classification. JAMIA 2006;13(2):206-219. • Demner-Fushman D, Seckman C, Fisher C, Hauser S, Clayton J, Thoma G. Prototype System To Support Evidence-based Practice. AMIA AnnuSymp Proc. November 2008:151-5.

8. Context Information • Study Information • e.g. date, location, ... • Population Information • e.g. gender, age, ... • Risk Factor or Intervention • e.g. duration of exposure, confounders • Disease • e.g. stage, confounders Loosely coupled to review focus Tightly coupled to review focus

9. Collaborative Information Synthesis

10. Key: Estimate Missing Information 2 1 What are people with Breast Cancer exposed to? What are people in a similar population exposed to? • Facts for each study • number of patients • age of patients • geographic location • risk-factor exposure … • Codebook • question asked • age, gender • % responses Database of risk factors BRFSS Studies with Breast Cancer patients 3 Are these rates significantly different? T. Tengs & N. D. Osgood (2001) “The link between smoking and Impotence: Two Decades of Evidence”, Preventive Medicine, 32:447-52

11. Information Synthesis Information Synthesis More than Automated Meta-Analysis • Traditional analysis • same study design • medicine = RCT • epidemiology = cohort • Information Synthesis • any study that includes required information • augment missing information Systematic Review Key Main topic Entire study Secondary Information External database

12. Natural Language Processing Human-assisted Discovery and Synthesis Natural Language Processing Core Genomics Education Discovery Science Evidence-based Practice News Human Discovery and Synthesis Chemistry DocSouth Breast Cancer Synthesis and Discovery Work Practices Heterogeneous Literature

13. METIS Information Extractor • Semantic Grammar • Features: words, numbers, and semantic types in the Unified Medical Language System (UMLS) • Information extracted : • risk factor exposure (tobacco and alcohol )  gender • age (min, max, mean)  start and end dates • number of subjects with medical condition  geographical location {term;’age’} {term:’of’} {number;10<n2<110}{term;’to’}{number;10<n2<110} The age of breast cancer subjects ranged between 20 to 64 years old. {semantic type: neoplastic process, or disease}

14. METIS Info Extractor – Evaluation • Diverse text corpus • epidemiology, surgery, biology, ... • cohort studies, case-control trials, ... • Evaluation • Metrics (precision, recall) • Annotators (developer, domain expert, expert annotator, novice) • Primary topic (breast cancer, impotence) • Secondary information (tobacco and alcohol consumption)

15. METIS Info Extractor – Recall

16. METIS Info Extractor – Precision

17. METIS Verifier Converted Article Electronic version of article Verify information extracted

18. METIS Verifier

19. METIS Analyzer • Meta-Analysis • Developed for agricultural application • Requires empirical studies with a quantitative outcome • Unit of study is an article - not a person • Result – a unitless metric called an effect size • Two common meta-analysis techniques • Fixed effects • Randomized-effects model Evaluation: Compared generated effect size with examples in text books and published articles , Result: Same effect size

20. Alcohol Consumption Synthetic Estimate Evaluation Tobacco Consumption