Quality Enhancement Research Initiative

1. Quality Enhancement Research Initiative Documenting and reportingimplementation research:New expectations, new opportunitiesBrian S. Mittman, PhDVA Center for Implementation Practiceand Research Support (CIPRS)January 6, 2009

2. Outline • Background, motivation and goals for improved reporting • Ongoing initiatives to strengthen reporting • Recommendations and future possibilities

3. Research documentation and reporting:Current practices and problems • Increasing focus on shortcomings in health research documentation and reporting • Variable reporting: different measures, design elements, analysis standards • Incomplete reporting: publication bias, selective reporting of outcomes (to favor evidence of efficacy) • Challenges for meta-analyses, replication, cumulative knowledge • Concerns over return on societal (public) investment in research, validity of conclusions (policy/practice implications)

4. Reporting problems in implementation science • Each of these problems exists in implementation research, and many are more severe than in mainstream clinical research • Selective reporting and publication bias • Variable, inconsistent reporting • Emphasis on impact reporting only • Limited “return on investment” and value of research

5. Outline • Background, motivation and goals for improved reporting • Ongoing initiatives to strengthen reporting • Recommendations and future possibilities

6. Major initiatives to strengthen reporting • CONSORT Statement, extensions and similar statements (e.g., SQUIRE guidelines; EQUATOR Network) • ClinicalTrials.gov and other registries • Ottawa statements and protocol publication • External validity reporting • Intervention reporting • Trial bank initiative

7. CONSORT; EQUATOR Network • Standardized format, structure, contents of reports of specific types of studies (experimental, observational, reviews, economic) • Consistent reporting of core features (design, methods, results) • “The EQUATOR Network is an international initiative that seeks to improve the quality of scientific publications by promoting transparent and accurate reporting of health research.” (www.equator-network.org) • Updated SQUIRE guidelines are a recent addition

8. Trial registries • Advance registration of clinical trials in a public registry reduces the likelihood of (un-detected) selective publication • The International Committee of Medical Journal Editors (ICMJE) (and most journals) require a priori trial registration as a condition for publication • Public Law 110-85, Title VIII expands the types of clinical trials that must be registered in ClinicalTrials.gov, increases the number of data elements that must be submitted, and requires submission of results data

9. Protocol publication • The Ottawa Statement on trial registration is a consensus document that aims to guide the implementation of global trial registration (http://ottawagroup.ohri.ca/) • The Statement addressed fundamental principles for trial registration (Part 1) and operationalization of protocol registration (Part 2) and results reporting (Part 3)

10. External validity; intervention details • Several recent articles and editorials promote and guide enhanced reporting of external validity to increase the “relevance, generalizability, and applicability” (i.e., value in policy and practice) of health research (Green, Glasgow) • Other initiatives include ongoing efforts to stimulate more complete, consistent reporting of implementation programs and interventions; contextual factors; etc.

11. The Trial Bank Project • “The prose reporting of trial information could be powerfully augmented by a computable repository of trial information --a global trial bank. Unlike prose, computable information is structured and coded for computation and allows the use of advanced information technologies for knowledge management.” • “The Global Trial Bank seeks to make available open-access and computable peer-reviewed results from all clinical trials conducted worldwide, regardless of whether the results are favorable or not, to provide an unbiased knowledge base for improving human health. “

12. Outline • Background, motivation and goals for improved reporting • Ongoing initiatives to strengthen reporting • Recommendations and future possibilities

13. Comprehensive documentation plan • Systematic, comprehensive documentation plan • Protocol • Baseline/interim results • Main findings • Supporting findings • Secondary analyses • Required elements of each component • Protocol includes current systematic review(s), power calculations for contribution to existing evidence base • Main findings linked to protocol; report analyses for all (vs. selected) outcomes • Supporting analyses explaining findings (mechanisms of action)

14. Rationalizing publication • Distinguish types of evidence/knowledge • Policy/practice relevant (systematic reviews) • Primary evidence (main findings) • Explanatory evidence (process evaluations) • Classify types of evidence, knowledge • Clinical domains • Health services domains • Allocate types of evidence and knowledge to distinct publication outlets (review journals, researcher journals/on-line resources, trial banks, etc.) • Expand registries to improve access to evidence, research output • Link funding programs to registries, trial banks

15. Successful Implementation Manuscripts: Beyond the Basics Jeroan J. Allison, MD, MS Catarina I. Kiefe, PhD, MD Co-Editors-in-Chief, Medical Care January 6, 2008 VA Implementation Research Seminar

16. Conceptual Model for Implementation Research Salanitro A. et al. In Essentials of Clinical Research. New York: Springer, 2008.

17. After carefully studying your manuscript, I am sorry to report that we will not be able to publish it in Medical Care. Medical Care currently receives over 800 original submissions each year, and we eventually accept less than 20% for publication. Because of the increasing competition for journal space, we are not able to assign a high priority to all worthy manuscripts.

18. Current Publication Practices May Distort Science • Problems • Preferential publication of extreme results • Artificial scarcity of publication outlets • Potential Solutions • Increase publication preference for negative studies • Base acceptance on methodological quality • Enhance post-publication review • Dissociate academic advancement from journal branding Young N., et al. PLoS Medicine. 2008; 5(10): e201

19. “Why, sometimes I’ve believed as many as six impossible things before breakfast.” L. Carol, Alice in Wonderland

20. Fundamental Assumption • Interesting question • High-quality data • Appropriate analysis • Crisp presentation • Important conclusions Necessary but not sufficient!

21. Potential Publication Barriers for Implementation Research • Small sample size • Local population • Lack of “strong” study design • Failure to distinguish care improvement from generalizable knowledge • Competition with other types of research

22. Beyond the Basics • Establish dialogue with readership • Frame manuscript to show substance • Demonstrate impact and relevance • Be persistent

23. Dialogue with Readership • Understand the target journal • Establish continuity with readership • Review relevant articles from target journal • Journal self-citation • Inappropriate impact factor padding • World Association of Medical Editors rich discussion www.wame.org

24. Medical Care Vision Statement • Health Care Delivery* • Safe • Effective • Patient-centered • Timely • Efficient • Equitable Health Care Populations Methods *From the Institute of Medicine in Crossing the Quality Chasm: A New Health System for the 21st Century, 2001. A Message from the New Medical Care Co-Editors-in-Chief: Building Upon a Legacy of Scholarship and Impact. Medical Care. 2007; 45(1): 1-2.

25. Substance: Avoid the Least Publishable Unit Plagiarism Exceptions • Same message multiple audiences • Repeated methods from large studies Original Submission Duplicate Publication www.famousplagiarists.com

26. Substance: Avoid the Least Publishable Unit Recently submitted manuscript • Concluded intervention equally effective by race/ethnicity • Reviewer comment: focus on “broader perspective of the impact of the intervention” • MedLine search by editor: request fulfilled in another journal

27. Impact: Advance the Paradigm • Address the “so what” question • Varies from journal to journal • Policy relevance for Medical Care • Understand the cutting edge • Health Disparities: from understanding to change • Risk Adjustment: statistical vs. tangible meaning T. Kuhn. The Structure of Scientific Revolutions. Chicago: University of Chicago Press, 1962.

28. “So-What” Issues for Implementation Research • Linking “basic” human/social sciences with process, structure, and outcomes • Detangling the confusing picture of practice-change interventions • Exploring unexpected consequences of emerging paradigms (e.g., P4P) • Demonstrating value by mixed-methods

29. “So-What” Issues for Implementation Research • Measuring the right outcomes • Testing interventions at interface of patient, doctor, system, and community • Promoting bi-directional information flow along implementation spectrum

30. Impact: Advance the Paradigm • Use bounded speculation • Focus on the core message It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material * Watson, J. D. and Crick, F. H. 1953. Molecular structure of nucleic acids. Nature 171: 737738. Quoted in Sand-Jensen, 2005. Oikos 116: 723-727

31. Why Some Ideas Survive and Others Die • Simple: focus on core message Personal lesson learned repeatedly • Unexpected: unfold with mystery • Credible • ... C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.

32. Why Some Papers are Published and Others Die • Simple: focus on core message • Personal lesson learned repeatedly • Unexpected: unfold with mystery • Credible • ... C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.

33. The Value of Persistence • Rejected manuscript • Poor quality • Poor fit • Bad luck • Peer-review system • Best available • Not Perfect • Notable shortcomings • Moniz won Nobel Prize in 1949 • Respected peer-review publications touted lobotomy for headaches, anxiety, delusions, confusion • 0.20 x 5 = 1 • No. journals in Medline, March 2007: 5,197

34. Take Home Points • Increasing competition for top journal space • More than science and traditional aspects of quality • Establish dialogue with readership • Address “cutting-edge” questions • Frame manuscript to show substance • Demonstrate impact and relevance • Be persistent

35. The real trouble with this world of ours is not that it is an unreasonable world, nor even that it is a reasonable one. The commonest kind of trouble is that is it nearly reasonable, but not quite. G. Chesterton

36. I am pleased to inform you that your work has now been accepted for publication in Medical Care.   All manuscript materials will be forwarded immediately to the production staff for placement in an upcoming issue.