Patient-Centered Outcomes Research Institute

1. Patient-Centered Outcomes Research Institute What is it and what does it mean for health services researchers? Patrick S. Romano, MD MPH UC Davis Center for Healthcare Policy and Research March 28, 2012

2. Speaker’s Verbal Disclosure Statement: Have you (or your spouse/partner) had a personal financial relationship in the last 12 months with the manufacturer of the products or services that will be discussed in this CME activity? _x_ Yes ___ No (If yes, please state disclosures and resolutions) I received a $200 honorarium and reimbursement of travel expenses for participating in a PCORI grant review meeting.

3. Educational objectives for this seminar: Please type between two and four educational objectives here. To describe the origin and mission of PCORI To differentiate Patient Centered Outcomes Research from Comparative Effectiveness Research (if possible?) To explain the PCORI grant review process and discuss future opportunities for UC Davis/CHPR research support

4. What is CER? • Comparative effectiveness research is the conduct and synthesis of systematic research comparing different interventions and strategies to prevent, diagnose, treat and monitor health conditions. The purpose of this research is to inform patients, providers, and decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances. To provide this information, comparative effectiveness research must assess a comprehensive array of health-related outcomes for diverse patient populations… • Federal Coordinating Council for Comparative Effectiveness Research, definition; [cited 28 July 2010].

5. What is PCORI? • Prior to several years ago, no coordinated effort for comparative effectiveness research (CER) • 2009 stimulus package provided $1.1 billion for CER (most went to AHRQ and NIH) • 2010 Affordable Care Act created Patient Centered Outcomes Research Institute • Funded by tax on government and private insurers for each covered life (~$500 million per year)

6. What is PCORI? • Independent, nonprofit, quasi-governmental organization • AHRQ almost eliminated in 1990s after published guideline that surgery for back pain was unnecessary • Provide political protection • PCORI funding does not depend on annual Congressional appropriation • No infrastructure for determining funding priorities, issuing opportunities, reviewing and administering grants • Outsourcing (NIH, AHRQ, CMS) • Peer review, contract management • May receive $ from PCORI to originate grants • PCORI will be its own entity

7. 1. Identifying national priorities for research.2. Creating a research agenda based on identified priorities.3. Funding research consistent with these priorities and agenda.4. Providing patients and their caregivers with useful research information.

8. PCORI Board of Directors • Director of AHRQ (Carolyn Clancy) • Director of NIH (Francis Collins) • 17 members appointed by the Comptroller General of the United States as follows: • 3 members representing patients and health care consumers • 5 members representing physicians and providers, including at least 1 surgeon, nurse, State-licensed integrative practitioner, and hospital • 3 members representing private payers, including at least 1 health insurance issuer and at least 1 self-insured employer • 3 members representing pharmaceutical, device, and diagnostic manufacturers or developers • 1 member representing quality improvement or independent health service researchers (Harlan Krumholz or Arnold Epstein) • 2 members representing the Federal Government or the States, including at least 1 representing a Federal health program or agency

9. PCORI Leadership • Eugene Washington, MD, MSc, (chair), Vice Chancellor of UCLA Health Sciences and Dean of the David Geffen School of Medicine • Steven Lipstein, MHA (vice chair), President and Chief Executive Officer of BJC HealthCare • Joe Selby, MD, MPHExecutive Director • Anne C. Beal, MD, MPHChief Operating Officer, abeal@pcori.org

10. PCORI Mission PCORI helps people make informed health care decisions and improves health care delivery and outcomes by producing and promoting high integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader health care community.

11. PCOR vs. CER • Why PCORI, not CERI? • Political concerns that determining which treatments better → “rationing” • Restrict patients’ ability to choose inferior treatment • Compromise focus on patient-centered outcomes research • Legislation barred PCORI from • Developing or using QALYs • Considering cost in making coverage recommendations • In practice, PCORI will not fund cost-effectiveness analyses

12. Patient-Centered Outcomes Research • Not just a semantic difference • Focus on PCOR essential role in shaping funding priorities and “rules of the game” for getting funding • Can’t just replace placebo control with active comparison group, do everything else the same • Distinct elements • Outcomes should be patient-centered • Effectiveness, not efficacy • Heterogeneity of treatment effects • Involvement of stakeholders

13. Patient-Centered Outcomes Research • Outcomes should be patient-centered • “…outcomes that people notice and care about such as survival, function, symptoms, and health-related quality of life” • Not biomarkers (BP, serum cholesterol, etc.)

14. Patient-Centered Outcomes Research • Effectiveness • Research done in clinically relevant populations • Representative of patients • Representative of settings • Heterogeneity of treatment effects • “Given my personal characteristics, conditions and preferences, what should I expect will happen to me?” • “What are my options and what are the benefits and harms of those options?”

15. Role of stakeholders • Government payors and regulators; health plans; hospitals and other care providers; industry • Consider how results will shape policy and practice through dissemination • Stakeholders should be intimately involved in the study design process to identify important outcomes • Trials of anti-seizure drugs in children • Traditional measure – seizure frequency • Patient-centered measure – parent want to know how drug will affect school performance • Drug ↓ seizures but ↑ sedation may not be preferable

16. Implications for research design • RCTs • Pragmatic - real-world settings; large N’s to assess subgroups • Focus on effect size, not just superiority • Bayesian and adaptive approaches to design, accounting for prior probabilities • Observational studies • Causal inference modeling • Tension for studies using pure administrative data – powerful and low-cost; lack high-quality patient-centric data • Cost and cost-effectiveness • PCORI not allowed to evaluate cost in making coverage recommendations, won’t fund “pure” CEA • Cost can be studied and used by decision-makers

17. Proposed National Priorities • Assessment of Prevention, Diagnosis, and Treatment Options. The research goal is to determine which option(s) work best for distinct populations with specific health problems. • Improving Healthcare Systems. Focuses on ways to improve healthcare services, such as the coordination of care for patients with multiple chronic conditions. • Communication and Dissemination. Looks at ways to provide information to patients so that they, in turn, can make informed healthcare decisions with clinicians. • Addressing Disparities. Assures that research addresses the healthcare needs of all patient populations. This is needed as treatments may not work equally well for everyone. • Accelerating Patient-Centered and Methodological Research. Includes patients and caregivers in the design of research that is quick, safe, and efficient.

19. PCORI Funding Opportunities • Developmental • RFPs (Sept 2011) • ~$30-$100K each • Methods for setting research priorities • Pilot Projects Grant Program (Dec 2011) • $13 million in year 1 for ~40 projects • Help develop national research priorities for PCOR • Develop ways to engage patients and bring stakeholders together • Develop measurement tools and data collection methods for novel patient-centered outcomes

20. PCORI Funding Opportunities • $150 million in current FY; ultimately perhaps $500 million/year • Mechanisms unknown, priorities under discussion • Likely NOT “traditional” research • Patient-centered outcomes • Including developing methods • Real-world populations • Heterogeneity of treatment effects • Involvement of stakeholders

21. Other Funding Opportunities • Existence of PCORI does not preclude government agencies or foundations from funding CER • AHRQ has Effective Healthcare Program • Large evidence-synthesis and dissemination projects • EBPCs, DEcIDE network, CERTS • Does not support investigator-initiated projects • Areas of CER explicitly outside PCORI’s mandate • CER with “traditional” outcomes • Cost-effectiveness analyses • Remains to be seen

22. PCORI Pilot Research Projects • $13 million in 2012 • Approximately 40 awards • 2 years maximum support • 2nd year of funding must be well justified and will be evaluated for successful completion of the first year goals • $250,000 per year (plus indirect costs) • Travel to annual PCORI meeting is an allowable expense • Requires stakeholder involvement as co-investigators • unless not feasible as for an application on analytic methods

23. Principles of grant review • Scientific and technical merit • Fair, competent and thorough • Conflict of interest • Confidentiality • Difference between overall impact and significance • Overall impact: Probability of whether the research will exert a sustained, powerful influence on the research field. • Significance: Does the project address an important problem or a critical barrier to progress in the field? If the aims are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?

24. Structure of application • Signed Face Page, Table of Contents • Project Summary: single paragraph using plain language to summarize the • Broad objectives • Specific aims • Relevance to PCORI Objectives • Research Design and methods for achieving the goals • Project/Performance Site(s) • Key Personnel and Other Significant Contributors • Biographical Sketches–Principal Investigator, Other Biographical Sketches • Personal statement allows each person to explain their role and goals for the project • Budget and Justification • Resources • Provide information on the facilities to be used in the proposed research project • Describe how the research environment contributes to the probability of success • Describe how project will benefit unique features of the research environment or community involvement or allow for useful collaborative arrangements • Describe institutional and community investment in the success of the research • Checklist • Addendum Form: Checklist to identify PCORI Areas of Interest

25. Research Plan • Specific Aims (1 page) • Research Strategy (10 pages) • References Cited • Protection of Human Subjects • Consortium/Contractual • Letters of Support • No appendices are allowed and will be removed by PCORI before sending to reviewers

26. PCORI Review Criteria • Significance (Research Plan) • Patient/Stakeholder Engagement (Research Plan) • Investigator (Biosketch) • Innovation (Research Plan) • Approach (Research Plan) • Environment (Project/ Performance Sites)

27. PCORI Review Criteria -Significance • Does the project address an important problem or critical barrier to PCOR? • Will the result produce new knowledge that can advance PCOR methods or infrastructure? • Does the investigator demonstrate thorough knowledge of previous and ongoing work related to their proposed topic?

28. PCORI Review Criteria -Patient/Stakeholder Engagement • Will the project make a unique contribution to learning about engagement of patients and stakeholders? • Does the research team demonstrate authentic, feasible partnerships with the patients/stakeholders? • Is there evidence that the stakeholders were involved in the preparation of the research proposal?

29. PCORI Review Criteria -Investigator • Is the research team well suited to the project? • Is there appropriate scientific expertise? • Does at least one member of research team have experience in patient/stakeholder engagement? • Will the research team be able to achieve the study aims? • Does the research team have complementary and integrated expertise? Is the approach and structure of the research team appropriate for the project?

30. PCORI Review Criteria -Innovation • Does the project address a new method or approach for the field of PCOR? • Does the project apply a proven method or approach in a novel way to the field of PCOR? • Is there a new application or refinement of theoretical concepts, approaches, methodologies, instrumentation or interventions proposed?

31. PCORI Review Criteria -Approach • Are the overall strategy, methodology, and analyses appropriate to accomplish the specific aims? • Are potential problems, alternative strategies, and benchmarks for success presented? • Will the strategy establish feasibility and will particular risky aspects be managed • Is the budget and timeframe appropriate for the research plan? • Are there plans for dissemination among key PCOR stakeholders in education, practice and policy?

32. PCORI Review Criteria -Environment • Will the environment contribute to the probability of success? • Are the institutional support, equipment and other physical resources adequate for the project? • Will the project benefit from unique features of the research environment, community involvement, patient populations, or non-traditional stakeholder collaborative arrangements?

33. PCORI Review Template Based Overall Impact Section Scored Review Criteria Sections: The six review criteria, Significance, Patient/Stakeholder Engagement, Investigators, Innovation, Approach, Environment, with a separate score for the first five criteria.

34. When Writing Critiques Overall Impact paragraph – • requires a summary narrative (paragraph) of the strengths and weaknesses. Do not cut and paste. • Focus on major strengths and weaknesses that impacted your overall rating of the application • Overall impact score is not an average of the criterion scores; weights can differ across reviewers • Use bulleted points to make succinct, focused comments (Be careful of being too brief – the reader needs to understand your comment) • Any criterion score between 3 to 9 needs to have weaknesses listed 34

35. Scoring 9-point score scale is used to provide: • Criterion Scores for each of the core review criteria • Overall Impact/Priority Score based on but not an average of the core criterion scores plus additional criteria All applications receive scores: • Not discussed applications (bottom half by pre-meeting scores) will receive only initial criterion scores from the 3 assigned reviewers. • Discussed applications also receive an averaged overall impact score from eligible panel members (i.e., those without conflicts of interest). • These scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.

36. 9-Point Score Scale Descriptors

37. Process During the Meeting • Chair calls for preliminary overall impact scores • Primary reviewer • Brief description • Discuss major/ score-driving strengths and weaknesses • Second and third reviewers add only major/score-driving strengths and weaknesses • Open discussion among all reviewers • Human subjects issues • Final scores from assigned reviewers • Unassigned reviewers may vote outside the score range • Record scores (in IAR and on hard copy) • Budget (does not influence score!) • 10-15 minutes per application

38. Reviewer Voter Sheet (available only during the meeting)

39. Acknowledgments • Michael Steinman MD, UCSF CTSI Director of Comparative Effectiveness Research • Jonathan Arias, PhD, NIH Center for Scientific Review • All of you