1 / 46

National Institutes of Health

Program Performance Monitoring System-Tools for Decision Makers Assessing the Core Criteria to Plan and Report Organizational Performance Goals Deborah Duran, Ph.D. Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives New Frontiers in Evaluation

keona
Download Presentation

National Institutes of Health

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Program Performance Monitoring System-Tools for Decision Makers Assessing the Core Criteria to Plan and Report Organizational Performance Goals Deborah Duran, Ph.D. Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives New Frontiers in Evaluation Vienna, Austria April 2006 National Institutes of Health

  2. Overview of Presentation • NIH Overview • Knowledge Management • Core Criteria • Program Performance Monitoring System (PPMS) • Evaluation Tools • Summary & Future Directions

  3. World’s premier medical research center and steward of medical and behavioral research for the nation Acquire new knowledge to help prevent, detect, diagnose, and treat disease and disability NIH RoleNIH Goal The Nation’s Leading Medical Research Agency

  4. NIH Organizational Structure Clinical Center Center for Information Technology Center for Scientific Review Office of the Director Staff Offices Office of Extramural Research Office of Intramural Research Office of Management/Chief Financial Officer Office of Science Policy Office of Portfolio Analysis & Strategic Initiatives Office of Communications and Public Liaison Office of Equal Opportunity and Diversity Management Office of Program Coordination Office of Legislative Policy and Analysis Office of Community Liaison Executive Office Office of the Ombudsman/Ctr. for Cooperative Resolution NIH Ethics Office Office of the Director Program Offices Office of Research on Women’s Health Office of AIDS Research Office of Behavioral and Social Sciences Research Office of Disease Prevention Office of the Director National Institute on Alcohol Abuse and Alcoholism National Institute of Allergy and Infectious Disease National Institute of Arthritis and Musculoskeletal and Skin Diseases National Eye Institute National Heart Lung and Blood Institute National Human Genome Research Institute National Cancer Institute National Institute on Aging National Institute of Biomedical Imaging and Bioengineering National Institute of Child Health and Human Development National Institute of Deafness and Other Communication Disorders National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute on Drug Abuse National Institute of Environmental Health Sciences National Institute of General Medical Sciences National Institute of Neurological Diseases and Stroke National Institute of Nursing Research National Institute of Mental Health National Library of Medicine John E. Fogarty Center for Advanced Study in the Health Sciences National Center for Complementary and Alternative Medicine National Center for Minority Health and Health Disparities National Center for Research Resources

  5. NIH Overview • 27 Institutes and Centers (ICs) work individually and collectively • Over $27 billion invested annually on medical research • 80% of budget awarded through 50,000 competitive grants to support over 200,000 scientists and research personnel in the U.S. and abroad • 10% of budget supports research led by 1,250 intramural principal investigators

  6. NIH HAS ONE PROGRAM: • Research

  7. Core Criteria for Planning and Reporting Performance Goals • Trans-NIH Approach • Representative Performance Goals (Disease/Disorders/Management) • Annual Budget & Performance Target-Based Approach • Balanced Scientific Research Portfolio (Intramural/Extramural, Basic/Clinical) • Scientific Risk (Low-Medium-High) • Time (Short-Term, Mid-Range, Long-Term)

  8. Trans-NIH Approach Performance Goals Institutes/Centers/Offices • Lead • Contributor(s) • Financial • Scientific

  9. Representative Performance Goals • Representative program sampling across disease, disorders & management • Aligned with • Department of Health and Human Services • Healthy People 2010 • President’s Management Agenda

  10. Annual Budget & Performance Goals Integration • Prospective Annual Targets/Measures • Performance & Strategic Targets • Scientific Adjustments for out years • Monitoring of Annual Targets & Budgets

  11. Balanced Scientific Research Portfolio of Performance Goals • Intramural and/or Extramural • Basic and/or Clinical Science

  12. Scientific Risk and Time Matrix Increasing Level of Difficulty 1-3 years 4-7 years 8-10 years Time

  13. Scientific Research Outcomes (SRO)

  14. NIH mission: to uncover new knowledge that will lead to better health for everyone. National Institutes of Health: Balanced Portfolio Government Performance and Results Act (GPRA) Performance Driver Map DRIVERS FUNCTIONAL AREAS PERFORMANCE Scientific Research Outcomes • Short/Long term • Low/High risk • Intramural/Extramural Generation of new scientific knowledge of new or improved prevention, diagnosis, and treatment of disease • Knowledge Dissemination • Translation • Field input • Data / Repositories Communication and Transfer of Results Shared knowledge for public benefit Capacity Building and Research Resources • Training • Technology • Facilities • Partnerships Highly trained basic and clinical scientists; leadership in technology and facilities Strategic Management of Human Capital • Performance-based approach • Workplace Management • Adequate Workforce Availability of highly skilled and diverse workforce; enhanced work environment Program Oversight and Improvement Enhanced program management and compliance with relevant laws • Results-oriented Management • Resource Management • Stewardship

  15. Performance Reporting Requirements • Government Performance Results Act • (GPRA) • Congressional Justification (CJ) • Program Assessment Rating Tool • (PART)

  16. # ICs Involved: 24 Range of # Collaborators per Goal: 0 – 20 Range of Budget (FY05): $31,000 – $1.5 Billion GPRA Performance Measures

  17. PART Scores

  18. Information ▪ Technology ▪ Knowledge • Age of Information Technology • Age of Knowledge • Capture data • Structure data • Analyze information • Design Application • Collect and Connect

  19. Program Performance Monitoring System (PPMS) “COLLECTS AND CONNECTS” • Utilizes the Systemic Assessments Branch (SAB) webpage as a single sign-on centralized portal to NIH performance monitoring • Centralizes performance monitoring & reporting • Enhances coordination among collaborators • Provides access to current & past performance information & reports • Provides a venue for analyzing performance-related information

  20. Components of PPMS • Systemic Assessments Branch (SAB) Website Useful links to systemic assessments resources and performance-related information at NIH • Visual Performance Suite (VPS) Centralized online database for collecting, storing, analyzing and reporting trans-NIH performance data

  21. Systemic Assessments Branch HomepageHTTP://NIHPERFORMANCE.NIH.GOV

  22. PPMS User Access Security Levels

  23. Visual Performance Suite (VPS) Modules • Visual Planning Create and modify performance goals and targets • Visual Scorecard View goal- and target-level components • Visual Performance Enter, communicate, and track performance data • Visual Strategy Map Organized, hierarchicalview of GPRA plans, performance goals, and targets • Graphical Analysis • Forecasting • Trending • Modeling

  24. VPS Data Entry Screen: Planning

  25. (Continued)

  26. VPS Data Entry Screen: Performance

  27. VPS Strategy Map

  28. Decision-Makers’ Tools for Planning and Reporting Performance Goals • Graphical Analyses-various types of charts & graphs • Summarize performance data • Compare dimensions • Drill into a dimension • Filter information • Trending • Modeling

  29. Summarize DataNumber of Goals by Lead Institute (Core Criterion: Trans-NIH Approach)

  30. Compare DimensionsNo. of Goals compared by Goal Classification across Risk (Core Criterion: Scientific Risk)

  31. Compare Dimensions Same Information Presented Differently

  32. Drill into a DimensionNumber of Goals in each Functional Area(Core Criterion: Representative Performance Goals)

  33. Drill into a DimensionSame information Presented Differently

  34. Filter InformationNumbers of Goals for Top 10 Lead Institutes (Core Criterion: Trans-NIH Approach)

  35. VPS Reports • Customized reports for each module or reporting requirement • Reports can be saved as • MS Word documents • MS Excel spreadsheets • Analytic graphics can be saved as jpeg or gif files & imported into PowerPoint

  36. Performance Goal Narrative

  37. Performance Summary Table

  38. Goals Summary Report

  39. GPRA Goal Development Criteria Criteria for all GPRA Goals: • be representative by fitting appropriately into the overall set of goals that together, signify the NIH mission • be measurable by having quantifiable annual targets that mark against a baseline • be trans-NIH in nature by having a lead IC and scientific and/or financial contributor(s) who work closely with other planning officers, budget officers, & program staff to provide a balanced portfolio • be meaningful to the research community, as well as the public and NIH stakeholders • refer to a metric and/or date for completion, as appropriate • reportable annually • make obvious the budget/performance integration • appear in managers’ performance plans • link to the objectives found in Health People 2010, the FY 2004-2009 HHS Strategic Plan, and the President’s Management Agenda

  40. GPRA Goal Development Criteria (Continued) In addition, Scientific Research Outcomes (SRO) goals should: • be specific to a definable problem or disease area with a focus on outcomes • be recognized as an outcome that could be achieved in the future, but may not be reachable for any number of reasons • be definable as an intramural and/or extramural goal • be classified as a basic or clinical goal • fit within the GPRA goal risk/time matrix to maintain a balanced portfolio • address the OMB R&D Investment Criteria of quality, relevance, and performance http://www.whitehouse.gov/omb/part/2006_part_guidance.pdf

  41. Replacement Goal-Result of Core Criteria Analyses

  42. ForeSee Survey

  43. WebTrends (11/2005 vs 1/2006)

  44. Collect & Connect Summary • Centralized Online System for Reporting • Repository for NIH performance information • Systematic reporting of information • Increased availability, timeliness, accuracy of data • Stakeholders Communication/ Collaboration • Increased transparency for planning • Enhanced coordination • Decreased duplication of efforts • Effective Use of IT to support Evaluation & Portfolio Management of Core Criteria for Performance Goal Planning & Reporting

  45. Future Directions • Develop Methodologies for Aggregation Rules for System Assessments • Assess the Value of Science

More Related