1 / 18

Evaluating Research Portfolios I: An Analytical Perspective

Evaluating Research Portfolios I: An Analytical Perspective. Impact Evaluation of Energy R&D Portfolios* Rosalie Ruegg, TIA Consulting, Inc. ruegg@ec.rr.com www.tiaconsultinginc.com American Evaluation Association Conference Washington, DC October 18, 2013

ouida
Download Presentation

Evaluating Research Portfolios I: An Analytical Perspective

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Research Portfolios I: An Analytical Perspective Impact Evaluation of Energy R&D Portfolios* Rosalie Ruegg, TIA Consulting, Inc. ruegg@ec.rr.com www.tiaconsultinginc.com American Evaluation Association Conference Washington, DC October 18, 2013 *Based on work funded by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy

  2. Impact Evaluation of Energy R&D Portfolios • Approach • Background • Why perform R&D evaluation at a portfolio level? • Methodological extensions required for DOE/EERE applications • Current status: (1) DOE/EERE Retrospective Portfolio Impact Studies (2) Revised B-C Guide (3) Aggregation Tool Development Ruegg

  3. R&D Evaluation at a Portfolio Level: Approach Note: “Portfolio” is used here to mean a grouping of investments but not to imply risk analysis or diversification. Ruegg 3

  4. Background • ATP, more than a decade ago, began performing traditional project-level impact studies. • OMB response was that project-level studies were not meaningful because they represented “cherry picking.” • ATP moved to evaluating portfolios of R&D projects to provide conservative estimates of return on larger pieces of its investment. • DOE/EERE extended the portfolio approach by: a) using a more systematic approach according to a Guide, b) broadening coverage of benefits. Ruegg

  5. Advantages of DOE/EERE’s Portfolio Approach • Rigorous evidence of return on EERE’s R&D investment • Systematic treatment allows aggregation across portfolios • Efficient use of the evaluation budget Ruegg

  6. Areas of Emphasis in DOE/EERE’s Evaluation • Characterizing technology content by type (e.g., infratechnologies, platform or generic technologies , product technologies) • Identifying next-best technology alternative • Detailed treatment of additionality • Data collection • Estimation of: - Resource impacts - Environmental impacts - Energy security impacts - Life-cycle effects - Knowledge effects - Combined impacts • Multiple economic performance metrics • Lessons learned Ruegg

  7. Methodological Extensions to Broaden Coverage: Impacts Measured in Dollars Ruegg

  8. Methodological Extensions to Broaden Coverage: Impacts Measured in Physical Units Ruegg

  9. Methodological Extensions to Broaden Coverage: Impacts Measured Qualitatively Ruegg

  10. Methodological Extensions: Overview Ruegg

  11. Economic Performance Metrics Ruegg

  12. Evaluative Questions Addressed • What has been the return on DOE/EERE R&D investment? • To what extent has EERE's investment produced energy, environmental, energy security, knowledge, and economic benefits? • Would today’s commercialized technologies likely have happened at the same time, and with the same scope and scale, without EERE’s efforts? • Was the public investment worth it? • What factors led to return on public investment? • What lessons learned can be applied to future R&D investments? Ruegg

  13. Current Status • Retrospective Impact Guide: 2008, 2011(R), 2013(R) 2008 & 2011 editions prepared by Ruegg & Jordan 2013 edition (in review) prepared by Ruegg, O’Connor, & Loomis • Portfolio Studies Completed: 5 • Data Aggregation Tool under Development Ruegg

  14. Portfolio Aggregation Tool Select from the following:

  15. Case example Ruegg

  16. Example from DOE/EERE: Advanced Combustion Engine R&D Portfolio (Link, 2010) Technology Type Identified: CRF & laser and optical diagnostics tools = Infratechnology “technologies that influence the R&D, production, or marketing of other technologies but are usually not physically embodied in resulting products” Next-Best Alternative Identified: Ruegg

  17. Comparing Actual and Counterfactual BTE (Link, 2010) Ruegg

  18. Analysis Linked MPG to Change in BTE and Translated Results to Fuel Savings and Reduced Air Emissions • Experts estimated 4.5% improvement in Brake Thermal Efficiency (BTE) from DOE-supported R&D • MPG was linked to BTE with an econometric model • Fuel savings were calculated from MPG fuel economy improvements and miles driven. • Resulting energy savings: 17.6 billion gallons of diesel fuel from 1995 to 2007. • (Other effects were also calculated). Ruegg

More Related