Brian Zuckerman, Ph.D. Alexis Wilson Science and Technology Policy Institute/IDA November 2, 2006

1. Evaluating Federal Research and Development Programs With Quantitative Measures: An “Outside-In” Look at Translational Research at NIH Brian Zuckerman, Ph.D. Alexis Wilson Science and Technology Policy Institute/IDA November 2, 2006

2. STPI supports the Office of Science and Technology Policy and other executive branch science agencies Current evaluation tasks for the NIH Office of the Director, plus three Institutes/Centers And generally for program staff or evaluation/policy offices We’re not directly involved with budget/performance integration Any opinions expressed stem from our analyses and insights, and do not reflect the opinion of NIH (or OSTP). Most of the insights from STPI work with NCI Translational Research Working Group (TRWG). STPI is a Federally Funded Research and Development Center operated by the Institute for Defense Analyses “Outside In” – STPI as External Relative to NIH Decision-Making

3. Overview • Introduction and Motivation • Desirable Metrics for Translational Research • Implications for the Future

4. Introduction: NIH relative to session framework • Should anything be done differently to evaluate basic vs. applied vs. implementation of research knowledge? • NIH engages heavily in both basic and “translational” research • “Science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability.” (http://www.nih.gov/about/index.html#mission.htm) • Our evaluations of basic research at NIH quite different from translational research • Does the customer of the R&D outcomes make a difference? • Our evaluations are directed at evaluation officers/program staff as immediate customer, with senior management/budget staff/public as indirect customer • Does an organization’s characteristics (size, structure,..) impact a quantitative evaluation? • NIH is ~$30B (US) organization – large relative to USDA, NASA • Each Institute/Center has somewhat different [evaluation] culture • Institutes/Centers collect organization-specific data

5. Lab New Tools & New Applications Population “Translational” Research Mix of “Applied” and “Development” Research • Research that transforms scientific discoveries arising in the lab, clinic or population into new clinical tools & applications that reduce [cancer] incidence, morbidity & mortality Clinic Definition: National Cancer Institute Translational Research Working Group

6. Overview • Introduction and Motivation • Desirable Metrics for Translational Research • Implications for the Future

7. Some Measures of Evaluating Government-Funded Translational Research that Intuitively Appear Quantitatively Tractable • Flow of innovations from research through development to implementation (success) • Or from publicly-funded researchers to private industry • Efficiency/cost-effectiveness of system • Uncertainty reduction/options creation

8. Measuring “Success” and “External Integration” Harder than it Looks Discovery Pre-clinical Clinical Trials Location on translational research continuum

9. Tracing “Successes” Through Programs Proved Difficult… • Some programs have complete information for one stage of the pipeline – but no links forward or backward • Others have complete information across the pipeline – but for a single point in time Source: http://dtp.nci.nih.gov/docs/raid/timeline.html Source: Early Detection Research Network 2005 Annual Report

10. Case Study Approach As Part of TRWG Process Identified Specific Lessons • Lessons learned included: • Translation occurs via diverse mechanisms • Often combinations between multiple programs/awards • Translation occurs via diverse stakeholder Interactions • Not just academics handing off to industry!

11. TRWG Draft Pathways to Clinical Goals as Key Output of Ongoing Process • NCI developed five draft “pathways” describing flow of innovations from discovery through clinical trials (http://www.cancer.gov/aboutnci/trwg/Pathways-to-Clinical-Goals) • Pathways as descriptor of flow of innovations through system • TRL level ~3 to ~6-8 • Potentially allows for innovation/project/program tracking for evaluation purposes

12. 2. Measuring “Efficiency” • Pathways to clinical goals describe translational research process as series of decision steps • At each stage, go/no-go/refinement decisions are made • Goal of translational research system to maximize flow of clinically-beneficial discoveries • Eliminating “losers” as early as possible desirable • Suggests value of information approach to assessing technologies and investment opportunities • Given the cost, sensitivity, and specificity of substitutable test/diagnostic/screening methods – which one should be chosen? • Where might new/improved tools provide most value and therefore greatest investment need?

13. Value of a test/model/trial as change in expected value of decision process as a result of applying the test Example: Oncotype Dx (Hornberger et al, American Journal of Managed Care, 2005) New diagnostic’s influence on cost of breast cancer treatment, survival evaluated (versus existing diagnostics) New diagnostic found likely to both improve outcomes and cut systemwide costs Value-of-Information Approach Is Used to Evaluate Individual Diagnostics…

14. And in Optimizing Design of Clinical Trials… • How large should clinical trials be? • When should a trial be stopped because it shows sufficient effect/no effect?

15. But Hard to Use Systemwide as Investment Criterion • “Systemwide” itself suggests new thinking • Optimizing across decision points within a pathway • Optimizing across pathways/diseases

16. 3. Value of Government-Funded Research to Reduce Uncertainty • Standard argument for government support of basic research • But may carry over to translational research/innovation development as well • Are there certain classes of innovations where industry is more/less risk-averse and therefore value in greater/less publicly funded effort?

17. NIH Review Criteria Incorporate Value of Uncertainty Reduction, Qualitatively • “Innovation” one of five NIH-wide review criteria for awards • Within translational research, may influence system toward: • support of “first in class” innovations • support for creating new, broadly applicable technologies or models • Relative to other translational research • Program planning may benefit from more formal analysis

18. Overview • Introduction and Motivation • Desirable Metrics for Translational Research • Implications for the Future

19. Implications for Future • Currently much easier to describe quantitative metrics for assessing translational research at NIH than it is to use them • Move toward developing data sources that could be applied in the future to allow for more quantitative approaches • In the interim, evaluators need to be humble in developing innovative but infeasible methodologies. • And policy-makers should be mindful of limitations of quantitative methods as well.