toward a standard benefit cost methodology for publicly funded s t programs n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Toward a standard benefit-cost methodology for publicly-funded S&T programs PowerPoint Presentation
Download Presentation
Toward a standard benefit-cost methodology for publicly-funded S&T programs

Loading in 2 Seconds...

play fullscreen
1 / 36

Toward a standard benefit-cost methodology for publicly-funded S&T programs - PowerPoint PPT Presentation


  • 136 Views
  • Uploaded on

Toward a standard benefit-cost methodology for publicly-funded S&T programs. Crossing Borders, Crossing Boundaries 2005 Joint CES/AEA Conference Toronto, Canada October 26-29, 2005 Jeanne Powell Economic Assessment Office Advanced Technology Program

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Toward a standard benefit-cost methodology for publicly-funded S&T programs' - andra


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
toward a standard benefit cost methodology for publicly funded s t programs

Toward a standard benefit-cost methodology for publicly-funded S&T programs

Crossing Borders, Crossing Boundaries

2005 Joint CES/AEA Conference

Toronto, Canada

October 26-29, 2005

Jeanne Powell

Economic Assessment Office

Advanced Technology Program

National Institute of Standards and Technology

outline
Outline
  • Background
  • AEA 2005: Analysis of Differences and Considering an Approach to Comparability
  • Moving Forward
background
Background
  • ATP Mission and Project Selection Criteria
  • Evaluation Framework
  • Role and Types of Benefit-Cost Analysis
  • Performance Metrics
  • Benefit-Cost Studies To Date
  • Study Similarities
slide4

ATP Mission …To accelerate the development of innovative technologies for broad national benefit through partnerships with the private sector.

Chemistry

Biotechnology

Electronics

Manufacturing

Advanced

Materials

Photonics

Information

Technology

atp selection criteria
ATP Selection Criteria
  • Scientific and Technological Merit (50%)
    • Technical innovation
    • High technical risk with evidence of feasibility
    • Detailed technical plan
  • Potential for Broad-Based Economic Benefits (50%)
    • National economic benefits
    • Need for ATP funding
    • Pathway to economic benefits
what we measure when

Short-Term

Mid-Term

Longer-Term

ECONOMIC IMPACTS

Total

Economic Benefits

Benefits to

Awardees

-1 0 1 2 3 4 5 6 7 8 9

10

or More

Years

Announce

Compe-

tition

Announce

Award

Complete Project

Post-Project Period

What We Measure When

(Inputs/Outputs)

  • Award/participant characteristics
  • Project funding
  • R&D partnering
  • Acceleration of R&D
  • Innovative technology development
    • Patents
    • Publications
    • Competitive advantage
    • Prototype products & processes

(Outputs/Outcomes)

  • Commercial activity
    • New products
    • New processes
    • Licensing
  • Attraction of capital
  • Strategic alliances
  • Company growth

(Impacts)

  • Broad national economic benefits
    • Return on investment
      • Public
      • Private
      • Social
    • Inter-industry diffusion
    • Increased GDP/ tax base
    • Societal impacts
types of b c case studies in atp
Types of B-C Case Studies in ATP
  • Traditional cash-flow based, return on investment measures
    • Benefit-to-cost ratios
    • Internal rates of return
    • Net present value
  • Hedonic price index models
  • Macroeconomic models
benefit cost analysis return on investment measures
Benefit-cost Analysis:Return on Investment Measures
  • Private return
    • return to ATP project participants on their own investment
  • Public return
    • return to nation on ATP investment
  • Social return
    • return to ATP project participants and nation on total ATP and private investment
performance metrics
Performance Metrics

Return on Investment Metrics

  • Benefit-to-Cost ratio is computed by dividing the present value of benefits by the present value of investments
  • Net present value (NPV) is the present value of benefits minus the present value of investments
  • Internal rate of return is calculated by iterative solution for a rate at which the discounted value of investments equals the discounted value of benefits
atp s b c studies to date
ATP’s B-C Studies To Date
  • 12 studies published, including quantitative case studies of 28 projects
  • 2 studies underway, including quantitative case studies of 4 projects
  • All studies performed by independent contractors
study similarities
Study Similarities
  • Address ATP Mission, emphasizing public benefits to industry customers and end users and need for ATP funding to achieve these benefits
  • Cash-flow based approaches use:
    • Good practices consistent with public finance literature; NPV, B:C, IRR
    • OMB-mandated 7% real discount rate (all but 1 study) and constant $ estimates
    • Year of ATP project start for base year
    • Relatively short study periods
  • Results are not presented as “representative”, but rather as portfolio minimums
aea 2005
AEA 2005
  • Analyzing Differences and Considering an Approach to Study Comparability
aea 2005 analyzing differences and considering an approach to study comparability
AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability
  • Analyzing four major differences :

1) Study timing relative to ATP funding and uncertainties about future outcomes

2) Identifying the counterfactual and attribution of benefits

3) Which metrics? Public versus social return on investment

4) Adjusting for timing differences across studies and projects

aea 2005 analyzing differences and considering an approach to study comparability1
AEA 2005: Analyzing Differences and Considering an Approach to Study Comparability
  • Considering a standard approach based on three criteria:

1) Meeting evaluation objective

2) Quality and accuracy

3) Comparability

1 study timing and uncertainties about future outcomes
1. Study Timing and Uncertainties About Future Outcomes

Issues:

  • Uncertainties concerning technology benefits are reduced over course of innovation process
    • At time of ATP funding, risks of technical failure (or of meeting only limited technical objectives) are very high.
    • Market/financial/business risks remain high during and after ATP funding phases of technology development and into commercialization phases
    • Innovation life-cycle/timing varies by technology area
  • Earliest studies conducted early in innovation process
    • No projects had matured
    • Direct experience with range of possible outcomes (and empirical data about innovation process) was limited
1 study timing and uncertainties about future outcomes contrasting studies1
1. Study Timing and Uncertainties About Future Outcomes: Contrasting Studies

Notes:

  • Tissue Engineering: 3 technologies made considerable technical progress and commercialization continues to evolve but much more slowly than anticipated; 3 technologies failed to develop and companies dissolved; 1 is in transition to new company, future as yet is uncertain
  • Component-Based Software: focus on existing products and short-term product life reduced the amount of research required per project and enabled study of more projects
1 study timing and uncertainties about future outcomes conclusions
1. Study Timing and Uncertainties About Future Outcomes: Conclusions

Under conditions of considerable technical/business uncertainty about future outcomes:

  • Rigorous, complex economic models may or may not generate useful performance metrics.
  • Probability-based outcome measures that consider the broad technical, market, and business risks, as well as technology benefits, are likely needed
  • Metrics are more reliable/capable of precision as projects overcome technical, market, and financial barriers
2 identifying the counterfactual and attribution of benefits
2. Identifying the Counterfactual and Attribution of Benefits

Issue:

  • Where does project being analyzed start and end? And what is the counterfactual, or baseline?
    • In theory, investment costs and benefits included in analysis need to measure increment over the counterfactual (what would have happened without the project being measured).
    • In practice, benefits (e.g., profit from sale of a product embodying new technology) derive from multiple public and private investments and from one or two of many product generations.
2 identifying the counterfactual and attribution of benefits conclusions
2. Identifying the Counterfactual and Attribution of Benefits—Conclusions
  • Researchers use a variety of mechanisms to model benefits relative to a counterfactual (e.g., increased probability of achieving benefits, acceleration, allocation in accordance with amount or importance of funding sources)
  • ATP-funded technologies are often embodied in multiple products, each of which resulted from a number of different public/private funding sources and “projects”. Researchers measure benefits of products with strongest ATP linkages and share attribution where needed
  • Social rate of return has weaker assumption about “causation” than public return on ATP $ does.
3 which metrics
3. Which metrics?
  • Social return metrics
    • Economics literature compares social rate of return to private rate of return (to see economic spillover) and then further compares private rate of return to private hurdle rate to identify market failure that justifies public funding
    • Cost-shared projects imply comparison of benefits to both public and private recipients with investments from all sources

AND/OR

  • Return on ATP investment
    • Directly addresses: What is the ATP program impact?
    • Involves substantially less historical/company proprietary information than social return metrics
3 which metrics contrasting studies
3. Which metrics? Contrasting Studies

Issue:

  • Some studies/contractors emphasize social return metrics; others emphasize public return
3 which metrics social versus public returns conclusions
3. Which metrics? Social versus Public Returns—Conclusions
  • Both have merit for different purposes, and both have advantages and disadvantages
    • Social return metrics reflect cost-sharing aspect of ATP and other public-private partnership projects; however, they require company-proprietary data and historical data about multiple funding sources (generally only rough estimate is feasible) .
    • Public return on ATP investment provides measure of direct program impact; however, it requires judgments about of “causation”
4 adjusting for timing differences across studies and projects constant and base year
4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year

Issue:

  • Different case studies based cash flow estimates on constant dollars of different years
  • Although nearly all studies used the same discount rate (7%-real), they discounted cash flow estimates to different points in time (start of each project). (This practice evolved in financial world’s prospective analysis of potential investments and is used in Excel.)
  • Thus, reported NPV metrics are not comparable across projects.
slide29
4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 1: Ranking does not change
slide30
4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example 2 : Ranking Changes
adjusting to constant 2005
Adjusting to Constant 2005$

NPV in Constant 2005$ =

Where: IPDGDP is Implicit Price Deflator for GDP

and IPDGDPstudy is IPDGDP for Constant $ year used in study

  • Q2 2005 IPDGDP – IPDGDPstudy
  • IDPGDPstudy

1 +

x Reported NPV

adjusting base year to 2005
Adjusting Base Year to 2005

Adjusted NPV for Base Year 2005 =

NPV in Constant 2005$ x 1.07n

Where: n = 2005 minus base year used in study

4 adjusting for timing differences across studies and projects constant and base year example
4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year Example

Note:

  • Rankings “before and after” adjustment are for illustrative purposes only.
    • NPVs indicate net benefit magnitudes
    • NPVs do NOT indicate investment efficiency
4 adjusting for timing differences across studies and projects constant and base year1
4. Adjusting for Timing Differences Across Studies and Projects: Constant $ and Base Year

Conclusions:

  • Both computed value of NPV and rank order can change by adjustment to constant $ and common base year
  • NPV adjustments to constant $ and common base year are easy to make—directly from reported NPV
  • Comparability and aggregation of NPV results indicate need for such adjustments although practitioners may differ on best approach
  • IRR and B:C do not require adjustment; however, they cannot be aggregated
moving forward
Moving Forward
  • Work with contractors/researchers toward greater understanding of need for comparability of B-C results while seeking improved methodologies
  • Recognize and build on strengths of individual contractors/researchers ; help them overcome inconsistencies with comparability goals
  • Improve understanding of B-C methodologies among S&T evaluation community
  • Work with NIST experts in both B-C analysis and standards development
for further information
For further information:
  • Call Jeanne Powell: 301-975-4196
  • Send e-mail to: jeanne.powell@nist.gov
  • Visit our website:www.atp.nist.gov
  • View our publications: www.atp.nist.gov/eao/eao_pubs.htm