Phil beenhouwer may 17 2006
Sponsored Links
This presentation is the property of its rightful owner.
1 / 27

Phil Beenhouwer May 17, 2006 PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Risk Analysis & Estimating Uncertainty …and what this has to do with the price of milk in McLean. The Society of Cost Estimating & Analysis (SCEA) Greater Washington DC Chapter. Phil Beenhouwer May 17, 2006. Agenda. The Problem Variability & Standard Deviation Briefing Goals

Download Presentation

Phil Beenhouwer May 17, 2006

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Phil beenhouwer may 17 2006

Risk Analysis & Estimating Uncertainty…and what this has to do with the price of milk in McLean.The Society of Cost Estimating & Analysis (SCEA)Greater Washington DC Chapter

Phil Beenhouwer

May 17, 2006



  • The Problem

  • Variability & Standard Deviation

  • Briefing Goals

  • An Uncertainty Primer

  • Definitions

  • Distributions and Simulation…

  • Excel Prowess

  • Benefits, Headlines, Other Disciplines

My premise

My Premise…

There are three kinds of people in the World…….,

…………….those who can count, and those who can’t.

The lack of risk/uncertainty analysis and

poor cost/schedule estimating (across all levels of costing)

are the primary reasons that Programs are over-budget.

The problem




Historical data point

Cost estimating relationship

Standard percent error bounds

Cost Driver

Input variable

The Problem

"It's tough to make predictions, especially about the future."

-- Yogi Berra.

Combined Cost Modeling and Cost Driver Uncertainty

Cost = a + bXc

Cost Modeling Uncertainty

Cost Driver Uncertainty

Source: Timothy P. Anderson,

The Aerospace Corporation,

2005 DoDCAS Symposium

Variability and standard deviation

“Variability” and “Standard Deviation”

“Figures lie and liars figure…”

"There are lies, damned lies and statistics."

  • When someone says it’ll cost you $100…

  • The average American carries $9,000 in credit card debt.1

    • In reality, most Americans owe nothing to credit card companies.

    • Most households that carry balances owe $2,000 or less.

    • Only about 1 in 20 American households owes $8,000 or more on credit cards.

1, a service that tracks credit card trends.

Briefing goals

Hmmm…, this Contractor’s saying that the kitchen remodel will cost twenty grand, but…

Briefing Goals

Timmy’s saying that the soccer trip to Europe will only cost three grand, but…

  • The “threshold” is for you to think in terms of risk, uncertainty, and variability…

You know, Phil….., we may have to place a sensor suite anywhere from 1.2 to 3.1 miles…

Hey Phil..., there’s a good chance that we’ll have to replace the hardware every two to three years…

Phil…, I think we should plan for a worst-case of 2.5 million transactions per year…

  • The “objective” is for you to speak in terms of risk, uncertainty, and variability…, and then to apply it…

An uncertainty primer slide 1 of 4

SDLC Phase

Concept and Business Case

Initiation and Authorization

Project Definition

System Design



Operational Readiness


Resolve the lack of milk at home

Spousal approval/funding

Buy a gallon of milk

Find merchant along route home

Drive to store and purchase milk

Did you have enough money?

Get a clean glass!

An Uncertainty Primer… (Slide 1 of 4)

An uncertainty primer slide 2 of 4


An Uncertainty Primer… (Slide 2 of 4)

  • Gallon of Milk Data Collection

    • Convenience Store: $3.49

    • Warehouse Club #1: $2.59

    • Warehouse Club #2: $2.69

    • Grocery Store #1: $2.69

    • Grocery Store #2: $3.19

    • Grocery Store #3: $2.89

    • Grocery Store #4: $2.79

    • Specialty Store #1: $3.09

    • Specialty Store #2: $2.99

    • Specialty Store #3: $3.29

An uncertainty primer slide 3 of 4

An Uncertainty Primer… (Slide 3 of 4)

  • So, if you budgeted $3.00 for milk (the mean), you can go to…

    • Warehouse Club #1: $2.59

    • Warehouse Club #2: $2.69

    • Grocery Store #3: $2.89

    • Grocery Store #1: $2.69

    • Specialty Store #2: $2.99

    • Grocery Store #4: $2.79

  • But you can’t go to…

    • Convenience Store: $3.49

    • Specialty Store #3: $3.29

    • Grocery Store #2: $3.19

    • Specialty Store #1: $3.09

I could go here with 21 cents more…

80% of the stores sell milk for less than $3.21 per gallon.

An uncertainty primer slide 4 of 4

An Uncertainty Primer… (Slide 4 of 4)

  • But you can’t go to…

    • Conv. Store: $3.49

    • Spec. Store #3: $3.29

    • Grocery Store #2: $3.19

    • Spec. Store #1: $3.09

  • What can you afford to buy at these stores?

    • Is a half-gallon acceptable?

    • Will you even leave with milk?

    • How will ‘Operational Readiness’ go when you get home?

    • Can you use ‘legacy’ orange juice in tomorrow’s cereal instead?

    • Will you need to reduce the number of cereal users?

    • Will you need to cut all cereal training from the budget?

    • Will there be a GAO report on your pillow in the morning?

  • “Sure”, you say, “if I end-up at the Convenience Store with only $3.21, I can find the 28 cents I need from under the driver’s seat…”.

    • But what if these weren’t dollars, but billions of dollars…?

    • Could you find $28M under the Program Manager’s seat?

Program risk versus estimating uncertainty

“Program Risk” versus“Estimating Uncertainty”

  • Total Risk = RiskProgram + (+/- UncertaintyEstimation)1

  • For the purposes of quantitative risk analysis, I have defined:

    • “Program Risk” – the probability and severity of loss linked to hazards2 (e.g., software development cannot begin if the environment is not ready, system testing might fail, etc.)

    • “Estimating Uncertainty” -- the estimated amount or percentage by which an observed or calculated value may differ from the true value3 (e.g., the number of users could be 25% less, the COTS license cost could be $1,000 more, training could take one week longer, etc.)

  • Should also consider benefits and schedule, in addition to cost

2Source: DoD Dictionary,

1Source: Keith Horenstein,

The MITRE Corporation




  • Simulation: any analytical method meant to imitate a real-life system.1

  • Monte Carlo simulation: a simulation that randomly generates values for uncertain variables over and over to simulate a model.1

  • xth percentile: the percentage at which x% of all outputs are at, or below, the associated cost value

    • i.e., the 80th percentile in the gallon of milk example means that 80% of the values are at, or below, $3.21.

    • Conversely, 20% of the values exceed $3.21.

1Source: Decisioneering’s website;

Description of monte carlo tools

Description of Monte Carlo Tools

  • “Crystal Ball applications transform your spreadsheets into dynamic models that solve almost any problem involving uncertainty, variability and risk.”1

  • “Simply by running a simulation, @RISK takes your spreadsheet model from representing just one possible outcome to representing thousands of possible outcomes.”2

2Source: Palisade

1Source: Decisioneering’s website;

Triangular distributions

Triangular Distributions…

Perfectly symmetrical;

used far too often

Probably the most realistic….,

‘low’ and ‘most likely’ are the same!!

Getting better…., skewed to the right

Triangular distributions1

…Triangular Distributions…

80th Percentile

80th Percentile

80th Percentile

Monte carlo simulation

Monte Carlo Simulation

Monte carlo output

Monte Carlo Output

Other common distributions

…Other Common Distributions…

Uniform distribution: represents a range with no ‘most likely’ value

Normal distribution: bell-curved shape represents exam. scores, height of the people in this room, etc.

Discrete distribution: distinct points represents the roll of a die

Another discrete distribution: represents the unique costs for four pieces of hardware

With a little excel prowess

With a little Excel prowess…

With every iteration, the ‘number of years’ changes, and this formula computes the annual cost based on the ‘number of years’.

With a little excel prowess1

With a little Excel prowess…

Benefits to quantifying risk uncertainty using monte carlo

Benefits to Quantifying Risk/Uncertainty& Using Monte Carlo

  • Identify and apply risk/uncertainty within a model where it really exists (I.e., risk/uncertainty does not really exist “+/- 10%” around software integration!)

  • Sensitivity analysis

  • Risk-adjusted estimates are included in the individual items of the model instead of as a bottom-line “tax”

    • Makes it harder for decision-makers to remove the “risk” line-item

Just present all your numbers

at the 80th percentile!



  • GAO Testimony: “CAPITOL VISITOR CENTER -- Results of Risk-based Analysis of Schedule and Cost”, GAO-06-440T, February 15, 2006

  • GAO Report: “INFORMATION TECHNOLOGY -- Agencies Need to Improve the Accuracy and Reliability of Investment Information”, GAO-06-250, January 2006

  • GAO Report: “DEFENSE MANAGEMENT -- Additional Actions Needed to Enhance DOD’s Risk-Based Approach for Making Resource Decisions”, GAO-06-13, November 2005

Relationship to other disciplines

Relationship to Other Disciplines

  • Budgeting / Investment Planning

    • Provides risk-adjusted requirements of the E300 and other planning activities

  • Acquisition

    • Cost, Schedule, and Performance are part of defining an acquisition strategy

  • Contracting

    • Provides a cost basis for negotiation

  • Program Management

    • Provides insight into Program risks; helps prioritize mitigations

    • It’s just good Program Management !!

  • Earned Value/Baseline Management

    • Provides an input to the management reserve level

    • More objective inputs to the EVMS than the typical Integrator provides

    • Quantification of risks and uncertainties will result in less re-baselines

  • Engineering

    • Offers an approach to incorporate uncertain aspects of the engineering design

Ms project monte carlo analysis

MS Project Monte Carlo Analysis



  • “Think” and “speak” in terms of risk and uncertainty (and then apply it…)

    • Collect uncertainty inputs when you gather data

    • Use Monte Carlo applications (e.g., Crystal Ball, @Risk, etc.)

      • These are relatively inexpensive compared to other applications in a coster’s toolkit

      • There is even a $15 application that we are currently investigating (Excel Business Solutions’ XL Modeling:

  • Include “Program Risk” and “Estimating Uncertainty” in cost, benefit, and schedule analyses

And if all else fails re define the word outlier

And if all else fails…..,re-define the word “outlier”…

Questions comments

Questions / Comments

  • Login