David Marzo Operations Research Analyst, Weapon Systems Cost Analysis Division U.S. Department of Defense OSD Cost Asses

2. Agenda OSD CA Overview Recurring Challenges & Considerations Joint Strike Fighter Example to illustrate CAPE estimating proclivities

3. OSD CAPE Overview Analytical arm within the Office of the Secretary of Defense Cost Assessment (CA): Mandatory independent cost estimates and internal discretionary reviews Program Evaluation (PE): Broader portfolio review & oversight CA also known as the CAIG or Cost Analysis Improvement CAIG Established in 1972 Bolstered through a series of regulations & acts, most recently the 2009 Weapon Systems Acquisition Reform Act

5. Agenda OSD CA Overview Recurring Challenges & Considerations Joint Strike Fighter Example to illustrate CAPE estimating proclivities

6. Cost Estimating Challenges Program definition uncertainty Fallacy of Detail “Black Box” Models Optimism Actuals: The “Gold Standard”

8. Fallacy of Detail Level of aggregation is probably one of the most important decisions made in an estimate Greater complexity/detail is often seen by the authors as an index of estimate quality (after all, it reflects great effort) Subject to the erosion of accuracy for the sake of precision Observation: Cost Analysis—as a discipline—inevitably relies on the proposition of correlation Highly correlated elements MUST be consolidated Failure to consolidate correlated elements will provide erroneous results This is the fundamental flaw in cost risk analysis as generally practiced Consolidating correlated cost elements simplifies the estimating effort, and focuses analytic attention on the fundamental drivers of cost Note that the availability of lower level data does not compel its use

9. “Black Box” Models and Evidence First requirement of an estimate is that it be trusted Estimator is asking senior managers to bet their careers . . . Must explain—in quite specific terms—the chain that runs from data/evidence, through models/methods to final estimate Proprietary “closed” models are inherently problematic Many commercial forms One is left with showing inputs, black box, and estimates The best one can say of credibility is the “trust” in the trademark As an estimate consumer, one inevitably wonders How would I know if X model is appropriate for this application? Have the inputs been gamed, particularly through “calibration”?

10. Optimism Delusions of Success By Dan Lavallo and Daniel Kahnemon, Harvard Business Review, July 2003 Key Points from the Article When planning major initiatives Routinely exaggerate benefits and discount costs when projecting risky project Sets up project for failure; psychologists “planning fallacy” -- spin success scenarios and overlook potential for mistakes Optimism traced to cognitive biases and organizational pressures People highly optimistic most of the time – exaggerate talents and degree of control; attribute negative consequences to external factors Competition ($, time) for new projects – incentive to accentuate the positive Anchoring magnifies optimism; initial plan accentuates positive, skews subsequent analysis Most pronounced for initiatives companies have never attempted before Temper with “outside view” Supplements traditional forecasting w/ statistical analysis of analogous efforts – reality check on initiative inside view Don’t remove optimism, but temper effects -- balance between optimism and realism - goals (motivate) and forecasts (decide whether to make commitment in the first place)

11. Actuals: The Gold Standard Actuals have one sterling advantage: they reflect all “unknown/unknown” problems of achieving a capability Actuals-based estimates are easy to explain Two distinct cases of actuals estimating: Early Actuals on a specific program to extrapolate future lots Actuals from other programs, often adjusted for content/scope, to forecast costs of new program

12. Is Cost A Self-Fulfilling Prophecy? Common Complaints “CAPE Estimates are inherently negative and reinforce past mistakes” “I don’t need all of this extra money—if you give it to me, the program will just spend it unnecessarily.” “Cost is a self-fulfilling prophecy” Thoughts Programs can address known “mistakes” from past programs and not repeat them – but each program has its unique unknowns—the inevitable challenges of complex weapon systems Trick is to find the dividing line between too much and not enough –the “thesis analogy” Good management has an important role

13. Agenda OSD CA Overview Recurring Challenges & Considerations Joint Strike Fighter Example to illustrate CAPE estimating proclivities

14. Joint Strike Fighter What is the program status? How is CAPE involved? How do we approach estimating it? What are some of the unique estimating challenges? How does the Department react to differing cost estimates? What are the impacts of international procurements?

15. JSF – Program Status Program underwent a “Nunn-McCurdy” cost breach in summer 2010 Program originally based in 2002 Projected unit cost >50% resulted in critical breach Department of Defense must re-assess requirements, cost, status Summer 2010: Department states no other alternative meets requirements within acceptable cost—proceed with JSF

16. JSF Program Status Fall 2010 : Comprehensive Technical Baseline review resulted in a restructured development program Block 3 Qualification Projected Complete in 2016 CAPE position: reasonable development baseline established; focus on execution; challenges remain

17. CAPE Involvement Departmental Internal Requests 2008 Joint Estimating Team (“JET 1”) 2009 Joint Estimating Team (“JET II”) Statutory Requirements for Cost Estimates 2010 Nunn McCurdy Cost Breach Review: Development and procurement 2011 Milestone B Review (Ongoing): Development, procurement, operating and support

18. JSF Estimating Approach Development Function of schedule and personnel staffing levels; “time is money” Remaining schedule is a function of test point burndown Project actual costs forward; shaped by legacy history Procurement Pre Nunn McCurdy: Based on analogy to F/A-18, F-22; adjusted for weight and system complexity Since Nunn McCurdy Based on actuals from JSF lots; bounded by legacy history Separately model each section of aircraft; and separate major contractors

19. Theoretical Example

20. Challenge of Modeling Commonality

21. Data Example #1

22. Data Example #2

23. History as a Bound

24. The Challenge of Concurrent Development and Production

25. Operating and Support Cost Operating and support (O&S) cost It will cost more to operate and maintain JSF than legacy fighter platforms; however, JSF is more capable Significant drivers of O&S cost (e.g., Depot Level Reparables, engine/fuel consumption) are driven by inherent aircraft characteristics already designed in (e.g., stealth, complex mission systems) Other significant drivers of O&S cost are driven by ops requirements (e.g., mission personnel & flight hours)

26. Annual “Steady State” Cost Example

27. Other Challenges of Estimation Concurrency of phases—same contractor staff charging simultaneously to development, production, and sustainment—need to account for correctly International buy profiles – international purchases move processes further down the curve; when US defers purchases, how do we model shifts in international profiles? Evolution of production processes – what is built in house now may be outsourced later

28. Department Treatment of Cost Estimates Department leadership has a choice against which estimates to baseline and budget against “Fund to program estimate” or “Fund to CAPE estimate” Each program is different For JSF, this has been an evolutionary process— For development, incremental movement over past 3 years towards funding to CAPE and Tech Baseline Review estimates For procurement, Department has funded to CAPE estimates. There is still ambiguity, however, with respect to the effects of commonality and its impact on costs Assessing risk and funding to the 50% estimates vs 80% estimates

29. Impact of International Participation We estimate that the US gets about a 5% unit cost benefit from international participation International participation does introduce complexities Establishment of production capabilities in other countries Shifting quantity profiles

30. Closing Comments CAPE has responsibility for performing cost estimates in response to both US statute as well as internal Departmental requests Our approach emphasizes: Using a combination of historical analogous data & actual costs when available Relatively simple but sound logic JSF is a good example of our involvement and cost estimating methodology; we recognize that estimating JSF is a dynamic process informed by ongoing dialogue with the contractor and program office