T&E Risk Management

1. T&E Risk Management

2. DoD Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs Available on-line through the Risk Management website on the ACC at https://acc.dau.mil • Redefines Risk, Issue and Opportunity Management Process • Provides detailed guidance on key activities: • Risk Planning • Risk Identification • Root Cause Determination • Risk Analysis • Risk Handling • Risk Monitoring

3. Risk, Issue, and Opportunity Definitions • Risks are future events or conditions that may have a negative effect on achieving program objectives for cost, schedule, and performance. Risks are defined by (1) the probability (greater than 0, less than 1) of an undesired event or condition and (2) the consequences, impact, or severity of the undesired event, were it to occur. • Issues are events or conditions with negative effect that have occurred (such as realized risks) or are certain to occur (probability of 1) in the future that should be addressed. • Opportunities are potential future benefits to the program’s cost, schedule, and/or performance baseline, usually achieved through reallocation of resources. Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

4. Risk Defined • Risk addresses the potential variation in the planned approach & its expected outcome. • Risk has three components: • A future root cause (yet to happen) • A probability (or likelihood) • The consequence (or effect)

5. Risk Management Defined Risk Management is a fundamental program management tool for effectively managing future uncertainties associated with system acquisition.

6. Overview of Potential Sources of Program Risks, Issues, and Opportunities Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

7. Risk Management vs. Issue Management • Risk management applies resources to mitigate future potential root causes and their consequences • Issue management applies resources to address and resolve current problems • If a root cause is described in the past tense, the root cause has already occurred, and hence, it is an issue to be resolved, but it is not a risk.

8. DoD Risk Management Process Steps Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

9. Risk Planning When planning for risk management: • Assign roles, responsibilities, and authorities • Select and document overall approach: • Process and procedures • Risk analysis criteria for likelihood and consequences • Risk handling procedures • Establish traceability of risk to technical requirements and overall program objectives • Align government and contractor roles, responsibilities, tools, and information exchange • Determine risk management resources, to include budget, facilities, personnel, schedule • Determine risk management battle rhythm Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

10. Risk Identification—What can go wrong? When identifying risks: • Example risk ID methodologies: • Independent assessments • SOW requirements • Brainstorming sessions with SMEs • Interviews with IPT leads, Systems Command/Center competencies • Review of similar/historical programs • Trade studies • Review analysis of Technical Performance Measurements, resource data, life cycle cost information, WBS/IMS/EVM data trends, and progress against critical path • Review Systems Engineering Technical Review Checklists Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

11. Risk Identification • Risk identification is the activity that examines each element of the program to identify root causes, begin their documentation and sets the stage for their successful management • The intent of risk identification is to answer the question “What can go wrong?” As an example… • Monitoring test results especially test failures (readiness results and readiness problems for the sustainment phase)

12. Risk Identification (cont.) • Reduce program elements to a level of detail that permits an evaluator to understand the significance of any risk and identify root causes • Typical risk sources include: • Test and Evaluation: adequacy and capability of the test and evaluation program to assess attainment of significant performance specifications and determine whether the system is operationally effective, operationally suitable, and interoperable • Modeling and Simulation (M&S): adequacy and capability of M&S to support all life-cycle phases of a program using verified, validated, and accredited models and simulations Ask the “why” question multiple times to get past the symptoms and find the root cause

13. Risk Analysis—How big is the risk? Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015 When analyzing risks: • Quantify the cost, schedule, and performance impacts: • RDT&E costs • Procurement costs • O&S costs • Performance thresholds • Schedule thresholds • Assess the likelihood of the risk being realized • Conduct analysis periodically to support cost, schedule, and performance risk assessments

14. Risk Analysis • The intent of risk analysis is to answer the question “How big is the risk?” by: • Considering the likelihood of the root cause occurrence • Identifying the possible consequences in terms of performance, schedule and cost • Identifying the risk level using the Risk Reporting Matrix

15. Levels of Likelihood Criteria For example, if the root cause has a 50% probability of occurring, the corresponding likelihood is Level 3. The level of likelihood of each root cause is established using specified criteria.

16. Recommended Consequence Criteria Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

17. Risk Reporting Matrix and Criteria Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

18. T&E Example of Risk Reporting Circuit Card Availability (S) Aggressive development project may not deliver circuit cards in time to support development testing. Develop interim test bench and test methods to support integral development and test activity until full capability is available.

19. Risk Handling—What’s the plan? When handling risks: • Consider the accept, avoid, and transfer handling options, not just the mitigation option • Choose the best handling option, then select the best implementation approach for that option • Take into consideration elevating the risk to the appropriate tier (management or executive level) • Include cross-program risks in order to consider the impact of risk management actions on other programs Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

20. Risk Handling Options • Control the cause or consequence. Use the design process (including testing, Design of Experiments, M&S, etc.) to lower risk to acceptable levels. • Avoidance by eliminating the root cause and/or consequence. Force the problem to go away; never give it an opportunity to occur. (Ex: change the requirements or design concepts) • Acceptance (Assumption) of the level of risk and continue on the current program path. (The risk is low in probability and/or consequence) • Transfer the risk. Move the problem from one area of design to another, or from one organization to another. (Ex: hardware to software, or Government to Contractor e.g. a warranty or change contract types)

21. Seeker Dome Risk Mitigation Timeline Diagram

22. Risk Monitoring—How has the risk changed? When monitoring risks: • Track the implementation and progress of the risk handling strategy, not just the development and planning of the selected strategy • Include Technical Performance Measures as an integral activity when monitoring risks after selecting the appropriate risk handling strategy • Conduct regular status updates to monitor risks for changes to likelihood and/or consequences • Document risks that can be retired as well as risks that are still being managed to prevent an unnoticed relapse of the retired risk • Keep lines of communication open to notify management when ability to handle the risk is ineffective Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

23. Value of T&E in Risk Management • OSD DT&E Study 1999 and 2002: • Industry sees test as a risk reduction • Some people within DoD see test as a risk “ Government weapon programs do not have the same market-created measures as in the private sector to demonstrate the value of testing such as warranties, recalls and class action law suits that are real in the private sector and that provide a cost risk to industry which testing helps reduce.” Thomas Christie, DOT&E before the Senate Armed Services Committee, May 21 2002.

24. Value of T&E in Risk Management (cont.) • Risk management is the means by which the program areas of vulnerability & concern are identified & managed. • T&E is the discipline that helps to illuminate those areas of vulnerability • As the GAO stated in their July 2000 report: • “GAO recommends that acquisition managers structure test plans around the attainment of increasing levels of product maturity, orchestrate the right mix of tools to validate these maturity levels, & build & resource acquisition strategies around this approach.”

25. Issues –100% Now/Future • OSD has found that program issues are, too often, mistakenly characterized as risks. • This practice is reactive and tends to blind the program to true risk management. Risk management applies resources to lessen the likelihood, or in some cases, the consequence, of a future event. • Issue management, on the other hand, applies resources to address and resolve a past or occurring event and its related consequences. • When a negative event has been identified and has a past or present impact to the cost, schedule, or performance of a program, it is not a risk. • These events should be cataloged as issues and should be addressed within the program’s normal issue management process. • In addition, even though an issue may introduce a likely future consequence, this does not make it a risk. • To ensure issues and risks are properly identified, programs should have an issue management approach to identify problems and track associated closure plans. • Programs should also assess whether issues are spawning prospective risks.

26. DoD Issue Management Process Steps Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

27. Issues—Identification • Issues are best identified before the beginning of a new project or contract and should be updated and reviewed periodically throughout the life cycle of the program. • Unlike opportunities and risks, there is no assessment of their likelihood because issues have either already occurred or are in the process of occurring. Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

28. Issues—Handle • Accept: Accept the consequence of the issue based on results of the cost/schedule/performance business case analysis. • Avoid: Eliminate the consequence of the event or condition by taking an alternate path. • Examples may involve changing a requirement, specification, design, or operating procedure. • Transfer: Reassign or reallocate the issue responsibility from one program to another, between the government and the prime contractor, within government agencies, or across two sides of an interface managed by the same organization. • Mitigate: Implement a strategy to reduce the issue consequences and residual risk to as low as possible or minimize its impact on the program.

29. Opportunities • An opportunity is the potential for improving the program in terms of cost, schedule, and performance. • Opportunity management supports USD(AT&L) Better Buying Power initiatives (currently BBP 3.0) to achieve “should cost” objectives. • In Better Buying Power 2.0, the USD(AT&L) implemented “should cost” management, stating, “Our goal should be to identify opportunities to do better and to manage toward that goal.” • Managers should scrutinize each element of cost under their control and assess how it can be reduced without unacceptable reductions in value received

30. DoD Opportunity Management Process Steps Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

31. Opportunity Management Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

32. Opportunity Matrix and Criteria Source: Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015

33. Summary • Life-Cycle Risk Management is a structured process • Process for risk identification and risk analysis • Risk Management is required and is smart business • A Robust T&E Program is Essential for Effective Risk Identification and Handling. • Provides a foundation for program risk management execution.

34. Team Assignments • Each Team: • Read your assigned case study • Identify three potential root causes for your case • Pick one root cause and identify a potential solution for the case • Identify and address any risks generated by implementing your solution • Each team will present their answers to the class

35. Team Assignments • Team Case Assignments: • Team 1—V-22 Joint Services Advanced Vertical Lift Aircraft • Team 2—CH-47F Improved Cargo Helicopter (CH-47F) • Team 3—Expeditionary Fighting Vehicle (EFV) • Team 4—Global Hawk Unmanned Aerial Vehicle • Team 5—C-5 Reliability Enhancement and Re-engining Program (C-5 RERP) • Cases are excerpts from: GAO-05-301 Assessments of Selected Major Weapon Programs

36. For Further Information • OSD/AT&L Defense Acquisition Portal • https://dap.dau.mil/Pages/Default.aspx • DAU Acquisition Community Connection (ACC) • https://acc.dau.mil/rm (Risk Management) • https://acc.dau.mil/t&e (Test & Evaluation) • Test and Evaluation References • Defense Acquisition Guidebook, Chapter 9, Integrated T & E • DoD Instruction 5000.2, Enclosure 5, Integrated T & E • Other T&E website references may be found in the “SE T&E Resources” file on your student CD-ROM