The Application Of Risk
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

Larry Szutenbach OASN(RD&A) PP&R [email protected] PowerPoint PPT Presentation


  • 142 Views
  • Uploaded on
  • Presentation posted in: General

The Application Of Risk In Metrics Used For Department of Navy Weapon Systems Acquisition (May 9, 2002). Larry Szutenbach OASN(RD&A) PP&R [email protected] This Presentation Will Briefly Touch on the Use of Risk in Our: Program Summary Documents (PSDs)

Download Presentation

Larry Szutenbach OASN(RD&A) PP&R [email protected]

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

The Application Of Risk In Metrics Used For Department of Navy Weapon Systems Acquisition(May 9, 2002)

Larry Szutenbach

OASN(RD&A) PP&[email protected]


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • This Presentation Will Briefly Touch on the Use of Risk in Our:

    • Program Summary Documents (PSDs)

    • Program Performance Metrics

    • Defense Acquisition Executive Summary (DAES)

    • Integrated Baseline Reviews

    • Consideration of Scorecards


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Risk as used in Defense:

  • The potential for a

  • Two characteristics:

    • Probability of occurrence

    • Consequences

  • Use of risk in our metrics:

  • Often intentionally mix performance with risk, but it can be important to recognize the difference.

  • Sometimes risk is indirectly addressed, such as in the EVM “Bullseye” chart

negative

future

reality.


Program assessments and psds

Program Assessment Indicators

Performance CharacteristicsGreen

ScheduleGreen

Test & EvaluationGreen

ContractsGreen

Logistics Reqts & Readiness Obj Green

ProductionGreen

CostGreen

Management StructureGreen

FundingGreen

Interoperability Green

Managers’ Assessment (for each Assessment Indicator rated Green Advisory, Yellow, Yellow Advisory, or Red).

Program Assessments and PSDs

Program Assessment Indicators/Program Manager’s Assessment(indicate Green/Yellow/Red, Program managers comments, an assessment based on current funding profile and current approved Acquisition Program Baseline)

DAES Report

  • Schedule Performance.

  • Contracts.

  • Production.

  • Management Structure.

  • Performance Characteristics.

  •  Test and Evaluation.

  •  Logistics Requirements and Readiness Objectives.  

  • Cost Performance.

  • Funding.


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

ACAT III and IV Program Status

Mr. Randel H. Stone Date of Review: 16 Nov 2000

PEO (W)

ACAT III & I V

ACAT

PM CODE

PROGRAM TITLE

APB DATE

COST

SCH

PERF

OVERALL

IIIPMA201GBU-24 E/BOCT 99

IIIPMA208IMPROVED TACTICAL AIR LAUNCHED DECOYDEC 99

IIIPMA242HARM BLOCK VI (IHUP)JUL 99

IIIPMA281AFLOAT PLANNING SYS (APS)SEP 99

IIIPMA281JOINT SERV IMAGING PROC SYS NAVY (JSIPS)SEP 98

IIIPMA282TACTICAL TOMAHAWK WEAPONS CONT SYS -

IIIPMA282ADV TOMAHAWK WPN CONTROL SYS (ATWCS) DEC 99

IVMPMA208SUPERSONIC SEA SKIMMING TRGT (SSST) JUN 99

IVMPMA20821ST CENTURY AERIAL TARGET (TARGET 21) DEC 99

IVMPMA208BQM-74E MOBILE SUBSONIC AERIAL TGTDEC 99

IVMPMA208VANDAL EXTENDED RANGE (EER)MAY 97

IVMPMA208AQM-37C SUPERSONIC TARGET SYSTEMDEC 99

IVMPMA208QF-4S FULL SCALE A/C TRGT SYS DEC 99

IVTPMA258AN / AWW-13 ADVANCED DATA LINK POD AUG 91


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Color Coded Rating System For DAES

  • And DoN PSDs and Program Performance Metrics

  • The DAESrating system [adapted for use in PSDs and the summary charts in program performance metrics] is intended to be used as an early warning report of both potential and actual problems.

    • In this regard, the system depends on the Program Manager exercising sound judgment in assessing the program’s status.

    • Rating an indicator as "on-track," solely because the APB might be "on-track" is counterproductive and leads to downstream problems.

  • If the problem is a potential one, the PM should clearly note this fact, so there is no doubt that this is an advisory and that the situation is being properly managed.

  • Early reporting of potential problems and that corrective action plans are underway is essential.


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Color Coded Rating System For DAES, and

  • DoN PSDs and Program Performance Metrics Summary Charts

  • GREEN:

  • On-Track: All aspects of the program are progressing satisfactorily. Some minor problem(s) may exist, but solutions are available. Costs are expected not to exceed approved funding levels and not to exceed contract target costs by more than 5%.

  • YELLOW:

  • Potential or Actual Problem: Some event, action or delay has occurred that impairs progress against major objectives in one or more segments of the program.

  • In the case of a potential risk to a major program objective or acquisition program baseline, the Program Manager should state this distinction in Section 3 (Program Manager’s Comments). Early reporting is encouraged.

  • RED:

  • Major Weakness (Red): Some event, action, or delay has occurred that seriously impedes successful accomplishment of one or more major program objectives.

  • ADVISORY (GREEN, YELLOW OR RED):

  • Advisory indicates the program is either assessed to be between ratings, or is moving from one rating to another, thus this provides advance notification of shifting status.

Mixing performance and risk

Mixing performance and risk


Program risk assessment

  • A brief description of Issue # 2 and rationale for its rating.

  • Approach to remedy/mitigation

  • A brief description of Issue # 3 and rationale for its rating.

  • Approach to remedy/mitigation

  • A brief description of Issue # 1 and rationale for its rating.

  • Approach to remedy/mitigation

  • A brief description of Issue # 5 and rationale for its rating.

  • Approach to remedy/mitigation

  • A brief description of Issue # 4 and rationale for its rating.

  • Approach to remedy/mitigation

  • A brief description of Issue # 6 and rationale for its rating.

  • Approach to remedy/mitigation

5

High

4

Medium

3

Likelihood

2

Risk Advisory Board

Mr. X, PMA xxx

Ms. Y, XYZ Corp

Ms. Z, DASN YY

etc

1

Low

1

3

4

5

2

Consequence

This is one of the most commonly used charts for depicting risk.

Program Risk Assessment

Shows the plan to manage risk

Shows the risks


Logistics risk assessment

PEOXXX

Low Risk

Medium Risk

High Risk

Logistics Areas (examples)

5

5

4

: Overall Assessment

1: Training

2: Support Equipment

3: Publications

4: Facilities

5: Maintenance Concept

6: Supply Support

7: MTBF

4

7

Likelihood

3

6

2

3

1

1

2

1

3

4

5

2

Consequence

RISK #5

Brief description of Issue and rationale for its rating.

Approach to remedy/mitigation.

Risk mitigation funding.

RISK # 4

Brief description of Issue and rationale for its rating.

Approach to remedy/mitigation.

Risk mitigation funding.

RISK # 6

Brief description of Issue and rationale for its rating.

Approach to remedy/mitigation.

Risk mitigation funding.

ProgramAcronymACAT XX

Logistics Risk Assessment

CAPT J. Doe, PMS-499

Date of Review: dd mmm yy


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

Low Risk

Medium Risk

High Risk

5

5

4

4

7

3

6

Likelihood 

2

3

1

1

2

1

3

4

5

2

Consequence 

Definitions Of

“Likelihood” And “Consequence” When Constructing This Chart

Risk Assessment: Address each issue which might

affect the success of the program.

Likelihood

  • Negligible - One can reasonably assume no occurrence (<10%)

  • Unlikely - Occurrence possible but less than likely (10-40%)

  • Likely - significant chance of occurrence (40-65%)

  • Highly Probable - Very high chances of occurrence (65-90%)

  • Near Certainty - Assume and anticipate occurrence (>90%)

    Consequence

  • Marginal - Remedy will cause disruption to the program

  • Significant - Shorts a significant mission need

  • Serious - Shorts a critical mission need but expect no breech

  • Very Serious - Potentially fails a KPP or OPEVAL.

  • Catastrophic - Jeopardizes an exit criterion of current


Budget

G

Solvency

Programmed (%Obligated)

Activity FY99 (%) FY00 (%) FY 01 FY 02 FY 03 FY 04 FY05 FY06

Display each significant task

Budget

Program Manager/Office______________ Date of Review_________

This addresses whether there enough money budgeted/planned over the course of the life of the program to accomplish the program goals and objectives


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

G

Program Solvency

This “Solvency” addresses how well the PM is able to cover the cost of the PROGRAM within funds already allocated to the program.

The color ratings used generally follow these themes:

GREEN-can handle within existing resources without descoping or affecting KPPs

YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and will request changing KPPs

RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds and will request Congressional approval be sought for a current year reprogramming; in short, program requires outside help.


October 2000 guidance example master chart contract performance for give short contract title

ProgramAcronymACAT XX

Y

October 2000 Guidance – Example Master ChartContract Performance for [give short contract title]

PEO and Program Manager/PMS-XXX

Briefed: FEB 02

Solvency

Nxxxxx-YY-Cxxxx

Contractor Name [Prime or Significant Sub]

[TCPIEAC = 0.76]

CV = $2.0 M

SV = $2.9 M

04/98

04/00

01/00

04/02

08/02

Total Calendar Schedule

PM’s EAC

$M

100%

108%

0 %

50%

42%

122%

$110

EAC

1.18

Ahead of Schedule and Underspent

$100

TAB

111%

Behind Schedule and Underspent

1.14

100%

$90

BAC

04/97

07/97

1.10

10/97

04/98

1.06

PM’s Projected

Performance at Completion

for CPI and Duration.

01/98

07/98

10/98

1/99

1.02

01/00

04/99

CPI

56%

$50

02/00

07/99

0.98

04/00

ACWP

03/00

0.960

0.94

EV % Spent

10/99

01/00

0.90

Total Spent

0.86

0.940

Behind Schedule and Overspent

Ahead of Schedule and Overspent

0%

0

0.82

KTR’s EAC:104M

0.82

0.86

0.90

0.94

0.98

1.02

1.06

1.10

1.14

1.18

1.18

SPI

Date of Last Rebaselining: JAN00Number of Rebaselinings: 1Date of Next Rebaselining: MMM YY

Date of Last Award Fee: MMM YYDate of Next Award Fee: MMM YY

YYMMDD


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

Y

Contract Solvency

This “Solvency” addresses how well the PM is able to cover the cost of the CONTRACT within funds already allocated to the program.

The color ratings used generally follow these themes:

GREEN-can handle within existing resources without descoping or affecting KPPs

YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and changing KPPS will be considered.

RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds. the program office has no way to resolve the funding shortfall and needs assistance from the sponsor, DASN, ASN, PEO or others.


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

This Is Not a Metrics Chart, It Is an Illustration of Aggressive Bidding and A Changing Contract

G

Solvency

OCT 00

Rebaselined/Renegotiated Contract - JAN 01

PM’s Est Cost

$115

Contractor’s Est Cost $100

Y

Contract Target Cost

Solvency

$112

PM Allocation for Contract

$100

Y

Solvency

Original Contract - APR 00

[Same status at OCT 00]

PM’s Est Cost $100

Contractor’s Est Cost $85

Contract Target Cost

PM Allocation for Contract

$68

$112

Contract performance is poor, but yields little budget risk

Contract performance is poor, but yields little budget risk

Contract performance is poor, but yields little budget risk

Current Status APR 01

PM’S Est Cost

Contractor’s Est Cost $109

$130

Contract Target Cost

PM Allocation for Contract

$112

$100


Cpars ipars award fee matrix

PEOXXX

ProgramAcronymACAT XX

CPARS/IPARS/Award Fee Matrix

CAPT J. Doe, PMS-499

Date of Review: ddmmmyy

At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.

At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.

At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

For Reference: Comparison of Color Ratings

(We have attempted to normalize these as far as we can)

Color

Rating

* PDUSD(A&T) CPARS/IPARS

24 AUG 1999

In DoN CPARS/IPARS Guidance

Applies to Numerous Indicators

DAES

Not for release to contractors. Brings up issues within Defense.

Applies to Numerous Indicators

* ACAT I & II Program Contracts (EVM)

5 OCT 2000

Applies to Cost and Schedule

Exceptional

Exceed many requirements; resolve all problems.

Blue

NOT USED

CPI and SPI > 1.15 and exceed many requirements, resolve all problems.

Purple

Very Good - Exceeds some requirements; some minor problems, which were corrected.

NOT USED

CPI and SPI > 1.1

Green

Satisfactory

Meets requirements; some minor problems, which were corrected

“On-Track.” [CPI >0.95]

Program progressing satisfactorily. Contract costs not expected to exceed contract target by > 5%, thus not an issue.

CPI and SPI > 1.0

Yellow

Marginal

Does not meet some requirements; serious problem with no corrective action identified.

CPI and SPI > 0.90

Potential or Actual Problem

Some event has occurred that impairs progress against major objectives of the program.

Red

Unsatisfactory

Does not meet most requirements and timely recovery unlikely.

Major Weakness

Some event seriously delaying major program objective; breach of program baseline or unit cost.

CPI and SPI < 0.90


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Risk Assessments In

  • Integrated Baseline Reviews

  • For Contracts

  • Managed At The Program Manager Level

  • No Consistent Form And Format For The IBRs.

  • Defense/Industry Developing Joint IBR Guide

  • Joint Strike Fighter IBR Offers A Good Example to Emulate


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

Extracts of Templates Used In

The Integrated Baseline Review

For The

Joint Strike Fighter


Sample wbs 1346 fuel sys interview summary

WBS 1300

(Sample)WBS 1346 Fuel Sys Interview Summary

Overall Assessment

  • The Team believes that this cost account can be successfully accomplished within cost and schedule at acceptable risk

Dialogue Highlights

  • Concerns

    • Staffing availability

    • Hardware fabrication cycle with new vendors (different than Phase ll)

    • Communication between IPT’s - late

      data / data updates could cause redos

  • Risks

    • Resource availability/timing

    • Design maturity / hardware schedule

  • Strengths

    • good discussion; CAM well-prepared

    • comprehensive plan

      • linkages well defined

  • Weakness

    • Resource availability


Ibr discussion evaluation summary

IBR Discussion Evaluation Summary

(Level 3 IPT)


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Our Guidance in Defense for Risk

  • Systems Engineering risk management calls for risk:

    • Avoidance

    • Control

    • Assumption22

    • Transfer

  • None of these imply an opportunity.

  • All of these focus on possible negative future occurrences.

  • The basic objective of risk management in systems engineering is to reduce all risk to an acceptable level.


Larry szutenbach oasn rd a pp r szutenbach larry hq navy mil

  • Industry Takes Risk a Step Farther

  • Industry sees the potential

  • Compensaton of Industry managers is often tied to the risk of success, which is tracked in scorecards.

  • In DASN(PP&R) we are reviewing industry use of scorecards and seeing if we can reasonably attach our metrics to a scorecard appropriate for the government and reaching for the risk of success.

success.

of

risk


  • Login