general guidance on conduct of technical reviews n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS PowerPoint Presentation
Download Presentation
GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS

Loading in 2 Seconds...

play fullscreen
1 / 73

GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS - PowerPoint PPT Presentation


  • 138 Views
  • Uploaded on

GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS. Type and Number. Selection should be based on the complexity of the program and the phase of development (modification). Participants.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS' - isabel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
general guidance on conduct of technical reviews
GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS

Type and Number

Selection should be based on the complexity of the program and the phase of development (modification).

Participants

Include tasking and performing activity personnel responsible for the area or item being reviewed, key representatives for lower level items, and other personnel who have a stake in the specific objectives of the review.

Objectives

1. Demonstrate progress 3. Verify the expected maturity

2. Ensure issues have been resolved 4. Confirm that risks are acceptable

focus in major reviews
FOCUS IN MAJOR REVIEWS

In General

  • Confirm that issues addressed during previous reviews have been satisfactorily resolved.
  • Only areas requiring close scrutiny should be addressed in detail. Proper integration and management of interim, subsystem, and functional reviews (as planned in SEMP) should make a detailed total evaluation unnecessary.
  • Opportunity to modify program emphasis for the next phase based on risk levels.
types of reviews
TYPES OF REVIEWS

Major: Demonstrate that risk levels are acceptable and provide an opportunity to modify program emphasis for next phase/effort. (Examples: ASR, SRR, SFR, PDR, CDR, SVR, PCA)

Functional: System wide review conducted by involved disciplines to address progress and issues associated with one functional area. (Examples: Development, Training, Support, Verification, Manufacturing, Disposal, etc.)

Subsystem: Multidisciplinary reviews to assess progress in defining and satisfying subsystem requirements.

Interim: Reviews held between major reviews that cover spectrum of system, segments, and configuration items.

integrated technical reviews cont
Integrated Technical Reviews (Cont.)

Major Technical Reviews include:

- Alternate System Review (ASR): The ASR forms the basis for determining which system concepts should be continued as candidate development programs. It provides the data for recommendations to eliminate potential systems concepts.

- System Requirements Review (SRR): The SRR forms the basis early in System Development and Demonstration whether customer requirements are understood. It should confirm the progress and direction of technology verifications and demonstrations and convergence on viable system requirements

integrated technical reviews cont1
Integrated Technical Reviews (Cont.)

Major Technical Reviews include:

- System Functional Review (SFR): The SFR (formally called the SDR) forms the basis to establish and verify an appropriate set of functional and performance requirements for the complete system.

- Preliminary Design Review (PDR): The PDR forms the basis for determining whether a preliminary physical architecture and design approach for configuration item or aggregation of configurations (including software) is ready to start detailed design. A configuration item development spec must be complete and ready for use in detailed design

slide6

Integrated Technical Reviews (Cont.)

Major Technical Reviews include:

- Critical Design Review (CDR): The CDR confirms the detailed design for each configuration item (including software), each aggregation of configuration items, and that requirements are satisfied.

- System Verification Review (SVR): The SVR demonstrates the total system (people, products, and processes) satisfies functional and allocated configuration documentation requirements and confirms readiness for production, support, training, deployment, operations, an disposal. The Production Readiness Review (PRR) is a part of the SVR.

customary system level review accomplishments
Customary System Level Review Accomplishments

SRR - Review system functional and performance requirements.

SDR/SFR - Review allocation of system requirements to

configuration item (CI) aggregates. Assess maturity of system spec. and CI aggregates.

PDR - Review design progress for each CI and

aggregate of CIs:

1. Review predicted performance

2. Review physical and functional interfaces

3. Verify integration of specialty eng. req.

4. Evaluate risk resolution

customary system level review accomplishments cont
Customary System Level Review Accomplishments (Cont.)

CDR - Review maturity of design for each CI and aggregate CIs:

1. Verify that functional and performance requirements of CI specs are satisfied.

2. Verify that engineering specialty requirements have been satisfied.

3. Verify compatibility between CIs and hardware, software, facilities and personnel.

4. Review preliminary product specs.

5. Assess producibility risks.

customary system level review accomplishments cont1
Customary System Level Review Accomplishments (Cont.)
  • SVR - Confirms readiness to enter full rate production:
  • 1. Verify work on all CIs – all functions & performance levels met.
  • FCA conducted for CIs.

See page 107 of Sys. Eng. Fundamentals for additional accomplishments.

systems engineering master schedule sems
Systems Engineering Master Schedule (SEMS)
  • Purpose:
    • Contractual Agreement Between Contractor and Customer on Critical Tasks Required for Completing Each Major Program Milestone and Defining the Criteria for Successful Completion of Those Tasks.
role of sems in acquisition planning
Role of SEMS in Acquisition Planning

The SEMS Outlines the Integration of All Major Tasks Necessary to Design, Develop, Test, Produce, Deploy and Support the System.

Reliability

Producibility

Schedule

Performance

Cost

Support

Training

Risk

Management

System

Balanced Process/Product

to Satisfy User Needs

Prime Mission

Equipment

sems key tasks
SEMS Key Tasks
  • Typical Tasks
    • Develop Test Plans
    • Develop Software Plans
  • Tasks Define Interim Steps to Developing and Producing System
  • Tasks Should Include Verification Criteria
sems accomplishment guidelines
SEMS Accomplishment Guidelines
  • Typical Accomplishment for Sample Tasks
    • Test Plan Complete
    • Software Development Plans Complete
  • Accomplishment Should Be Easily Measurable
  • Example of Difficult to Measure Accomplishment
    • Test Plan 85% Complete
sems sample accomplishment
SEMS Sample Accomplishment
  • Milestone 3 - Test Readiness Review (TRR)
  • Accomplishments
  • 3.1 Complete Software Test Procedures
  • 3.2 Complete System Integration Test Procedures
          • 3.3 Complete All Major Subsystem TRRs
          • 3.4 Complete All Action Items from Milestone 2
slide17

Section L - Instructions, Conditions, and Notices to Offerors

Systems Engineering Master Schedule

(SEMS)

volume 3 attachment 1

section l instructions conditions and notices to offerors cont
Section L - Instructions, Conditions, and Notices to Offerors (Cont.)

3.0 SEMS PROGRAM EVENTS

Listed below are the Major Program Events and their scheduled time of achievement. Each event shall be considered achieved upon successful demonstration of criteria accomplishments. Criteria for each event is defined in Section 4.0.

Event Number Event Title Event Date

1. System Hardware Preliminary 5 MACA

Design Review(PDR)

2. System Software PDR 7 MACA

3. ILS Management Team (ILSMT) 9 MACA

Meeting

4. System Hardware Critical Design 11 MACA

Review (CDR)

5. System Software CDR 13 MACA

6. System Test Readiness Review 22 MACA

7. Ground Durability/Qualification 24 MACA

Test Release

section l instructions conditions and notices to offerors cont1
Section L - Instructions, Conditions, and Notices to Offerors (Cont.)

Event Number Event Title Event Date

8. (Reserved)

9. RF-4C #1 Development Flight 29 MACA

Test Release

10. RF-4C DT&E/IOT&E start 29 MACA

11. UARS #1 Development Flight 29 MACA

Test Release

12. UARS DT&E/IOT&E start 29 MACA

13. F/A-18 DT&E/IOT&E Start 29 MACA

14. FSD Maintainability Demonstration 34 MACA

15. Production Readiness Review 35 MACA

16. System Function Configuration 36 MACA

Audit (FCA)

17. RF-4C DT&E/IOT&E Complete 38 MACA

18. UARS DT&E/IOT&E Complete 38 MACA

section l instructions conditions and notices to offerors cont2
Section L - Instructions, Conditions, and Notices to Offerors (Cont.)

Event Number Event Title Event Date

19. F/A-18 DT&E/IOT&E Complete 38 MACA

20. Formal Qualification Review (FQR) 38 MACA

21. Milestone IIIA 39 MACA

22. System Physical Configuration 57 MACA

Audit (PCA)

23. LRIF Maintainability Evaluation 57 MACA

24. Milestone IIIB 66 MACA

25. F-14D/TARPS DT&E/IOT&E 72 MACA

Start

26. F-14D/TARPS DT&E/IOT&E 81 MACA

Complete

27. Initial Operational Capability See

(IOC) Appendix

28. Delivery Per Table 4-2 As Required

*MACA: Months After Contract Award

exercise 2
EXERCISE #2

Exercise: Develop the portion of a Systems Engineering Master Schedule (SEMS) for the AJS design and development tasks. Assume the system level schedule for Peace Whey includes a SRR, SFR, PDR, CDR, and PRR. Your SEMS should identify a list of AJS program reviews with accomplishment criteria.

  • Part 1: Referring to the system level Peace Whey reviews, identify and list the major reviews which need to be supported by inputs from the AJS design and development tasks. Based upon this list, determine a set or reviews for the AJS development effort required to support the Peace Whey program.
  • Part 2: Identify the accomplishment criteria for one AJS review.
  • You will have 20 minutes to complete this exercise.
imp sems ims seds relationships
IMP/SEMS & IMS/SEDS Relationships

Integrated Master Plan

(IMP)

Integrated Master Schdule

(IMS)

System

Engineering

Detailed

Schedule

(SEDS)

System

Engineering

Master

Schedule

(SEMS)

Program

Office

Contracts

Security

Program

Office

Contracts

Security

slide31

A

NN

NN

NN

NN

Product

Event No.

Accomplishment

Criteria

IMS Task

IMP/IMS Development Steps

1. Define Events, Accomplishments (RFP “Shalls”), and Criteria

SYSTEM PRODUCT

HIERARCHY

SPEC TREE

2. Define a Product Oriented

Management Structure

(SPEC Tree, WBS, and

IPT Organization)

A

WBS

IPT

SPEC TREE

B

C

D

E

WBS

IPT

SPEC TREE

WBS

J

K

L

M

IPT

SPEC TREE

WBS

F

G

H

I

IPT

3. Build a Top Down, Layered IMP/IMS with Activity Numbers as Shown Below

integrated master plan imp and integrated master schedule ims terms
Integrated Master Plan (IMP) and Integrated Master Schedule (IMS) Terms
  • Product - Hardware, Software, Facilities, Data, or Materials
  • Event - Decision Point at End of Major Project Activity (EX. - CDR)
  • Accomplishment - Desired Result at Specified Events
  • Criteria - Measure of Meeting Accomplishment
  • Task - Specific Activity to Complete a Criteria
slide33

Top Level Task Relationships

Product

Requirements

Suppliers

Development

• Primary

Production

• Derived

Product Definition

• Inventory Control

Delivered

• Parts Production

Product

• Assembly

Hardware

Design

Parts

Assemblies

Requirements

Software

For Design

Design

Component

& System

Weapon

Testing

Software

System

Implementation

Integration

Requirements

FEEDBACK

& Testing

For

• Requirements

Integrated Test

• Design

slide34

Network Chart Example

(D) 6 Weeks

(A) 4 Weeks

(E) 3 Weeks

(H) 2 Weeks

(B) 2 Weeks

(F) 8 Weeks

(I) 4 Weeks

(C) 1 Week

(G) 4 Weeks

(K) 8 Weeks

(J) 5 Weeks

Network Layout

C

S

A

D

E

H

B

F

I

C

G

J

K

14

2

4

6

8

12

10

Conversion to Milestone Chart

integrated master schedule ims and network schedules
Integrated Master Schedule(IMS) and Network Schedules
  • IMS Show Dates for Major Tasks and Milestones
    • - Usually Presented as Gantt (Bar) Chart
    • - Shows Tasks Corresponding to Entry/Exit Criteria
    • - Must Include All IMP Milestones
  • Network Schedules Are Essential to the IMS
    • - Helps Determine if IMS Dates and IMP Events Are Achievable
    • - Critical-Path Method (CPM) Schedule Software is Essential
    • - Essentials: Tasks, Relationships, Durations, Resources
imp ims summary
IMP/IMS Summary
  • Negotiated IMP Will Be Place on Contract, Changes Will Require a Contract Change Proposal
  • IMS Shall Support the IMP. IMS Updates Shall Be Approved Prior to Implementation of Changes. IMS is Time-Based
  • Customer May Provide Government IMP in RFP. Government IMP Should Provide Detailed Information for Next Acquisition Phase to Identify Specific Events, Accomplishments and Criteria Necessary to Satisfy Planned and Required Technical Exit Criteria
  • IMP Is a Key Control Element of SE Process. IMP Can Be Used as the Basis for Quantitative Requirements for Award Fees
exercise 3
Exercise # 3
  • Exercise:
      • Referring to the SEMS from Exercise #2, develop a System Engineering Detailed Schedule (SEDS) for the AJS design and development task. The AJS Safety of Flight unit must be available in month 26 and first production aircraft in month 36. Be prepared to report your results to the class. You will have 20 minutes for this exercise.

Hint – Use building block tasks on next page.

slide38

Electronic

Warfare

slide40

Progressive Subsystem

Informal Subsystem

Formal Subsystem

Limited Formality

Formal

Issues

Roll-Up

integrated technical reviews cont2
Integrated Technical Reviews (Cont.)

Technical reviews are a series of systems engineering activities by which the technical progress of a program is demonstrated relative to its technical or contractual requirements. They are conducted at logical transition points in the development effort to reduce risk by assessing progress against SEMS requirements and identifying and correcting problems/issues resulting from the work completed before the program is disrupted or delayed. They provide a method for the contractor and government to determine if the development of a system and/or configuration item and its documentation have met contract requirements.

planning guidance from dodd 5000 1 defense acquisition
Planning Guidance from DoDD 5000.1(Defense Acquisition)

Part 2.B.3 Acquisition Strategies, Exit Criteria, and Risk

Management

“Event driven acquisition strategies and program plans must be based on rigorous, objective assessments of a program’s status and the plans for managing risk during the next phase and the remainder of the program. The acquisition strategy and associated contracting activities must explicitly link milestone decision reviews to events and demonstrated accomplishments in development, testing and initial production. The acquisition strategy must reflect the interrelationships and schedule of acquisition phases and events based on logical sequence of demonstrated accomplishments not on fiscal or calendar expediency.”

integrated technical reviews
Integrated Technical Reviews
  • The government assesses the status, maturity, risk, and verification status by conducting reviews that will:
    • Be adequately planned prior to the initiation technical work
    • Have event-driven entry and exit criteria with defined accomplishments and success metrics via the SEMS.
    • Be conducted incrementally (as appropriately) to measure progress towards achieving major accomplishments and obviate unexpected issues resulting from incomplete or inconclusive development activities.
    • Be conducted by multidisciplinary teams to ensure that all the system's functions are addressed and all system elements are integrated and balanced across the system.
risk assessment and management dod 5000 1
Risk Assessment and Management (DoD 5000.1)

“Program Managers and other acquisition managers shall continually assess program risks. Risks must be well understood, and risk management approaches developed, before decision authorities can authorize a program to proceed into the next phase of the acquisition process. To assess and manage risk Program Managers and other acquisition managers shall use a variety of techniques, including technology demonstrations, prototyping, and test and evaluation. Risk management encompasses identification, mitigation, and continuous tracking, and control procedures that feed back through the program assessment process to decision authorities. To ensure an equitable and sensible allocation of risk between government and industry, Program Managers and other acquisition managers shall develop a contracting approach appropriate to the type of system being acquired.”

cost schedule and performance risk management dod 5000 2 r
Cost, Schedule, and Performance Risk Management (DoD 5000.2-R)

“The Program Manager shall establish a risk management program for each acquisition program to identify and control performance, cost, and schedule risks. The risk management program shall identify and track risk drivers, define risk abatement plans, and perform periodic assessments to determine how risks have changed. Risk reduction measures shall be included in cost-performance tradeoffs, where applicable. The risk management program shall plan for back-ups in risk areas and identify design requirements where performance increase is small relative to cost, schedule, and performance risk. The acquisition strategy shall include identification of the risk areas of the program and a discussion of how the Program Manager intends to manage those risks.”

risk management structure dod risk management study
Risk Management Structure(DoD Risk Management Study)

Risk Management

Risk

Handling

Risk

Monitoring

Risk Planning

Risk

Assessment

Risk

Documentation &

Communications

Risk

Identification

Risk

Analysis

definitions
Definitions
  • Risk
  • - A measure of likelihood to achieve objectives
  • - Two components (probability and consequences)
  • Risk Management
  • - Act or practice of controlling risk
  • + Identifying and tracking risk drivers
  • + Defining risk mitigation plans
  • + Performing periodic risk assessments
risk planning
Risk Planning
  • Process has two segments
    • Implementing a comprehensive and active strategy to continuously identify, mitigate and track program risks
      • Who does it
      • What do they do
      • When do they do it
      • How risk is shared
    • Documenting risk elements of program activities

How do I get

there from here?

risk assessment
Risk Assessment
  • Process of identifying and analyzing program risks to increase the chances of meeting performance, schedule, and cost objectives
  • Two segments
    • Risk Identification
    • Risk Analysis
risk identification
Risk Identification
  • Process of specifying, describing and documenting program risks and their sensitivities to other risks
    • Internal
    • External
risk analysis
Risk Analysis
  • Process of evaluating program risks for their impacts to performance, cost, and schedule objectives
  • Process includes assessing each risk’s:
    • Probability of occurrence, and
    • Consequences of failure to mitigate the risk
risk handling
Risk Handling
  • Process that identifies, evaluates, selects, and implements risk handling options
    • to set risk at acceptable levels
    • give program constraints
  • Typical risk handling strategies can include:
    • replan to eliminate the identified risk
    • avoid risk by changing requirements
    • transfer the risk
    • control the risk through active steps
    • assume the risk without special efforts
slide54

Risk Monitoring

  • Process that systematically tracks and evaluates the performance of risk mitigation actions - against established metrics throughout the acquisition* process, and - develops further risk handling options as appropriate* Acquisition includes any procurement from government or contractor sources within all phases from early research through logistics, operations, support, and disposal
risk management process

Program

Requirements

Evaluate Risk

Handling

Options

Analyze

Impacts

  • Performance
  • Cost
  • Schedule
  • Avoid
  • Information Gathering
  • Transfer
  • Assume
  • Control

Assess

Risk

  • Identify
  • Analyze
  • Quantify Risk

Evaluate

Subcontractor

Risks

Risk Management Process

Manage Risk

  • Review Indicators
  • Abatement Actions
assess risks
Assess Risks

Evaluate Risk

Handling Options

Establish Cost,

Schedule, &

Performance Impacts

Program

Requirements

Assess

Risks

Manage

Risks

Evaluate

Subcontractor

Risks

Identify

Risk

Areas

Create Risk

Management

Tools

Establish

Approach

Develop

Team

Analyze

Risk

slide57

Risk Identification

The WBS is normally used to organize and ensure completeness of the risk identification effort.

Identification is generally performed at the 3rd or 4th level of the WBS.

risk categories and consequences
Risk Categories and Consequences
  • Risk Categories
    • Requirements - Are the Necessary Requirements ( Operational or Design) Fully Defined? Is the Basis for the Requirements Stable (e.g. No Expected Threat Change)
    • Technology - Is the Technology Available Proven in Previous Use?
    • Engineering - How Much New Design is Needed to Achieve Requirements?
    • Manufacturing - Are the Required Manufacturing Processes, Facilities, and Sources of Materials Known and Available?
    • Support - Are the Required Support Resources Defined and Available?
    • Management - Are the Processes, Resources, and Experience Available to Successfully Perform this Program?
  • Risk Consequences
    • Performance - Can the Item Meet Its Requirements (Operational, Support and Manufacturing)?
    • Cost - Can the Item Be Developed and Operated within the Funding Allocated to It?
    • Schedule - Can the Item Be Developed and Deployed with the Time Allocated to It?
sample risk identification
Sample Risk Identification

WBS 113 - Guidance System

Issues: New Design – Uses new chipset from CHIPLEAP program. Some concern exists on producibility and thermal characteristics.

Assumptions: Megalith Corporation will design guidance system. They were a participant in the CHIPLEAP effort.

slide62

Organization of Risk Assessment

Additional risk categories such as customer satisfaction or customer expectation can be developed for identification purposes.

Risk assessment templates (shown on later charts) for subcategories can be developed and maintained by organizations.

funding
Funding

Product

Funding

Money Phasing

Design

Production

Facilities

Logistics

Management

Test

Manufacturing

Plan

Logistics

Support

Analysis

Manufacturing

Strategy

Design Ref

Mission Profile

Integrated

Test

Modernization

Quality MFG

Process

Design

Requirements

Personnel

Requirements

Trade Studies

Failure

Reporting

System

Factory

Improvements

Piece Part

Control

Manpower

And Personnel

Design Policy

Design Process

Data

Requirements

Design

Analysis

Productivity

Center

Parts and

Materials

Selection

Uniform

Test Report

Subcontractor

Control

Support and

Test Equipment

Technical

Risk Assessment

Software

Design

Defect Control

Software Test

Computer

Aided Design

Training Material

And Equipment

Tool Planning

Design Limit

Production

Breaks

Design For

Testing

Built –In Test

Life

Special Test

Equipment

Spares

Configuration

Control

Test Analyze

And Fix

Design

Reviews

Computer-Aided

MFG

Technical

Manuals

Design Release

Field Feedback

Manufacturing

Screening

Transition Plan

Ref. – Best Practices

slide70

Traditional Risk Analysis

1

Probability (Likelihood)

x

Probability is Assigned

0

Consequences are Estimated

Potential Degradation

Sys Reqt not Achieved

x

Performance

Element Increase < 10%

Element Increase > 50%

x

Consequences

Cost

Element Increase

System Increase > 40%

x

Schedule

slide71

Traditional Risk Analysis

1

High Risk – Severe disruption expected to performance, cost, and / or schedule even with risk mitigation plans in place.

Probability (Likelihood)

Moderate Risk –Expected disruption to performance, cost, and / or schedule can be overcome by implementing risk mitigation plans.

x

x

0

Consequence

Potential Degradation

Sys Reqt not Achieved

x

Performance

Low Risk – Little disruption expected to performance, cost, and / or schedule.

Element Increase > 50%

Element Increase > 50%

x

Cost

Element Increase

System Increase > 40%

x

Schedule

slide72

Weaknesses of Traditional Risk Analysis Process

  • Roll-up of risks characterized as high, moderate, or low at the 3rd or 4th WBS level are difficult. Example – Are 10 low risks and 1 high risk at WBS Level 4 elements expressed as low, moderate, or high at the parent WBS at Level 3?
  • Characterizing a risk as high, moderate, or low alerts the customer to the severity of the outcomes without giving insight into the likely capability of the delivered product.