27 ismor
1 / 21

27 ISMOR - PowerPoint PPT Presentation

  • Uploaded on

Learning from Experience (LFE) - MCA, SWIFT & HAZOP. 27 ISMOR. Chris Tilney, Atkins Ltd 2 September 2010. Structure of Presentation. Multi Criteria Analysis (MCA) Lessons learned from decision support applications in defence

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '27 ISMOR' - yardan

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
27 ismor

Learning from Experience (LFE) - MCA, SWIFT & HAZOP


Chris Tilney, Atkins Ltd

2 September 2010

Structure of presentation
Structure of Presentation

Multi Criteria Analysis (MCA)

Lessons learned from decision support applications in defence

Structured What If Technique (SWIFT) & Hazard & Operability (HAZOP)

Brief introduction to the techniques

Lessons learned from recent nuclear safety applications

Potential non safety related applications in defence

Mca lfe 1
MCA – LFE (1)

Context for lessons learned

Primarily focused on application of MCA to UK MoD procurement competitions i.e. down select decisions on Pre Qualification Questionnaire (PQQ) responses and Tenders

Personal opinions of presenter, not necessarily shared by other Atkins staff or MoD personnel!

Mca lfe 2
MCA – LFE (2)

Structure of Assessment Hierarchies

Derivation of Weighting Factors

Preparing the Evaluation Team

Scoring Criteria


Audit Trail/Toolset

Presentation of Results

Mca lfe 3
MCA – LFE (3)

Structure of Assessment Hierarchies

Tower 1

Tower 2

Tower 3

  • Senior Decision Makers/Scrutiny favour:

    • Towers not weighted relative to each other

    • No single, overall Figure Of Merit (FOM) score

  • Small set of criteria (15 – 20) works best

Merits of solution offered

Capability to deliver solution

Commercial Aspects


System Requirements Document (SRD) Compliance





Delivery Schedule


Terms & Conditions (T&Cs)

Mca lfe 4
MCA – LFE (4)

Derivation of Weighting Factors

Pairwise comparison works best. Three ‘rounds’ to derive weights:

1st round: initial pairwise scores followed by explanation of outliers

2nd round: score again then calculate weights based on consensus scores

3rd round: review weights, adjust if necessary (with rationale)

Group dynamics, watch out for/control:

Dominant personalities

Deferral to rank (if military Subject Matter Experts (SMEs))

Non assertive gurus

Preparing the Evaluation Team

Evaluation Guide: helps enforce coherency, transparency, robustness of scoring process

Kick-off group brief: ditto

1-2-1s on toolset: reduce risk of failing to keep to evaluation schedule

Mca lfe 5
MCA – LFE (5)

Scoring Criteria

Scoring scale


0 to 10 point scale

3 reference points: 10, 7 & 4 with guidance for assessors

No ‘extra’ marks for exceeding requirement

Confidence adjusted scores (demonstrable/evidence based capability)

Minimum of 2 Assessors: Lead & Shadow(s)

Moderators: resolve disagreements, holistic view/input to assessors


Automate at lower levels of hierarchy

“Man-in-loop” at higher levels of hierarchy

Trigger points/capability thresholds for all criteria that are scored

Mca lfe 6
MCA – LFE (6)

Audit Trail/Toolset - AWARD

Simultaneous, distributed input (over Restricted LAN Interconnect (RLI))

Strong functionality for:

Capturing assessor rationales, issues & concerns

Monitoring progress of evaluation


Presentation of Results

KISS principle to avoid information overload for Assessment Panel Members

Executive Summaries of what is being offered

Bar graphs for Towers 1 & 2

‘Horse blanket’ for Tower 3 T&Cs

Risk adjusted Whole Life Cost (WLC) plots for price

Combined Operational Effectiveness & Investment Appraisal (COEIA) plots for Value for Money (VfM)

Slides on major concerns, issues and ‘red flags’

Mca lfe 7
MCA – LFE (7)

Commercial Issues

Central finding of recent court judgements (e.g. case “Lettings International Ltd v London Borough of Newham”):

Any aspect of the Authority’s requirements or the way that it will evaluate tenders that could affect the way a bidder prepares its bid must be fully disclosed before tenders are returned.

Implications may include, for example:

Assessment hierarchies – fully weighted, automatic roll up of weighted scores

Disclosure of assessor scoring guidance to tenderers

Predefined, prescriptive COEIA methodology (i.e. can’t be tailored after tenders received). Dstl study is investigating this issue.

Are we entering dangerous waters?

i.e. OA/Decision Support Analysis no longer informing decision makers but effectively making the decision (by option scores and option ranking) by slavish adherence to a detailed, predefined process

Mca lfe 8
MCA – LFE (8)

Overall LFE findings

Well understood/accepted by Senior Decision Makers

Works well with careful preparation and checks & balances on aggregation


Can be very resource intensive

Commercial issues may prevent/limit future application of some of the LFE and undermine central tenet of OA to inform decision makers

Swift intro to technique 1
SWIFT – Intro to Technique (1) (HAZOP) – Intro to techniques

Structured brainstorming method for analysing process or system

Usually applied to systems or processes not deemed to be safety critical but which do have safety related failure modes


Select system, subsystem or process

Chair poses pre-planned ‘What if’ questions, identified via

Task analysis

Basis of Design

Generic Checklists

Process Description

Standards, regulations & guidelines

Past incidents & accidents

Multi disciplinary team (design, operation & maintenance) of SMEs answer ‘What if’ questions

Results captured in log sheet by scribe

Swift intro to technique 2
SWIFT – Intro to Technique (2) (HAZOP) – Intro to techniques

Example SWIFT Log sheet

Step 2

Step 1

Hazop intro to technique 1
HAZOP – Intro to Technique (1) (HAZOP) – Intro to techniques

Considers the operation (operability) of a system or subsystem

Systematically identify deviations from the design intent which could lead to Hazards or operability problems

Multi-disciplinary SME team activity

Makes recommendations which could influence the design


Use Guide Word / attribute combination to suggest possible deviation(s)

Discuss and agree all credible causes

Identify all consequences, identifying Hazards

List existing safeguards

Discuss and agree further actions


Repeat until complete

Hazop intro to technique 2
HAZOP – Intro to Technique (2) (HAZOP) – Intro to techniques

Example HAZOP Log sheet

SWIFT & HAZOP - LFE (HAZOP) – Intro to techniques

Swift hazop lfe 1
SWIFT & HAZOP – LFE (1) (HAZOP) – Intro to techniques


Key to effective workshop

Don’t circulate ‘what if’ Qs in advance to help avoid preformed views

Ensure reference material to-hand during workshop e.g. large-scale design drawings


Number: No more than 8, including Chair & Scribe

Expertise: Full coverage required (tension with number!)



Independent of the project

Ideally knows attendees to draw them out, but not too well (bias/favouritism)

Have sufficient technical knowledge to understand the discussion and be able to apply judgement when directing the study, e.g. distinguishing between superficial/trivial issues and important issues

Scribe: Understands domain and technical jargon

Swift hazop lfe 2
SWIFT & HAZOP – LFE (2) (HAZOP) – Intro to techniques



Don’t have to have finished design i.e. can apply to concepts

Quick and easy to implement (cf HAZOP)


Mature design needed

Exhaustive analysis, completeness of analysis

Time consuming/resource intensive

Swift hazop potential non safety applications
SWIFT & HAZOP – Potential Non Safety Applications (HAZOP) – Intro to techniques

Explore/define CONEMP

Develop OA scenarios & vignettes

Explore and assess merits of business change programme

General risk identification & analysis