Improving Confidence in the Assessment of
Download
1 / 44

Improving Confidence in the Assessment of System Performance in Differing Scenarios. - PowerPoint PPT Presentation


  • 47 Views
  • Uploaded on

Improving Confidence in the Assessment of System Performance in Differing Scenarios. T D Clayton. Cardinal Consultants. 1. Context 2. Scenario Dependency of Input Data 3. Choosing Scenarios to Assess 4. Modelling Widely Differing Scenarios 5. Example Study

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Improving Confidence in the Assessment of System Performance in Differing Scenarios.' - gerda


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Improving Confidence in the Assessment of

System Performance in Differing Scenarios.

T D Clayton

Cardinal Consultants


1. Context

2. Scenario Dependency of Input Data

3. Choosing Scenarios to Assess

4. Modelling Widely Differing Scenarios

5. Example Study

6. Summary and Conclusions


SYSTEM

EFFECTIVENESS

ASSESSMENT


SYSTEM

EFFECTIVENESS

ASSESSMENT

Warhead

Lethality


SYSTEM

EFFECTIVENESS

ASSESSMENT

Warhead

Lethality

Combat

modelling


Sensor

Performance

Guidance

System

Wargaming

SYSTEM

EFFECTIVENESS

ASSESSMENT

Warhead / Fuze

Performance

Combat

modelling

Operator

Performance

Tactical / Strategic

studies

Other

subsystems


Purpose of System Effectiveness Studies

  • Research / long term development objectives

  • Medium term procurement objectives

  • Design optimisation

  • Procurement decisions

  • Input to Operational / Tactical Studies


But, whatever the purpose, scenario assumptions are critical.

or, we should assume they are, unless proven otherwise.


Rule 1 critical.

Everything is scenario dependent.


Sensor critical.

Performance

Guidance

System

Wargaming

SYSTEM

EFFECTIVENESS

ASSESSMENT

Warhead / Fuze

Performance

Combat

modelling

Operator

Performance

Tactical / Strategic

studies

Other

subsystems


Sensor critical.

Performance

Guidance

System

Wargaming

SYSTEM

EFFECTIVENESS

ASSESSMENT

Warhead / Fuze

Performance

Combat

modelling

Operator

Performance

Tactical / Strategic

studies

Other

subsystems


P critical.k = 0.47


  • Presence of adjacent trees, or protective earthworks

  • Azimuth distribution

  • Elevation distribution

  • Relative value of M-kill, F-kill, P-kill, K-kill

  • Likelihood of multiple hits

  • Using an MFK value as a probability ?


The Multi-Disciplinary Problem critical.

Lethality

Expert

Combat

Modeller

Systems

Modeller


The critical.ManagementSolution

Establish roles and responsibilities for managing the interfaces between expert groups.


Responsibilities of the Interface Manager critical.

  • Understand methodologies and assumptions at all levels

  • Organise training / briefings to assist expert groups widen knowledge

  • Conduct studies to measure Scenario Dependencies of results

  • Maintain knowledge base of dependencies and “corrections”

  • Involvement in planning of studies, addressing assumptions

  • Involvement in reporting of studies, esp. assumptions


Study 1 critical.

Study 2

Study 3

MAIN DATABASE OF STUDY RESULTS

‘Offline’ analysis tools

Comparison & Analysis

Study planning and analysis

Data provided to

other studies

DATABASE OF

SCENARIO COMPENSATION FACTORS


Study 1 critical.

Study 2

Study 3

MAIN DATABASE OF STUDY RESULTS

Calculate SCF’s from new studies

DATABASE OF

SCENARIO COMPENSATION FACTORS

Assessment and comparison of SCF’s

Modified SCF’s


Rule 2 critical.

You will never assess the right scenarios.


Opposing critical. - Technology

Ground - Numbers

Equipment - Own Intell.

Posture & - Posture (Defensive, attacking)

Deployment - Deployment and detectablity

Air - Aircraft types

Capability - Level of technology

- Numbers

- Own Intell.

Anti-Air - Numbers of units

Capability - Capability

- Own Intell.

Maritime - Maritime involvement

- Capability

BLUE ROLE - Peace keeping, combat (defensive)

combat (hunt and kill)

Scenario Parameters

Climate - Temperature

- Precipitation

Ground - Vegetation

- Topology

- Roads

Geography - Geographic isolation

& Politics - Neighbouring countries

- Local cilvilian population

Opposing - Nuc., Chem., Bio.

Max. Cap. - Short range Long range

Opposing - Numbers

Troops - Capability




Rule 3 critical.

A combat model cannot address

widely differing scenarios.


Example Study critical.

Comparative assessment of two potential candidates

for a cannon system for light armoured vehicles.


ORIGINAL STUDY PLAN critical.

Input data

Engagement Model (developed for this study)

Combat model (existing)

3 Scenarios


REVIEW OF PROVIDED DATA critical.

1. When multiple hits are likely, SSKP may not be appropriate.

2. Lethality figures give no azimuth dependency.

3. No information on range dependency.

4. Data required for wider range of target types.

Lethality models re-run, in concert with Engagement model.


REVIEW OF EXISTING COMBAT MODEL critical.

1. Tends to choose tanks as preferred target type.

2. All targets are land vehicles.

3. Terrain in all 3 scenarios tends to give long engagement ranges.

4. No variations in met-vis or day/night > long ranges

5. Same Blue positions for both System A and System B.

6. Units are static when firing.


THE ALTERNATIVE APPROACH critical.

1. Use a range of methods, including Military Judgement, to derive intermediate data and distributions reflecting a wide range of scenarios.

  • relative frequencies of target types engaged

  • engagement range distributions

  • azimuth distributions

  • probability of kill per burst - function of range and target type


THE ALTERNATIVE APPROACH critical.

1. Use a range of methods, including Military Judgement, to derive intermediate data and distributions reflecting a wide range of scenarios.

2. Develop a simple tool to calculate specific Measures of Effectiveness from the input data and distributions.

MoE 1: Military Worth of kills per burst

MoE 2: Military Worth of kills per ammunition load


THE ALTERNATIVE APPROACH critical.

1. Use a range of methods, including Military Judgement, to derive intermediate data and distributions reflecting a wide range of scenarios.

2. Develop a simple tool to calculate specific Measures of Effectiveness from the input data and distributions.

  • quick to develop

  • quick to run

  • facilitates review and scrutiny of data

  • stores data and maintains audit trails


THE ALTERNATIVE APPROACH critical.

1. Use a range of methods, including Military Judgement, to derive intermediate data and distributions reflecting a wide range of scenarios.

2. Develop a simple tool to calculate specific Measures of Effectiveness from the input data and distributions.

  • permit results to be adjusted by Military Judgement

    • to account for factors not addressed by calculations

- the value of the ability to fire on the move

- the value of the greater manoeuvrability afforded

by the lighter system


SUMMARY AND CONCLUSIONS critical.

Appropriate methods of addressing scenario dependencies are

essential to ensure study conclusions are valid.

1. ALL DATA should be regarded as being scenario-dependent.

It is very useful to have an analyst in every team with

special responsibility for addressing this problem.

2. Using combat models to compare performance of systems

can be hazardous.

Consider using a range of methods to generate

intermediate results which are open to scrutiny

and to sensitivity studies.


Framework critical.

Title

Model results

Feedback

Alternative approach

Contents

Rule 2

Data screen 1

Study levels

Results screen

Scen Pars

Study purpose

Conclusions

Histogram

Rule 1

Graph

Highlight top-level

Rule 3

Highlight all

Further Dev’t

TarDes pic

Example study

Data

Leth’y depends

Current issues

Original plan

MutliDisciplinary

Data review

Management Soln

Responsibilities

Model review


Further Development of the CST Tool critical.

1. Development of proper library of routines

2. Improved statistical routines for increase in speed

3. Automated methods for parametric studies

4. Use of EDMS technologies to manage and access study reports


CURRENT ISSUES / PROBLEMS WITH CST-01 critical.

1. It is not clear how best to address the problem of

firing multiple bursts at a target, depending upon

whether it is perceived to be killed.

2. It is not clear whether (and how) costs (or numbers of units)

should be included, or handled separately.


ad