Agenda

1. An Introduction to the Development, Application, Uses and Limitations of STAMPER: Systematic Task Analysis for Measuring Performance and Evaluating Risk Eugenia KalantzisDirectorate Land Combat Development Operational Research TeamCanadian Department of National Defence21st ISMORNorton Manor, Sutton Scotney, Hampshire, UKSeptember 2nd, 2004

2. Agenda • Sponsor Mandate and Objectives • The Seminar Wargame Series • OR Roles and Objectives • Methodology • STAMPER • Concluding Remarks

3. The Sponsor • Director General Land Combat Development (DGLCD) • Director Land Strategic Concept • Director of Army Doctrine

4. DGLCD Mandate Linking the Army of Today to the Interim Model, Army of Tomorrow and the Future Army. Today Tomorrow Future Interim model

5. Four Corners Four Corners Four Corners Four Corners NEO NEO Complex Complex PSO PSO Coy Gp Coy Gp BG/ BG/ Bn Bn Gp Gp Comand Sense Act Shield Sustain Bde Bde Area/Bde Area/Bde Domestic Domestic Warfighting Warfighting Four Corners Four Corners Four Corners Four Corners Four Corners Four Corners NEO NEO NEO Complex Complex Complex Four Corners Four Corners Four Corners Four Corners Four Corners Four Corners PSO PSO PSO Coy Gp Coy Gp Coy Gp BG/ BG/ BG/ Bn Bn Bn Gp Gp Gp NEO NEO NEO Complex Complex Complex Four Corners Four Corners Four Corners Four Corners Four Corners Four Corners PSO PSO PSO Coy Gp Coy Gp Coy Gp BG/ BG/ BG/ Bn Bn Bn Gp Gp Gp NEO NEO NEO Complex Complex Complex Four Corners Four Corners Four Corners Four Corners Four Corners Four Corners PSO PSO PSO Coy Gp Coy Gp Coy Gp BG/ BG/ BG/ Bn Bn Bn Gp Gp Gp NEO NEO NEO Bde Bde Bde Complex Complex Complex Area/Bde Area/Bde Area/Bde PSO PSO PSO Coy Gp Coy Gp Coy Gp BG/ BG/ BG/ Bn Bn Bn Gp Gp Gp Domestic Domestic Domestic Warfighting Warfighting Warfighting Comand Sense Act Shield Sustain Bde Bde Bde Area/Bde Area/Bde Area/Bde Domestic Domestic Domestic Warfighting Warfighting Warfighting Bde Bde Bde Area/Bde Area/Bde Area/Bde Domestic Domestic Domestic Warfighting Warfighting Warfighting Bde Bde Bde x x Area/Bde Area/Bde Area/Bde MCF Brigade MCF Brigade Domestic Domestic Domestic Warfighting Warfighting Warfighting Group HQ Group HQ Command Command Support Support MP MP Force Employment Seminar Wargame Series • Series of Seminar Wargames to evaluate evolving FE concepts Subsequent Iterations Baseline Iterations War Fighting Domestic Feb 2004 May 2004 Non-Combatant Evac Peace Support Current Structures Baseline Performance Baseline Performance Updated Concepts Updated Concepts Updated Performance Updated Performance Updated Performance Updated Structures x x x MCF Brigade MCF Brigade MCF Brigade x x x Group HQ Group HQ Group HQ Command Command Command MCF Brigade MCF Brigade MCF Brigade Support Support Support Group HQ Group HQ Group HQ Command Command Command Support Support Support MP MP MP MP MP MP

6. Force Employment Seminar Wargame Series • Seminar wargame format • 5 days and 45 participants • Detailed scenarios, vignettes and forces • Joint Conflict and Tactical Simulation (JCATS) for situational awareness • STAMPER • Structures judgments and insights sessions

7. HEAVY FEATHERS RABBIT REDUX WILD HONEY LEATHER STOCKINGS X HERA Baku MEDUSA WILD HONEY Gormanskaya APHRODITE HEAVY FEATHERS Traditional Combat in Open Complex Terrain (but not urban) Urban Operations

8. OR Roles and Objectives • Role: • Problem definition, guidance, data collection and analysis, presentation of results • Objectives: • Develop a comprehensive and robust methodology to measure performance and evaluate risk factors • Present results in a clear concise manner to be briefed to decision-makers

9. Special Considerations • Imperfect knowledge of models, equipment, structures, particularly for the Interim Army • Transient nature of the wargaming teams • Nature of the seminar wargame - hidden complexity in the verbal interplay • Multiple levels participation

10. Analysis plan – Two Tier Approach • Quantitative • STAMPER • Qualitative • Observation Sheets • Strengths/ Weaknesses/ Issues Sheets • Judgment & Insights Sessions

11. STAMPER – Systematic Task Analysis for Measuring Performance and Evaluating Risk • Development of the DGLCD Task Lists • Development of Survey Instruments • Ratings rolled-up for overall performance per operational function, and evaluation of risk • Interpretation of results • Comparison of results from one seminar to another

12. DGLCD Task Lists • Developed by Combat Developers and assessment teams • Comprehensive list of tasks per Operational Function: Command, Sense, Act, Shield and Sustain • Level 1, Level 2 and Level 3 Tasks

13. Sample of Command Task List Breakdown

14. DGLCD Task Lists • Client-analyst collaboration was key • Process: Iterative “straw man” approach • Benefits: • Engagement of participants – sense of ownership • Constant review and refinement • Synergies achieved

15. Survey Instruments • Based on the DGLCD Task Lists • One EXCEL based survey instrument per operational function • Assessors asked to rate tasks along two dimensions: • Performance • Impact/Importance

16. Survey Instruments… Survey Questions Response Options List of tasks Assessor’s responses for all 6 vignettes

17. Performance Measurement The simple model… Performance score of a task = weighted average of the performance scores of the tasks belonging to it. For example: The performance of a Level 2 task is the weighted sum of the performance scores of the Level 3 tasks that make up that Level 2 task.

18. Roll-up of scores…

19. Assessment of Risk For Level 3 tasks only… • Low score = higher risk • Color-coded for visual cuing • Numerical values for ranking

20. Automated Tool Risk Performance

21. Example 1. High Risk – Low Performance Command Tasks

22. Example 2. Operational Function Overall Scores

23. Strengths • Client/analyst collaboration • Structured evaluation of performance for combat developers • Confirmation of seminar discussion results • Visualization of results • Colour-coding and graphs • Quick identification of performance/risk deltas • Near real-time results • Available for discussion at end-of-day J&I • Clarification of remaining issues

24. Caution… • Should not compare results across op functions • Within operational function • Across scenarios/vignettes (e.g. open vs. urban) • Across capability sets (i.e. across seminar wargames) • Deltas should be used as indicators, pointers to an underlying phenomenon requiring investigation • Results to be interpreted along with qualitative results

25. Abusing the Simple Model Ex. 1 Assuming all tasks are performed at an acceptable level Ex. 2 Assuming equal improvement/ investment across operational functions

26. Concluding Remarks • Seminar wargame format appropriate for scope of investigation • Complimentary data collection methodologies: quantitative and qualitative (STAMPER) provided complete picture • STAMPER • Met the objectives: consistent, repeatable, and simple • Structured evaluation • Good communications tool • Results should be interpreted with caution • Provided an indication of performance/risk to a level deemed appropriate, and not beyond

27. Questions?

29. New Structure Same mission New System: Modified Capability Set Mission: Objectives, threat, environment, etc. From Baseline to Next Iterations Baseline System: MCF Mission: Objectives, threat, environment, etc.

30. Specification Error Error Error Error Complexity Complexity Complexity Model Error Models, Complexity and Error Measurement Error Minimize Model Error

31. Specification Error Error Error Error Complexity Complexity Complexity Model Error Models, Complexity and Error Measurement Error Minimize Model Error by decreasing complexity

32. Survey Instrument Questions Question 1: Performance Given your experience and the discussions held during the seminar wargame, please indicate the level of overall performance you believe we would most likely achieve in the completion of this task during this mission, given the capability-set assumed for this seminar wargame.

33. Survey Instrument Questions Question 2: Impact For this particular mission, how would you characterize the impact or importance of this task in the overall performance of the higher-level task it belongs to.

34. Point Allocation Matrix Assessor Response to Question 1 Assessor Response to Question 2 Score = Sum of points Sum of weights

35. Example 4. Distribution of Sense L3 Scores For Level 3 Tasks: Q1 results Q2 results

36. Example 2. Comparison Across Seminars

37. Sponsor Objectives • To assess/measure the impact of changes in structures, equipment and capabilities that will come into effect during the Interim Army timeframe with a view to further refining the force employment concept in preparation for field trials at Canadian Manoeuvre Training Centre and eventual incorporation into doctrine • To identify and prioritize discrete issues requiring additional analysis and to direct the examination of these spin-off activities to the appropriate organization

38. Level 2 Tasks Level 1 Tasks Operational Function Survey Instruments Responses Provided by Assessors Question 1: Performance Derived from the Roll-Up Level 3 Tasks Question 2: Impact Level 3 Tasks Level 2 Tasks Level 1 Tasks

39. Future Development • Refinement of DGLCD Task Lists • Refinement of weights using techniques such as ranking, pair-wise comparison, etc. • Increase number of surveys collected – statistical analysis • Hand-held data entry devices for instantaneous collection of surveys

40. Abusing the Simple Model Ex. 3 Assuming improvement in Command exceeds improvement in other operational function; basing investment decisions on this