Major Donald Herod, USMC In Partnership with SEED

1. Ascertaining Validity in the Abstract Realm of Constructive Simulation Models: An Analysis of the MAGTF Tactical Warfare Simulation (MTWS) Major Donald Herod, USMC In Partnership with SEED Training as we fight; Fighting as we train! Advisor: William Becker, Ph.d Co-Advisor: Prof Curtis Blais

2. The Problem • As a simulations community, we do a very good job of verifying and validating algorithmic functionality and “real world” platform-based performance in a constructive simulation • As a simulation community, we have NEVER determined how to conduct a holistic validation of a constructive simulation so as to determine if the model as a training tool produces “real world” responses/outputs. • This can lead to negative training transfer and unnecessary loss of life in combat.

3. The Marine Corps premier constructive simulation… capable of full range of MAGTF combat, combat support and combat service support applications, including amphibious operations. • MTWS… still meeting the doctrinal expectations of the current operational environment.

4. System Overview • Constructive, stochastic, aggregate level simulation of ground, air, maritime, and amphibious operations • Computer-assisted command and control training system • Provides exercise control services and tactical combat simulation • Multi-sided, real-time • Full range of battle space capabilities • All phases of military operations

5. System Benefits • User-definable parametric data (characteristics and behaviors) • Global play box; user-definable regions for digital terrain data • Tool for course of action evaluation in operational planning

6. Thesis Overview • Research Question:Can the outputs provided by a constructive simulation be data-farmed to determine if the model’s performance parameters produce valid outputs that stimulate the training audience? Additionally, is there a range of values or specific design points that cause the system to produce algorithmic anomalies (verify)? • Abstract: The requirement to routinely assess staff training simulations is necessary to ensure the interaction within the simulation provides a realistic training environment. This project will explore methods to apply data farming to MTWS to obtain information about its computational characteristics as part of a more general study into use of data farming for simulation validation.”

7. Getting Started • Scenario Development • Think small… simulation is like Pandora's Box • What is the training objective? • Identify Factors of Interest • Deep dive into the parametrics that govern computations in the model • The number of factors will determine the number of Design Points required. • Methodology • Design of Experiments (DOE) • Nearly Orthogonal Latin Hyper-cubes (NOLH) • Batch File Replication • Simulation Run (AKA: Micro-Battles) • Data Mining • Statistical Analysis/Prediction Profiling

8. Orthogonal Latin • Hyper-cube

9. MTWS Output

10. Multivariate Correlation & Scatterplot

11. 3D Scatterplot of the Sample Space

12. Stepwise Fit Model

13. Stepwise Fit Model

14. Prediction Profiler

15. Future Work • Casting a wider net… • Data Farm all User Adjustable Factors of Interest • NOLH with 20 factors of interest (10 per side) • K-Kill Hardness • Mounted Hardness • F-Kill Hardness • Cbt Power • Hit Probability • Firing Rate • M-Kill Hardness • Vulnerability • Destructive Pwr • Range • DOE containing 129 DP (30 reps per DP) • 3870 Micro-Battles • Generate MTWSBatch Files • Trouble-shoot errors and dependencies • StepwiseFit a Model with interactions • Generate Prediction Profiles for Factors on Interest

16. Impacts • Present a new approach to VV&A that provides a holistic assessment of stochastic constructive simulations. • Recommend enhancements to MTWS based on observed anomalies to verify that the model is producing expected outcomes. • Generate a more scientific means of determining appropriate parametircs that are valid for a specific training requirement.

17. Questions