GUIDANCE FOR DEVELOPERS AND MANAGERS ON VERIFYING, VALIDATING, AND ACCREDITING MILITARY MODELS

1. GUIDANCE FOR DEVELOPERS AND MANAGERS ON VERIFYING, VALIDATING, AND ACCREDITING MILITARY MODELS Paul K. Davis RAND and the RAND Graduate School Santa Monica, Ca Prepared for presentation at the 1993 SCS Simulation Multiconference, March 29-April 1, 1993, Arlington, VA. Based on work sponsored by the Defense Modeling and Simulation Office

2. BACKGROUND • Fuller treatment in Paul K. Davis, Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations, RAND, R-4249-ACQ, 1992 • Reflects work by Applications and Methodology Working Group of the Defense Modeling and Simulation Office (DMSO) • Reflects participation in ongoing SIMVAL activities of Military Operations Research Society (MORS) • Perspective and details, however, represent author’s views: no claim of consensus or official basis

3. DEFINITIONS Model= “Bare model” plus data base Verification: process of determining that model implementation accurately represents developer’s conceptual description and specification Validation: process of determining: (a) manner in which and degree to which model accurately represents real world from perspective of intended uses (b) subjective confidence that should be placed on this assessment Accreditation: an official determination that model is acceptable: —to a class of applications —more particularly, to a specific analysis or other application

4. TAXONOMY FOR VV&A VV&A VERIFICATION VALIDATION ACCREDITATION PROVISIONAL FOR APPLICATION CLASS LOGIC ANDMATH IMPLEMEN- TATION AS PART OF SPECIFIC ANALYSIS PLAN EMPIRICAL • Historical data • Field-test data • Laboratory data • Data from maneuvers and other exercises THEORETICAL • For analytic rigor • For verisimilitude • By criteria of clarity and economy • By comparisons with other theories and models OTHER • Expert opinion • Doctrine • Other models • Other sources of information

5. OBSERVATIONS • Verification is not merely auditing transcription of specs. into computer code: includes checking basic logic and math • Generalized validation involves many types of evaluations and results in holistic judgments: • Narrow testing (e.g., agreement with another “blessed model” is not enough) • Basic issue is: How much would one bet on model’s correctness? • A key element of validation is theoretical analysis • Accreditation can be generic and provisional, or specific. Latter is more important.

6. TAXONOMY CAN GUIDE MANAGEMENT PLAN • Few individual projects can afford serious validation • Most models are used for years • Higher-level organizations (e.g. programs, departments) should cover all relevant boxes of validation hierarchy in multiyear plan • Individual projects should focus on application-specific aspects, but contribute to long-term overall VV&A • Existence of long-term VV&A plan can improve incentives for VV&A: • Demonstrate its importance to organization • Provide time for serious work • Provide opportunities to reward related work

7. TYPES OF VALIDITY • Descriptive: can explain or organize • Structural: has proper entities, attributes, ... • Predictive: can predict desired system features, at least within specified domains

8. ISSUES OF MANNER, DEGREE AND CONFIDENCE Models are seldom perfectly valid in any of dimensions; also, there are dimensions of confidence: • Model and data may be known to be highly uncertain • Model and data may represent best-estimate consensus of experts, but may be fundamentally wrong • Experts may or may not know they could be very wrong • Model may be deterministic, while world is stochastic • Lack of knowledge (e.g., about other commander’s tactics) • Effectively stochastic phenomena (e.g., cloud cover) • Decision models may be “optimal” (for perfect information), “representative of doctrine,” or merely “reasonable.” What is “correct?” How does one assess “validity?”

9. ILLUSTRATIVE JUDGMENT ABOUT VALIDITY The strategic mobility model itself is solid, for aggregate predictions, but predictions depend on planning factors and on decisions unknowable in advance. Thus, we should plan for buildup rates +/- 30% around baseline data. Also, we should recognize that the CINC may make significant changes in the TPFDL, so we must anticipate the most likely kinds of changes and consider their consequences. Note: this model is descriptively and structurally valid, but its predictiveness is limited.

10. ILLUSTRATIVE JUDGMENTS (2) • Because of uncertainties, including random factors and intrabattle decisions, we have no confidence in predicting winner or loser (or casualty rates)—unless we can stack the deck with a 6:1 local force ratio after bombing. • The ECM-ECCM model is very accurate for aircraft flying against the SA-99 as we know it, but the enemy may have changed subsystems, in which case noise jamming would still apply, but false-target generation might not work at all. We simply don’t know whether he has changed subsystems.

11. SUBSTANTIVE-PROCESS VIEW OF ACCREDITATION Previous V&V information ADAPT MODEL Objectives, information on model and previous VV&A IRevised model and documen- tationl DO V&V ON ADJUSTED MODEL Requirements DEVELOP ANALYTIC PLAN FOR STUDY Improved base of V&V infor- mation Specific needs and test cases Test results, including sensitivities and previous information Detailed analysis plan, including hypotheses and inference logic PERFORM ACCREDITATIONREVIEW Study-specific accreditation plus guidance and constraints

12. OBSERVATIONS ABOUT ACCREDITATION • Generic accreditation (by class of applications) should be provisional and restrained: e.g., • Does model perform as advertised? • Intended uses? General strengths and weaknesses? • Is baseline data base reasonable? • Study-specific accreditation is key • Model and data need to be tailored to study • Validity may depend sensitively on analysis details • Major threat to scientific quality of applications: • Managers will insist on using poorly suited “accredited” models for assumed effects on credibility • Managers will reject better unaccredited models and discourage new modeling or major adaptations • Managers will focus on model rather than analysis

13. RECOMMENDATIONS TO MANAGERS • Insist on and provide incentives for continuingVV&A, as part of standard professional practice. Pay for it! ( 20%?). • Bring staff into process of defining policies and procedures. • Place authority for specific accreditation with those responsible for study or exercise. • When doing studies or exercises, focus on applications and analysts, not models per se. • Reject arguments based on model “acceptance”—especially when proposal is to use off-the-shelf model “mechanically.”

14. IMPLICATIONS OF MODERN TECHNOLOGY • Classic recommendations: • separate model design from program implementation • separate VV&A according to model or program • require hard-copy documentation of model and program • New technology: • improves ability to design model and program simultaneously (graphical inputs, high-level languages,...) • permits revolutionary approach to documentation (on-line emphasis, hypermedia methods,...) • encourages highly interactive style of modeling and analysis(e.g., spreadsheets, RAND-ABEL and ANABEL,...) • permits distributed operations • Observations: • Need to rethink design, documentation, and VV&A • Need to exploit high-level languages, networking, etc. • Need to exploit commercial tools • Need to avoid inappropriate standards (e.g., Ada )

15. ONE IMPLICATION OF DISTRIBUTED SIMULATION • New requirement: ability, quickly, to comprehend and adapt models developed by other organizations • Commercial software is partial existence proof (Word, Power Point, Excel,...) for rapid learnability • Needed: comprehensive modeling and analysis environments that will facilitate design, development, exporting, importing, etc. Initial thoughts on issue: • Robert Anderson, Steven C. Bankes, Paul K. Davis, Edward Hall, and Norman Shapiro, Toward a Comprehensive Environment for Computer Modeling, Simulation, and Analysis, RAND, 1993.