Comments on Modeling & Validation, etc.

1. Comments on Modeling & Validation, etc. By Curmudgeon (aka Gene Visco) 3rd Workshop on ABS Validation 8-10 July 2008 Northrop Grumman Fairfax, VA genevisco@embarqmail.com

2. This is what you’re up against! Plan to be spontaneous tomorrow [actual Fortune Cookie]

3. Dilbert

4. Dilbert

5. Dilbert

6. From the Sublime to the Ridiculous • “All models are wrong…” • E = mc2 • Replacement bumper sticker: “All models are incomplete, but we really don’t care!” • Balance of this presentation • Model Improvement Programs • Operational Realism in Modeling… • Simulation Validation • Usefulness • VV&A from DoD viewpoint • It’s Analysis, S….d! • Ultimate responsibility

7. More History Than You Ever Wanted • First there was the computer! • AMIP (MISMA, not MIASMA) • MORIMOC I, II, & III: useful, relevant; impact • SIMVAL: here’s where the cheese gets binding! • MORS: influencing, but not making policy

8. History Continued • OSD & DMSO (whatever?) • What Hath God Wrot? (Where’s the beef?) • What we forgot: “…Law of Unforeseen Consequences…Bad always follows good…Any action, no matter how noble the intentions behind it, sooner or later has unanticipated ramifications that are mischievous at best, disastrous at worst, and if that action is legislative or judicial, the potential for unforeseen and undesirable consequences increases exponentially."

9. A Sage: Catechism due to Gene Woolsey • Did you know what they were doing before you modeled it? • If yes, how did you know? (The only acceptable answer is “Because I did it the old way first.”) • Is your model in use? • If yes, how do you know? • Does it work? • If yes, is there a measurable, verifiable reduction in cost over what was done before or a measurable, verifiable increase in readiness? • If yes, show it to me now.

10. Some Good Signs • Dealing with assumptions • Agent Based Modeling • Human behavioral data • Cop out: The saving grace is that for all of our efforts, weaknesses, and continued efforts to reach “truth,” we have to face the humbling observation that few if any significant military decisions are made solely on the basis of a model’s output.

11. We Come to the End • To the Barricades…(“Arise, you prisoners of starvation…”) • Put the responsibility for proper tools, quality, relevance, accuracy, pertinence, & validation where they belong: in the hands of the analyst • After all, in the last analysis, it is analysis & not models!

12. Worthwhile Readings • Zimmerman: “A Monte Carlo Model for Military Analysis” • Davis & Henninger, Analysis, Analysis Practices, & Implications for Modeling & Simulation • Akst: Phalanx, Dec. 2007 & “Musings on Validation”