ABS VV&A Framework Study Phase II Pythagoras COIN – Application of the Validation Framework

1. ABS VV&A Framework Study Phase IIPythagoras COIN – Application of the Validation Framework Lisa Jean Moya WernerAnderson, Inc. lmoya@werneranderson.com Phase II Workshop 3 9 July 2008 P-COIN Validation Briefing Wkshp3 Moya

2. Scenario P-COIN Validation Briefing Wkshp3 Moya

3. Analysis Context • Can Pythagoras be used to model population dynamics? • In a Disaster Relief/Humanitarian Assistance mission for the stated scenario, is it better to base the MAGTF ashore or afloat? • Alternative selection drivers • Do no harm: create no increase in insurgency activity. • Improve the political situation: create an improvement in GOVT and Pro-GOVT sectors. • Measures • Box & Whisker plot comparisons of the percent of population by population segment in each insurgency sector at end state (18 months) P-COIN Validation Briefing Wkshp3 Moya

4. Conceptual Model of Civilian Population Population Segments Civilian Population Natural Drift Salience Influencing events Insurgency Behavior Orientation FARC Pro-FARC Neutral Pro-GoC GoC FARC = Revolutionary Armed Forces of Colombia GoC = Govt of Colombia P-COIN Validation Briefing Wkshp3 Moya

5. P-COIN Validation Briefing Wkshp3 Moya

6. P-COIN Validation Briefing Wkshp3 Moya

7. Areas of Interest • Core • MAGTF influence on Insurgency orientation • Cases • MAGTF/No MAGTF • Ashore/Afloat • Dynamic Influences • Natural Drift • Salience • Background • Population segments P-COIN Validation Briefing Wkshp3 Moya

8. Areas of Interest • Core • MAGTF influence on Insurgency orientation • Cases • MAGTF/No MAGTF • Ashore/Afloat • Dynamic Influences • Natural Drift • Salience • Background • Population segments First order assessment P-COIN Validation Briefing Wkshp3 Moya

9. Pythagoras-COIN Building Blocks P-COIN Validation Briefing Wkshp3 Moya

10. Salience as a Dynamic Influence P-COIN Validation Briefing Wkshp3 Moya

11. Salience as a Dynamic Influence P-COIN Validation Briefing Wkshp3 Moya

12. As the simulation might progress … An extreme example to demonstrate the issue P-COIN Validation Briefing Wkshp3 Moya

13. Implications • Influence changers only applied w.r.t. initial state; changes in orientation do not change the influence • Dynamic effects of salience and natural drift are not accounted for • Secondary and tertiary effects of MAGTF arrival not accounted for • Dampening on the insurgency orientation • Risk is that the simulation does not model the desired population dynamics • Risk is that the dynamics of the MAGTF arrival are not adequately captured P-COIN Validation Briefing Wkshp3 Moya

14. Data • Data imprecise • Mitigated by “tolerance” and multiple runs • Data processing • Need to verify that process results in expected directional & magnitude shifts • Data is perishable • Natural drift data has an embedded perishibility • Would an actual model use require a “warm-up period” on the Markov Chain? • Other influencing events might significantly change the data values "Humanitarian exchange." Wikipedia, The Free Encyclopedia. 3 Jul 2008, 10:26 UTC. Wikimedia Foundation, Inc. 9 Jul 2008 <http://en.wikipedia.org/w/index.php?title=Humanitarian_exchange&oldid=223273181>. P-COIN Validation Briefing Wkshp3 Moya

15. Uses of the Model • Markov assumptions in referent descriptions • No long term effects • May need a “warm-up” period • Outside the salience or along with salience? • Data precision • No exact results; results in the distribution • Data perishability • Need to collect new data after significant events • Including MAGTF departure • Were the dynamics captured … • Could add other influencing events • Could add additional dynamics • Q: Can we apply the influencers more robustly? • Q: Would changing our initial starting agents help? P-COIN Validation Briefing Wkshp3 Moya

16. Analysis Results • Given in box & whiskers plots at end state with data table • No statistical comparisons • No “hard” description of better • Point estimate in time (18 months) P-COIN Validation Briefing Wkshp3 Moya

17. MAGTF Influence Data • Expect: Sea better than shore – Urban Middle Class & Urban Poor “drive” result • Catholic Church drive more right with Sea vs Shore • Displaced Persons “wash”? • Salience causes Urban Poor and Middle Class to be like Military; Military to be like Catholic Church – in opposition to the direct MAGTF influence … what would we expect? P-COIN Validation Briefing Wkshp3 Moya

18. Second Order Effects What do we expect in the interactions P-COIN Validation Briefing Wkshp3 Moya

19. Earlier iteration (Military) No MAGTF • Not clear from documentation what is being reported • These multiple influences appear to be captured • Presuming no population segment strays “too far” from initial state • Except … Military does! Afloat Ashore P-COIN Validation Briefing Wkshp3 Moya

20. Analysis Conclusions (IPR#5) • “Which COA is better?” cannot be answered with much confidence • “What is the chance that ashore is better than afloat?” can be answered with greater confidence • More pro-Government sentiment if Marines stay afloat • Lower pro-FARC sentiment if Marines stay afloat • Marine arrival has a polarizing effect (fewer neutrals) • Marine arrival in either case increases anti-Government sentiments of the Illicit Organizations and the Military • Afloat seems to usually do less harm. • There is no factor in our influence estimation that BOTH reduces the negative impact of Ashore AND increase the negative impact of Afloat 20

21. Analysis Conclusions (IPR#5) (cont) • Because the current Markov chain will eventually return to the same steady state, regardless of MAGTF action, once the MAGTF leaves, we need to consider: • Does the MAGTF commander care about leaving a lasting impression? • At what point in time do we measure ‘better’? • Pythagoras could change the final steady state as a function of one or more population segments exceeding or falling below some target value. However, this data was not collected 21

22. Validation Conclusions • The P-COIN simulation fails to capture the dynamic effects intended in the conceptual model of the insurgency in Colombia provided to the P-COIN developer. That is, P-COIN does not capture the secondary and tertiary effects of the natural drift of population segments between insurgency sectors or the salience between population segments resulting from the influencing event of the MAGTF. • The data supporting the P-COIN model is perishable and of low precision. Care should be taken when using the data beyond its origination date; perhaps “warming-up” the Markov chains supporting the data used to build the P-COIN model. Further, the data cannot be deemed valid if an influencing event occurs that would cause the base data used in this simulation to change. P-COIN Validation Briefing Wkshp3 Moya

23. Validation Conclusions (cont) • The P-COIN model should not be used to evaluate long term effects on the population resulting from the influencing event of the MAGTF arrival. • This model and simulation cannot be deemed as predictive of the actual population distributions amongst insurgency sectors in the event that the scenario described in the scenario documentation actually occurs. • There is little risk in using the results of the analysis since the analysis does not advocate a change in current Marine Corps procedure. However, item 1 implies that P-COIN also provides little insight into the ashore or afloat question in its current implementation. Recommend: Applying influencers more robustly P-COIN Validation Briefing Wkshp3 Moya

24. What Would Be Useful • Better documentation on the P-COIN instantiation • Time series data • A descriptive walk-thru of results charts (meaning & implications) • Verification cases (isolated effects) to ensure dynamics have expected direction (first derivative) and order of magnitude • Descriptions of why we believe it is correct • Referent • Better explanations of expected resulting effects from data values • Most had to be inferred • Order of magnitude differences unknown • Expected interaction effects would be “spectacular” P-COIN Validation Briefing Wkshp3 Moya

25. Can the Results Be Trusted? • Without trusting the dynamics • Take caution but … • Recommendation is innocuous • Under current political circumstances • No … new data is required • Can Pythagoras model population dynamics • Probably … more care is required in the instantiation P-COIN Validation Briefing Wkshp3 Moya

26. Levels of Validation Process Formal Proof Automated (level 5) Automated validation Indnt Observer Confident (level 4) Tier 5, “I’m confident that this simn is valid.” Objective referent Conceptual Model Indnt Observer Accurate (level 3) Tier 4, “For what it does; its repns are accurate enough.” Objective results Complete (level 2) SME Tier 3, “It does the right things; its repns are complete enough.” Objective requirements SME Subjective (level 1) Tier 1, “It works; trust me.” Subjective validation None Validated Analyzed Correlated with statistical estimates of uncertainties Initial (level 0) Tier 0, “I have no idea.” Referent None SME opinion Single source Multiple sources Rigorously derived Adapted from Harmon & Youngblood (2005) p. 186 P-COIN Validation Briefing Wkshp3 Moya