MIS 643 Agent-Based Modeling and Simulation (ABMS) Bertan Badur badur@boun.tr Department of

1. MIS 643 Agent-Based Modeling and Simulation (ABMS) Bertan Badur badur@boun.edu.tr Department of Management Information Systems Boğaziçi University

2. Model Analysis • Chapter 21-23, of Agent-Based and Individual-Based Modeling: A Practical Introduction,by S. F. Railsback and V. Grimm

3. Outline • Chapter 21: Introduction to Part IV • Chapter 22: Analyzing and Understanding ABMs • Chapter 23: Sensitivity, Uncertainty and Robustness Analysis • Chapter 24:

4. Chapter 21: Introduction to Part IV • 21.1 Objectives of Part IV • 21.2 Overview of Part IV

5. 21.1 Objectives of Part IV • Testing – checking whether a model or submodel is correctly implemented and does what itis supposed to do • Analysing a model: trying to understand what amodel does • Understanding not automatic • from begining of modeling cycle • sukbmodels or simple models • POM for sturucture, theory calibration • Full models • frase design at some point • understand how it works and behave

6. not too soon • once the model • key processes • represent real system reasonably • version number • two or three versions is likely • Programming and testing easy • What is science? • relation between model and real system – POM Part III • analyse throughly – what it does • simlfy or extend by adding new elements • formulation few days, analysing months yearns

7. 21.2 Overview of Part IV • Chapter 22 • general strategies of analyzing ABMs • specific to ABMs • structural richness and realism • through controled simulation experiments • change assuptions submodels ... • Chapter 23 • sensitivity, uncertainty and robustness Sensitivity, Uncertainty and Robustness Analysis

8. Chapter 22: Analyzing and Understanding ABMs • 22.1 Introduction • 22.2 Example Analysis: The Segregation Model • 22.3 Additional Heuristics for Understanding ABMs • 22.4 Statistics for Understanding • 22.5 Summary and Conclusions

9. 22.1 Introduction • controlled experments • varying one factor at a time – efeects on results • establishng causal relationships – understanding how the results are affected by each factor • Scientific method – reproducable experiments • compleatly dercribing the model - lab or field • documenting • parameter values- input data- initial conditions • anaylsing results of experments

10. controlled simulation experiments • design, test and calibrate - models • understanding and analyzing what models do • How to analyse • model, the system and questions addressed, • experience and problem solving heuristics • Heuristics or rule of tumbs • often usefull but not always • not unscientific

11. learning objectives • Understan purpose and goals of analyzing full AMBs • finished or preliminary • ten heuristics • statistical anaysis for ABMs

12. 22.2 Example Analysis: The Segregation Model • ODD • purpose • entities, state variables and scales • turtles – households • loaction, heppyness • houses - patches • space 51*51 • time • stop – all heppy

13. Processes • if all happy stop • far all aent • if lo limit move • update heppyness • produce output

14. submodels • move • update

15. Analysis

16. Heukristic: try extream values of parameters • model outcomes is often easy to predict or understand • Set tolernce low • Set tolarance high

17. Heuristics: findtipping points in model behavior • qualitatively diferent behavior at extream values of parameters • vary the parameter try to find “tipping point” • the parameter range – model behavior suddenly changes • regiems of control • process A after some point process B may dominent

18. Heuristics • try different visual representations of the model • color size patches • run the model step by srep • look at striking or strange patterns • at interesting points keep the parmeter and vary other parameters

19. 22.3 Additional Heuristics for Understanding ABMs • use several “currencies” for evaluating your simulation experiments • analyze simplified version of your model • analyze from the buttom up • explore unrealistic senarios

20. Heuristics: use several “currencies” for evaluating your simulation experiments • ABMs are rich in structure • “currincies” summary statistics or observations • emprical measures in the real system • Ex: population modeling • measure – population size wealth • analyze time series of population size • even mena or range • good currincies – observation in ODD design concept • several currincies – how sensitive they are

21. statistical distributions • mean standard deviation, range • distribution – normal, exponential • characteristics of time series • trend, autocorrolation time units to reach a state • measures of spatical distributions • spatial autocorrelation, fractile dimension • measures of difference among agents • how some charcetristics different, distributions • stability properties • network characteristics • clustering coefficient, degree

22. Heuristics: analyze simplified version of your model • simplfy • ABM so many foctors affect output • reduce complexity • undertand what mechnizms what cause what results • make the environment constant • make space homogenuous • all patches same over time • reduce stocasticity • fixed initial conditions – all agent alike • insteaad of randomness use mean values • reduce the system size • turn off some actions in model schedule • manually create simplified initail configrations

23. Heuristics: analyze from the buttom up • ABMs hard to understand • behavior of its parts – agents and their behavior • first test and undertsnd these • then full model • anaysis of submodels • developing theory for agnet bahavior

24. Heuristic: explore unrealistic senarios • simulate senarios – never occur in reality • to see direct effect of a process or mechanizm on resutls – remove it • Ex 2: How investor behavior affects double –auction markets • interesting contrast: • models – unrealistically simple investor behavior • produce system level results not so unrealistic • conclusion • complex agent behavior – not reasn for complex market dynamics • market rules themselfs might be important

25. 22.4 Statistics for Understanding • statistics – analysis and understanding • infer causal relatinships from a limited and fixed data • ABM – • generates as much data aa possible • additional mechnizms • if cannot explain • add new mechanizms • change assuptions • purpose and mind-set of • statistics and simulation modeling • are quite different

26. summary sttistics • aggregagting model outputs - mean, standard deviation • extream values might be importnat so outliers are usefull • Contrasting senarios • detect and quantify differences between senarios • assumptions may affect resutls – number of treatments • easier to change assuptions • t test ANOVA

27. Quantifying correlative relationships • regression ANOVA • statistical relationsships between inputs – outputs • inputs: paramerters, initial conditions, time series • not directly idenfy causal relations • but idenfity relavant factors • meta-models • Comparing model outputs to emprical patterns

28. 22.5 Summary and Conclusions • combin • reasoning, strong inference, systematic anaysis, intiution and creativity • once build an ABM or freeze it • understand what is does – controlled simulation experiments • heuristics • publications • heuristics in figure 22.3 • add your own

29. Chapter 23: Sensitivity, Uncertainty and Robustness Analysis • 23.1 Introduction and Objectives • 23.2 Sensitivity Analysis • 23.3 Uncertainty Analysis • 23.4 Robustness Analysis • 23.5 Summary and Conclusions

30. 23.1 Introduction and Objectives • Does an ABM reproduce observed patterns robustly • or sensitive to change in model • parameter • structure • how uncertain are model results • if model reproduce patterns foır • parameters – limited range or values • key processes are modelsed one exact way • unlikely to capture real mechanizm underlying hhe patterns

31. Basic Definitions • Sensitivity analysis (SA) exokıres how sensitive model’s outputs are to changes in parameter values • Uncertainty Analysis (UA) looks at how uncertainty in parameter values affect the relaibility of model results • Robustness analysis (RA) explores robustness ofresults and conclusions of a model to changes in its structure

32. Learning objectives • local SA with BehavioSpace • visualizations – SA with several parameters or global SA • stamdard UA methods with BehaviorSpace • steps of conducting RA

33. 23.2 Sensitivity Analysis • to perform SA • full version of the model • “reference” parameter set • one or two key outputs • controled simulation conditions

34. 23.2.1 Local Sensitivity Analysis • Objective – how sensitive the model • currency seleced • parameters one at a time • usually all parameters • Steps • range of parameter – +or-5% • run model for reference P and p-dP p+dp – replicate • mean C values • calculate sensitivity – approximatins to partial derivative

35. Three types of parameters • high values of S • processes imortant in the model • high value of S and highuncertrainty in reference valus • little information to estimate their values • special attantion as calibration • target of emprical research to reduce uncertainty • low values of S • relatively unimportant processes - removable

36. Alternatives • only positive change • C’/C absolugtechange • distibuton of C – variance • diferent values of P • rgression of C on P

37. Limitations • linear response so parameter change is small • parameter interractios missing • around reference parameter set

38. 23.2.2 Analysisof Parameter Interractions via Countour Plots • contour plots – interractions of two parameters • all other parameters are kept constant • Multi-panel contour figures – model sensitivity • many parameters at onces

39. 23.2.3 Global Sensitivity Analysis • vary all parameters over their full range • look at several currencies - understanding • “brute force” - analysis • for each parameter several values • replicaitons • hard to measure currencies • regression analyis – respose surface methods • design of simulation experiments • not all combination of parameters

40. 23.3 Uncertainty Analysis • similar to SA but • to understand how • the uncertainty in parameter values and • model’s sentitivity to parameters • interract to cause uncertainty in model results • parameters – measurment errors • steps of a UA • identify the parameters • for each parameter – define a distribution • belief or measurment errors • run the modelmany times – drawing from distributions • analyze distribution of model results

41. 23.4 Robustness Analysis • Weisberg (2006) • Whether the results depends on the • esentials of the model or • details of the simplfying assuptions • study number of distinct similar models of the same phenomena • despte different assumptions – similar results • robust theorm - free of details of the model • modeling, POM • robust explanations of observed patterns

42. A full model – frozen • two heuristics: • analyze simplified versions • explore unrelistic senarios • more complex versions • General steps of RA • start with a well tested model version • which elements to modify • test modified model – reproduce observed patterns

43. theory development – agent behavior • testing alternative submodels • RA • testing alternative versions • 23.4.1 Example: Robustness Analysis of the Breeding Synchrony Model • left as an exercise

44. 23.5 Summary and Conclusions