Confronting Agent-Based Models with Data: Methodological Issues and Open Problems

1. Confronting Agent-Based Modelswith Data: Methodological Issues and Open Problems Giorgio Fagiolo University of Verona, Italy Sant’Anna School of Advanced Studies, Pisa, Italy Alessio Moneta Max Planck Institute of Economics, Jena, Germany https://mail.sssup.it/~amoneta/ Paul Windrum Manchester Metropolitan University Business School, Manchester, U.K. MERIT, University of Maastricht, The Netherlands Artificial Economics 2006 Aalborg, September 15

2. Outline • Motivation • Core issues of empirical validation • Heterogeneity in ABM empirical validation Exercises • Taxonomy of different approaches • Open-ended issues

3. Motivations (1/2) • Agent-Based Models (ABM) in Economics • Tremendous development in the last 20 years • Quite successful in exploring market- and industry-dynamics • Emergence of macro properties out of dynamics involving non-trivial interactions among simple micro-economic entities • Improvement in computing power and programming languages • Widespread dissatisfaction with “mainstream approaches” • Need for more “realistic” assumptions in order to get better performances in replicating and explaining real-world observations

4. Motivations (2/2) • Is ABM an alternative paradigm to neoclassical economics? • Did ABMs find a place in the standard economics toolbox? • Published ABM papers in top economics journals • Figures are maybe too pessimistic but overall impact not that big… • New vs. established scientific paradigm (Kuhn, Lyotard) • ABMs are hardly perceived as a robust, alternative paradigm • Why? Keywords: Heterogeneity and poor comparability • Assumptions and modeling design • Analysis of the properties of an ABM • Empirical validation

5. … an important remark … • Too much heterogeneity could be bad • Difficult to compare alternative models of same phenomenon • Difficult to advance a new paradigm and contrast it with already existing ones • Having a (few) commonly accepted protocol(s) for empirical validation (and model building) would be in general better for the profession • … debatable issue … • Also established paradigms are to some extent heterogeneous • Heterogeneity and flexibility of assumptions might be considered as the values added of ABMs • Heterogeneity is a prerequisite for the emergence of a “paradigm” (social process, scientific debate, etc.)

6. Empirical validation • Lack of a stern discipline to confirm or disconfirm simulations models • How to compare and adjudicate competing simulation models? • Standard statistical criteria as goodness of fit are eschewed • Observed time series are seen as one of many possible realizations of a non-ergodic stochastic process • Standard statistical approaches (Fisher, Neyman-Pearson) to hypotheses testing are seen as not appropriate

7. Core issues of empirical validation • Confronting a real-world Data Generating Process with a model Data Generating Process • Concretisation vs. isolation • Realism vs. instrumentalism • Methodological pluralism vs. strong apriorism • Analytical solvability vs. descriptive accuracy • The identification / under-determination problem • Duhem-Quine thesis

8. Simulation approach in Neoclassical Economics • Adelmans simulation (1959) of Keynesian macroeconomic model. Validation was performed looking at the second moments of the variables. • Real Business Cycles School (Kydland-Prescott 1982). • Lucas (1981): “In general, I believe that one who claims to understand the principles of flight can reasonably be expected to be able to make a flying machine, and that understanding business cycles means the ability to make them too, in roughly the same sense”.

9. Calibration Approach • Method of empirical assessment of Real Business Cycle models (Kydland – Prescott 1982) • Lucas (1980): a model should provide artificial system in which policy experiments can be tested. • A model has not to fit the data according criteria dictated by statistical theory. • A model has to be calibrated: its parameters are usually picked in microeconomic empirical investigations. • Simulated model should matches some moments of the data.

10. The limits of the neoclassical approach • Adelmans’ test is not sufficient for being able of answering counterfactual questions. • Micro-foundations, rational expectations and representative agent. • Hard core never confronted with data. • Lucas appeals to Simon’s notion of artifact, but Simon’s approach to complexity is neglected.

11. AB models tend to contain the following features: Bottom-up (agent-based) perspective (Tesfatsion, 1997) Agents live in complex systems evolving through time (Kirman, 1998) Agents are (or might be) heterogeneous in almost all their characteristics Environment is too complex: “hyper-rationality” not viable (Dosi et al., 1996) Agents behave as boundedly rational entities with adaptive expectations “True” dynamics: Systems are typically non-reversible The nature of learning (Dosi et al. 2005) Endogenous and persistent novelty (technological change): open-ended spaces Selection-based market mechanisms (Nelson & Winter: 1982)

12. The (Minimal) Structure of an ABM • Time t = 0, 1, 2, … … Quarters, Years • Set of Agents I= {1, 2, …,N} … Firms • Vectors of Micro-States i  xi,t … Fast micro-vars • Aggregate VariablesXt = f ( x1,t , …, xN,t )… Fast macro-vars • Vectors of Micro-Parameters i  i … Slow micro-vars • Vector of Macro-Parameters   m … Slow macro-vars • Interaction Structures Gt  ( I) … Networks • Micro Decision Rules Ri,t( • | • ) … Production Rule

13. ABMs as Data Generation Processes (DGPs) • ABM provides the DGP of the phenomenon under study • Micro-Macro Parameters • Micro-Macro Initial Conditions • Studying ABMs • Non-linearities and randomness in • individual behaviors • interaction networks • Micro and macro variables are governed by complicated stochastic processes which can hardly be analyzed analytically • Need for computer simulations

14. A possible procedure for studying the output of an AB model

15. Ex 1: Qualitative Simulation Modeling • No empirical validation • Model as a laboratory to gain knowledge on the underlying causal relationships only, not taken to the data • Stylized Qualitative Models (Evolutionary-Games) • Weak relation between micro-macro variables/parameters in the model and empirically observed counterparts • Interest in explaining the emergence of qualitative aggregate pattern (cooperation, coordination, etc.) • Early Evolutionary- and Industry-Dynamics Models • Much more micro-founded and empirically-driven, but… • If any, empirical validation is done in very weak ways • A pessimistic view about empirical validation? • Socio-economics: open-endedness, interdependence, structural change • Precise quantitative implications are difficult to obtain (Valente 2005).

16. Ex 2: Replication of Stylized-Facts • Indirect Calibration • Detailed data able to restrict the set of initial conditions and micro/macro parameters is difficult to gather (Kaldor) • Empirical validation is done at the aggregate (macroeconomic) level • Parameters and initial conditions are not restricted a priori • Validation requires joint reproduction of a set of “stylized facts” (SFs) • Four-Step Procedure (Fagiolo et al., 2004) • Step 1: Identifying set of SFs of interest to be explained/reproduced • Step 2: Keep microeconomics as close as possible to “real-world” • Step 3: Find parameters and initial conditions for which the model is statistically able jointly to replicate the set of SFs • Step 4: Investigation of subspace of parameters and initial conditions which “resist” to Step 3 in order to seek for causal relationships (explanations)

17. Ex 3: Empirical Calibration of ABMs • Werker and Brenner (2005) • Dealing with space of initial conditions and micro/macro parameters • Difficult to employ theoretical arguments to restrict the set • Use empirical knowledge first to calibrate initial conditions and micro/macro parameters and then to validate • Three-Step Procedure • Step 1: Employ empirical knowledge to calibrate initial conditions and parameters ranges • Step 2: Further restricting initial conditions and parameters space by empirically validate simulated output with real-world data • Step 3: Abduction. Seek explanations of the phenomena under study by exploring properties of the “possible worlds” that resist to previous steps

18. Ex 4: History-Friendly Industry Models • Malerba, Nelson, Orsenigo, and co-authors • Models built upon detailed empirical, anecdotic, historical knowledge of phenomenon under study and employed to replicate its precise (qualitative) history • Prominent role for empirical data • Detailed empirical (historical) data on the phenomenon under study assisting model building and validation • Specify agents’ representation • Identify parameters and initial conditions • Empirically validate the model by comparing “simulated trace histories” with “actual history” of an industry

19. Where do they differ? • Domain of application • Micro (industries, markets) • Macro (countries, world economy) • Which kind of empirical observations does one employ? • Empirical data about micro/macro variables • Casual, historical and anecdotic knowledge • How to employ empirical observations? • Assisting in model building (agents, behaviors, interactions,…) • Calibrating initial conditions and parameters • Validating simulated output • What to do first? • First calibrate, then validate • First validate, then calibrate • Validate only

20. Open-ended issues for empirically validating AB models • Alternative strategies for constructing empirically-based models • Problems with over-parameterization • Is it possible to construct counterfactual histories? • The problem of “unconditional objects” in economics (Brock 1999) • Availability, quality and bias of datasets

21. Conclusions • Critical overview of empirical validation in ABMs • When models are taken to the data, many competing approaches • Investigating possible reasons • Methodological debate in social sciences and economics still open • Neoclassical models suffer from similar degree of heterogeneity • A lot of variety in other fields employing simulations as modeling tool • Heterogeneity in economics ABMs’ structure • Crucial problems in empirical validation of ABMs • Treatment of parameters and initial conditions • Comparing simulated distributions with unique real-world observations • Learning about generating mechanisms from replication of SFs • Need for additional data