- 42 Views
- Uploaded on
- Presentation posted in: General

Model Validation as an Integrated Social Process

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Model Validation asan Integrated Social Process

George P. Richardson

Rockefeller College of Public Affairs and Policy

University at Albany - State University of New York

- No model has ever been or ever will be thoroughly validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring confidence’ are more apt descriptors applying to models than ‘valid’(Greenberger et al. 1976).
- Validation is a process of establishing confidence in the soundness and usefulness of a model. (Forrester 1973, Forrester and Senge 1980).

- Not ‘Is the model valid,’ but
- Is the model suitable for its purposes and the problem it addresses?
- Is the model consistent with the slice of reality it tries to capture? (Richardson & Pugh 1981)

Adapted from Saeed 1992

Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981

- Conceptualizing:
- Do we have the right people?
- The right dynamic problem definition?
- The right level of aggregation?

- Mapping: Developing promising dynamic hypotheses
- Formulating: Clarity, logic, and extremes
- Simulating: Right behavior for right reasons
- Deciding: Implementable conclusions
- Implementing: Requires conviction!

High

Opposition

Low

Problem Frame

Low

Support

High

Weak

Strong

Stakeholder Power

Bryson, Strategic Planning for Public and Nonprofit Organizations

High

Interest

Low

Weak

Strong

Power

Eden & Ackerman 1998

- Think causally, not correlationally
- Think stocks and flows, even if you don’t draw them
- Use units to make the causal logic plausible, even if you don’t write them down
- Be able to tell a story for every link and loop
- Move progressively from less precise to more precise -- from informal map to formal map

- We start with some understandings of the problem and its systemic context, and then we conceptualize (map) the system.
- Then we build the beginnings of a model, which we then test to understand it.
- Then we reformulate, or reconceptualize, or revise our understandings, or do some of all three, and then continue…

The words here represent stocks.

This is not a causal diagram.

No explicit stocks or flows, no clear units, but it tells a compelling story – It’s a good start.

- Recognizable parameters
- Robust equation forms
- Phase relations
- Richardson’s Rule: Every complicated, ugly, excessively mathematical equation and every equation flaw saps confidence in the model.

- P of int'l conflict =
DELAY FIXED ((Lateral pressure/10*Military force effect/Trade and bargaining leverage + International conflict)/Lateral conflict break point, 1 , 0)

- Flaws
Complexity, discreteness, units confusion and disagreement, disembodied parameter, confusion of the effect of a concept [leverage] with the concept itself, and the wonder what keeps this probability between 0 and 1?

Constant Perceived Value suggests continually rising Resources, but that doesn’t seem correct

Here, the Perceived Value of Integrated Information sets a planned level of resources

- Generally, a weak test of model validity
- Whole-model procedures
- Optimization

- Partial-model procedures
- Reporting results
- Graphically
- Numerically: Theil statistics

- Logistic curve
- dx/dt = ax - bx2

- Gompertz curve
- dx/dt = ax - bx ln(x)

- Theil statistics, for example
- Based on a breakdown of the mean squared error:
- 1 = Bias + Variation + Covariation

- Have clear a priori expectations
- Follow up all unanticipated behavior to appropriate resolution
- Confirm all behavioral hypotheses through appropriate model tests (Mass 1991/1981)

- Testing the symmetry of policy response (up and down)
- Testing large amplitude versus small amplitude response
- Testing policies entering at different points
- Testing different patterns of behavior
- Isolating uniqueness of equilibrium or steady state
- Understanding forces producing equilibrium positions (Mass 1991/1981)

- Modelers, stakeholders, problem experts, and others in the modeling process pursue validity at every step along the way.
- We have rigorous traditions guiding model creation, formulation, exploration, and implications.
- We have a powerful, intimidating battery of tests of model structure and behavior.
- Model-based conclusions that make it through all this deserve the confidence of everyone in the process.

- Reason is itself a matter of faith. It is an act of faith to assert that our thoughts have any relation to reality. (G.K. Chesterton)
- I have no exquisite reason for’t, but I have reason good enough. (Sir Andrew, Twelfth Night)