Model validation as an integrated social process
This presentation is the property of its rightful owner.
Sponsored Links
1 / 42

Model Validation as an Integrated Social Process PowerPoint PPT Presentation


  • 48 Views
  • Uploaded on
  • Presentation posted in: General

Model Validation as an Integrated Social Process. George P. Richardson Rockefeller College of Public Affairs and Policy University at Albany - State University of New York [email protected] What do we mean by ‘validation’?.

Download Presentation

Model Validation as an Integrated Social Process

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Model validation as an integrated social process

Model Validation asan Integrated Social Process

George P. Richardson

Rockefeller College of Public Affairs and Policy

University at Albany - State University of New York

[email protected]


What do we mean by validation

What do we mean by ‘validation’?

  • No model has ever been or ever will be thoroughly validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring confidence’ are more apt descriptors applying to models than ‘valid’(Greenberger et al. 1976).

  • Validation is a process of establishing confidence in the soundness and usefulness of a model. (Forrester 1973, Forrester and Senge 1980).


The classic questions

The classic questions

  • Not ‘Is the model valid,’ but

  • Is the model suitable for its purposes and the problem it addresses?

  • Is the model consistent with the slice of reality it tries to capture? (Richardson & Pugh 1981)


The system dynamics modeling process

The system dynamics modeling process

Adapted from Saeed 1992


Processes focusing on system structure

Processes focusing on system structure


Processes focusing on system behavior

Processes focusing on system behavior


Two kinds of validating processes

Two kinds of validating processes


The classic tests

The classic tests

Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981


Validation is present at every step

Validation is present at every step

  • Conceptualizing:

    • Do we have the right people?

    • The right dynamic problem definition?

    • The right level of aggregation?

  • Mapping: Developing promising dynamic hypotheses

  • Formulating: Clarity, logic, and extremes

  • Simulating: Right behavior for right reasons

  • Deciding: Implementable conclusions

  • Implementing: Requires conviction!


Do we have the right people

Do we have the right people?


Problem frame stakeholder map

Problem frame stakeholder map

High

Opposition

Low

Problem Frame

Low

Support

High

Weak

Strong

Stakeholder Power

Bryson, Strategic Planning for Public and Nonprofit Organizations


Power versus interest grid

Power versus Interest grid

High

Interest

Low

Weak

Strong

Power

Eden & Ackerman 1998


Pursuing validity in mapping

Pursuing validity in mapping

  • Think causally, not correlationally

  • Think stocks and flows, even if you don’t draw them

  • Use units to make the causal logic plausible, even if you don’t write them down

  • Be able to tell a story for every link and loop

  • Move progressively from less precise to more precise -- from informal map to formal map


The standard cautions

The standard cautions


These arrows mean and then

These arrows mean ‘and then’

  • We start with some understandings of the problem and its systemic context, and then we conceptualize (map) the system.

  • Then we build the beginnings of a model, which we then test to understand it.

  • Then we reformulate, or reconceptualize, or revise our understandings, or do some of all three, and then continue…


Arrows here are flows of material

Arrows here are flows of material

The words here represent stocks.

This is not a causal diagram.


Only this one is a causal loop

Only this one is a causal loop

No explicit stocks or flows, no clear units, but it tells a compelling story – It’s a good start.


Project modeling core structure

Project modeling core structure


Identical structure without explicit stocks and flows

Identical structure without explicit stocks and flows


Pursuing validity writing equations

Pursuing validity writing equations

  • Recognizable parameters

  • Robust equation forms

  • Phase relations

  • Richardson’s Rule: Every complicated, ugly, excessively mathematical equation and every equation flaw saps confidence in the model.


Modeling conflict within between nations

Modeling conflict within & between nations


Complexity flaws destroy confidence

Complexity & flaws destroy confidence

  • P of int'l conflict =

    DELAY FIXED ((Lateral pressure/10*Military force effect/Trade and bargaining leverage + International conflict)/Lateral conflict break point, 1 , 0)

  • Flaws

    Complexity, discreteness, units confusion and disagreement, disembodied parameter, confusion of the effect of a concept [leverage] with the concept itself, and the wonder what keeps this probability between 0 and 1?


Robust equation forms

Robust equation forms


Causal mish mash

Causal mish-mash


Robust equation formulations

Robust equation formulations


Robust equation formulations1

Robust equation formulations


Robust equation formulations2

Robust equation formulations


Robust equation formulations3

Robust equation formulations


Pursuing validity in equations phasing

Pursuing validity in equations: Phasing


Phase relations

Phase relations

Constant Perceived Value suggests continually rising Resources, but that doesn’t seem correct


Phase relations1

Phase relations

Here, the Perceived Value of Integrated Information sets a planned level of resources


Pursuing validity fitting to data

Pursuing validity fitting to data

  • Generally, a weak test of model validity

  • Whole-model procedures

    • Optimization

  • Partial-model procedures

  • Reporting results

    • Graphically

    • Numerically: Theil statistics


Example of weakness of fitting to data

Example of weakness of fitting to data

  • Logistic curve

    • dx/dt = ax - bx2

  • Gompertz curve

    • dx/dt = ax - bx ln(x)


Fitting global petroleum with logistic

Fitting global petroleum with Logistic


Fitting global petroleum with gompertz

Fitting global petroleum with Gompertz


Presenting model fit visually

Presenting model fit visually


Presenting model fit numerically

Presenting model fit numerically

  • Theil statistics, for example

    • Based on a breakdown of the mean squared error:

    • 1 = Bias + Variation + Covariation


Presenting model fit numerically1

Presenting model fit numerically


Learning from surprise model behavior

Learning from surprise model behavior

  • Have clear a priori expectations

  • Follow up all unanticipated behavior to appropriate resolution

  • Confirm all behavioral hypotheses through appropriate model tests (Mass 1991/1981)


Tests to reveal and resolve surprise behavior

Tests to reveal and resolve surprise behavior

  • Testing the symmetry of policy response (up and down)

  • Testing large amplitude versus small amplitude response

  • Testing policies entering at different points

  • Testing different patterns of behavior

  • Isolating uniqueness of equilibrium or steady state

  • Understanding forces producing equilibrium positions (Mass 1991/1981)


Summary

Summary

  • Modelers, stakeholders, problem experts, and others in the modeling process pursue validity at every step along the way.

  • We have rigorous traditions guiding model creation, formulation, exploration, and implications.

  • We have a powerful, intimidating battery of tests of model structure and behavior.

  • Model-based conclusions that make it through all this deserve the confidence of everyone in the process.


Epilog

Epilog

  • Reason is itself a matter of faith. It is an act of faith to assert that our thoughts have any relation to reality. (G.K. Chesterton)

  • I have no exquisite reason for’t, but I have reason good enough. (Sir Andrew, Twelfth Night)


  • Login