Verification and validation as applied epistemology sand 2007 2628
1 / 26

Verification and Validation as Applied Epistemology SAND 2007-2628 - PowerPoint PPT Presentation

  • Uploaded on

Verification and Validation as Applied Epistemology SAND 2007-2628. Laura A. McNamara, George Backus Exploratory Simulation Technologies Timothy G. Trucano Optimization and Uncertainty Quantification Sandia National Laboratories. Supported by Sandia National Laboratories LDRD Program 08-1158.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Verification and Validation as Applied Epistemology SAND 2007-2628' - nodin

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Verification and validation as applied epistemology sand 2007 2628

Verification and Validation as Applied EpistemologySAND 2007-2628

Laura A. McNamara, George Backus

Exploratory Simulation Technologies

Timothy G. Trucano

Optimization and Uncertainty Quantification

Sandia National Laboratories

Supported by Sandia National Laboratories LDRD Program 08-1158

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

Here s the plot
Here’s the plot:

  • Since 1998, the Department of Energy/NNSA National Laboratories have invested millions in strategies for assessing the credibility of computational science and engineering (CSE) models used in high consequence decision making.

  • The answer? There is no answer. There’s a process - and a lot of politics.

  • The importance of model evaluation (verification, validation, uncertainty quantification, and assessment) increases in direct proportion to the significance of the model as input to a decision.

  • Other fields, including computational social science, can learn from the experience of the national laboratories.

  • Some implications for evaluating ‘low cognition agents’

Why applied epistemology
Why “Applied Epistemology?”

  • Epistemology considers the question, How do we know what we [think we] know?

    • Science: What makes Western science special in producing reliable, predictive knowledge about the world?

  • V&V takes epistemology out of the realm of thought and puts it into practice.

    • What is the role of modeling and simulation in the production of reliable, credible scientific knowledge about the world?

    • What steps, investments, practices do I pursue to convince myself that the model I have developed is producing credible knowledge?

  • In large organizations, getting stakeholders to sign up for V&V can be… umm, a challenge.

    • Too hard, too slow, too expensive…

Rom harre on the pitfalls of models
Rom Harre on the pitfalls of models:

“If we regard theories as descriptions… of reality produced by the human imagination, it is clear that there must be some account of the constraints upon that imagination, for the human imaginative faculty is well-known for its capacity to generate mere fantasy: and yet, it is plain that the conceptions of reality which scientists have drawn upon from time to time are not fantasies, though at the end some have been abandoned as unrealistic.”

V&V is a methodology of constraint

Why care about v v for computational social science
Why care about V&V for computational social science?

  • Too much skepticism about models

    • “paradigm-busting” role of modeling and simulation in the social sciences

    • V&V can strengthen claims by tying model back to real world

  • Too little skepticism about models

    • Search in intelligence, defense, policymaking communities for predictive technologies

    • V&V can give ‘reality check’ about where model is (and isn’t) valid

La pr vision c est tr s chic
La prévision, c’est très chic!

  • NSTC: “Network models and analyses … will allow for the development of more effective predictive strategies.”

  • IEEE: “The US intelligence community… is looking for better ways to identify terrorists… and predict where and when they will strike.”

  • DNI: “Predictive network-centric intelligence assessment offers the ability to foresee or warn of threats, be these strategic or tactical and conventional or asymmetric.”

The purpose of computing is not insight
The purpose of computing is not insight.

  • The purpose of computing is to provide “high-performance, full-system, high-fidelity-physics predictive codes to support weapon assessments, renewal process analyses, accident analyses, and certification.” (DOE/DP-99-000010592)

Drivers for v v and uq in cs e
Drivers for V&V and UQ in CS&E

  • Codes have historically played a critical role in certifying high-consequence systems that can’t be repeatedly tested in a fully representative environment

    • Nuclear power plant operations

    • Nuclear waste storage and containment

    • Aerospace applications

  • As computational power increases, so expands the demand for high-fidelity, predictive computational simulations

    • Nuclear weapons performance, safety, reliability

    • Should drive demands for higher credibility

V v definitions
V&V Definitions


    • The process of determining that a computational software implementation correctly represents a model of a physical process… or

    • The process of determining that the equations are solved correctly


    • The process of determining the degree to which a computer model is an accurate representation of the real world from the perspective of the intended model applications…or

    • The process of determining that equations are correct

V v is a methodology
V&V is a methodology.




  • Requirements and planning




Validation Experiments


Design, Execution

& Analysis







Validation Metrics








& Credibility





Let s simplify and also bring this up to date
Let’s simplify and also bring this up to date:


Risk-Informed Decision Analysis (RIDA)

  • V&V results mean what?

  • Use within decisionenvironment?




  • Metrics

  • Assessment



  • Application(s)

  • Requirements

Simulation as a third corner of science





Code Verification

Simulation as a third corner of science



data valid?

Experimental design,

Simulation predictions

Observation and Analysis;

hypothesis formulation

Computer Simulation

Conceptual Model

Model development and

code predictions

Ang, Trucano, Luginbuhl 1998

Too hard too slow too expensive
“Too hard, too slow, too expensive…”

  • CS&E

    • Computational science and engineering is mature, capable, and reliable for the responsibilities being demanded of it.

    • I’ve used this code for such a long time that I trust it. I turn the knobs, I calibrate, and I get the right answer.

    • V&V is… (see above)

  • CSS

    • Models don’t make decisions; they provide insights to decision makers

    • The code isn’t ready for V&V. When it’s more mature, we’ll do V&V.

    • V&V for this code? We don’t have the money. Besides, it’s up to the analyst to use the code right.

    • If V&V reveals problems with the code we’ve invested so much money, time, energy… (finish the sentence).

Busting the rational utility maximizing decision maker
Busting the Rational, Utility Maximizing Decision Maker

  • Individuals gather, weigh, process information to maximize utility (rational behavior)

    • Problem: this assumes a lot of cognitive power in very high dimensional environments

    • Problem: classical economic theory doesn’t predict business cycles (or other cycles) very well

  • What if agents are more efficient in processing information, making decisions, than traditionally assumed?

Low cognition agents

  • DATA: Aggregate data seem to follow an exponential or power law distribution

    • This implies something about the relationship between agent and structure

  • MODELING THEORY: ‘Degree of cognition that needs to be assigned to agents decreases as system matures’


    • People have minimal cognitive ability

    • OR - People develop heuristics that help them simplify complex information environments

    • OR - ‘Structure’ establishes boundaries on quality, quantity of information required for agents to make a ‘good’ decision

    • OR - None of the above – ‘low cognition agents’ don’t actually say much about agents themselves.

Low cognition agents1





Code Verification

Low cognition agents



data valid?

Experimental design,

Simulation predictions

Observation and Analysis;

hypothesis formulation

Computer Simulation

Conceptual Model

Implementation of

conceptual model in code

Mathematics works right?

Ang, Trucano, Luginbuhl 1998


  • Ang, J., Trucano, T. 1998. Confidence in ASCI Scientific Simulations. Albuquerque, NM: Sandia National Laboratories. SAND 98-1525c.

  • Axelrod, R. 2003. Advancing the Art of Simulation in the Social Sciences. Japanese Journal for Information Management Systems.12(3)

  • Goldstein, H. 2006.Modeling terrorists: New simulators could help intelligence analysts think like the enemy. IEEE Spectrum September: 34-43.

  • Harre, H.R. 2003. Modeling: Gateway to the Unknown. Amsterdam, NL: Elsevier Press.

  • Marks, Robert E. 2003. ‘Coffee, Segregation, Energy and the Law: Validating Simulation Models.’ GET FULL CITATION

  • Trucano, T., Garasi, C., Mehlhorn, T. 2005. ALEGRA-HEDP Validation Strategy. Albuquerque, NM: Sandia National Laboratories (SAND 2005-6890).

  • Oberkampf, W.L., Trucano, T. 2007. Verification and Validation Benchmarks. Albuquerque, New Mexico: Sandia National Laboratories (SAND 2007-0853).

  • Pilch, M., Trucano, T., Moya, J., Groehlich, G., Hodges, A., Peercy, D. 2000. “Guidelines for Sandia ASCI Verification and Validation Plans – Content and Format: Version 2.0.” Albuquerque, NM: Sandia National Laboratories, SAND 2000-3101.

  • Smith, T. J.2007. “Predictive Network Centric Intelligence: Toward a Total Systems Transformation of Analysis and Assessment.” Washington, DC: Director of National Intelligence.

Agency and structure
Agency and structure

  • Where does society reside, if not in the individual?

  • How is the social reproduced across generations?

  • Is the social world comprised by parts whose dynamics are structured in terms of an externally imposed cause (structuralism)?

  • To what extent are individual actions constrained by society?

  • To what extent do the actions of individuals change society?

  • How much of a rebel am I, really?

  • Nature or nurture? What is free will?

  • Do low cognition agents have anything to say about these questions?

Css vs cse

High-consensus ‘laws,’ rules, theories exist

Implemented mathematically with (relative) ease

Theory is explanatory and predictive

Multiple theories explain the same set of phenomena

Theories expressed in narrative form

Theories are explanatory and descriptive


So where is the computational social science community
So where is the computational social science community?

  • Axelrod: Does the program correctly implement the model? (internal validity)

  • Carley: processes and techniques for addressing comparability between simulated world and ‘real’ world … (external validation)

  • Marks: How successfully the model’s output exhibits the historical behaviors of the real world target system (‘output validation’, cf Manson 2002)

Q why do we create models
Q: Why do we create models?

  • Kinds of models

    • To highlight features of a phenomenon we have observed (to describe, explain, predict)

    • As our observations mature, so can our conceptual models

      • The Ptolemaic universe

      • The Copernican universe

    • To represent a conception of a phenomenon we have not yet observed

      • Superstrings in cosmological physics

  • Roles that models play

    • Models fix a mental representation, collective or otherwise, of a phenomenon occurring in the world around us

    • Models are frameworks for organizing inquiry

    • Models enable knowledge to evolve

A: Because we can’t do science without them.

Uses of agent based simulations
Uses of agent-based simulations

  • Explain a phenomenon, explore a phenomenon, understand interactions that produce a phenomenon (Marks 2003)

  • Insight into system control, make predictions, derive general principles (Haefner)

  • Prediction, performance, training (flight simulators), entertainment, education (SimCity), existence proofs, discovery, gedankenexperiment (Axelrod 2003)


  • CSE: Computational Science and Engineering

  • CSS: Computational Social Science (to include agent-based models)

  • V&V: Verification and Validation

  • UQ: Uncertainty Quantification