Sensitivity and Importance Analysis

1. Sensitivity and Importance Analysis Charles Yoe cyoe1@verzion.net

2. Sensitivity Analysis Defined • Study of how the variation in the output of a model can be apportioned, qualitatively or quantitatively, to different “sources of variation” in the inputs for the purpose of increasing confidence in the analysis • Include assumptions • Input uncertainty • Scenario/model uncertainty

3. The Point • Complex analysis may have dozens of input and output variables that are linked by a system of equations • Analysts and decision makers must understand the relative importance of the components of an analysis • Some outcomes and decisions are sensitive to minor changes in assumptions and input values

4. Sensitivity Analysis • If it is not obvious which assumptions and uncertainties most affect outputs, conclusions and decisions the purpose of sensitivity analysis is to systematically find this out

5. Systematic Investigation of… • Future scenarios • Model parameters • Model inputs • Assumptions • Model functional form

6. Assumptions Sensitivity • List the key assumptions (scenarios) of your analysis • Explore what happens as you change/drop each one individually • Do your answers change? • Challenging assumptions can be effective sensitivity analysis

7. Input Sensitivity • Parameter-how sensitive is our output to forecast error or other changes in inputs? Unexpected change or error • Decision variables (Inputs we control)-might changes in our decisions/actions improve our outputs

8. Sensitivity Analysis Methods • Deterministic one-at-a-time analysis of each factor • Deterministic joint analysis • Scenario analysis • Subjective estimates • Parametric analysis--range of values • Probabilistic analysis can be used for importance analysis

9. One-At-A-Time Analysis • Hold each parameter constant • Expected value • Representative value • Let one input vary • Assumption • Input • Parameter • Common, useful, dangerous

10. One-At-A-Time Analysis • Do not equate magnitude with influence • A=U(107,108), B=U(2,6) • C = A + B; A dominates • C = AB; B dominates

11. One-At-A-Time Analysis • Dependence and branching in model creates flaws with this logic If A<50 then C = B + 1 Else C = B100 What value do we set A equal to?

12. Joint Analysis • Change combinations of variables at same time • Enables analysts to take dependencies explicitly into account • Can have same limitations as OAAT analysis

13. Subjective Estimates • Subjective estimates of uncertain values can be used to identify threshold values of importance to the risk assessment

14. Range of Values • A specific (not subjective) range of values is used • E.g., 10th, 50th, 90th percentiles • Ceteris paribus approach • All possible combinations approach • All 10th percentiles, 10th with 90th and so on

15. Importance Analysis • How much does each model input contribute to the variation in the output? • Typically a few key inputs account for most output variation • These are your important inputs. • Not particularly good at identifying nonlinear or multivariate relationships

19. Advanced Statistical Methods • Apportion variation in output to inputs via • Regression analysis • Analysis of variance • Response surface methods • Fourier amplitude sensitivity test (FAST) • Mutual information index (MII) • Classification and regression trees (CART)

20. So What? • When decision is sensitive to changes or uncertainties within realm of possibility then more precision and additional information may be required • More data (research) • Better models • Conservative risk management

21. Take Away Points • “What if” analysis is essential to good risk assessment • Systematic investigations of model parameters, model inputs, assumptions, model functional form • Essential to good risk management

22. Questions? Charles Yoe, Ph.D. cyoe1@verizon.net