Using databases to test methods for decision under uncertainty R. T. Haftka, R. I. Rosca, E. Nikolaidis September 2004
Introduction Decision under uncertainty Uncertain event Information Outcome Decision Consequence
Need unbiased and efficient testing approach for methods for decision under uncertainty • Decisions can be sensitive to modeling errors • Look good on paper but disappoint in practice • Tests can reveal weaknesses in methods that are difficult to identify by examining their theoretical foundations
Challenge • Large number of tests to obtain statistically meaningful results expensive to test methods • Proposed testing approach • Database with real life data • Invent decision scenario in which data represent outcomes of uncertain events/decisions • Use data to make decisions and evaluate payoffs
Database from student competition Competitor Rosca
Outline • Testing approach • Demonstration: Comparison of probability and possibility • Results and lessons learnt • Conclusion
1. Testing approach A Step A: Find database Evaluate payoff of decision using entire database Step B: Create decision scenario Step C: Select fitting* (learning) subset No Completed all replications? Fitting subset No: Select another fitting subset Yes Step D: Estimate probability of winning (or expected utility), draw conclusions Model uncertainties, make decision A * Fitting subset contains incomplete information available to decision maker
A. Select database Uncertain variables: Maximum built heights of Rosca and Competitor
B. Create decision scenarioVariables in database represent uncertainties Consequences Competitor's tower collapses Rosca wins Rosca builds stable tower with height, nguar nguar, Rosca losses Decision: Rosca chooses guarantee height, nguar Competitor builds stable tower with height nguar + handicap+1 Rosca losses Rosca's tower collapses Rorca must select guaranteed height and get a handicap in return. She wins if she builds the tower she promised and is at least as high as the competitor’s minus the handicap.
Testing approach: Steps C and D A Step A: Find database Find if Rosca wins or losses all combinations of heights in database Step B: Create decision scenario No Step C: Select fitting subset Completed all replications? No: Select another fitting subset Yes Step D: Estimate probability of winning, draw conclusions Model uncertainties, make decision A
Abundant databases -- different decision problems • Student academic records • Student design projects in undergraduate class on mechanisms
Probabilistic formulation Select nguar To max Gamma and normal PDF for max built height Possibilistic formulation Select nguar To max Triangular possibility distribution 2. Demonstration: Comparison of probability and possibility Probability maximizes right objective function: probability of winning. Fitting error in approximating discrete PDF of max built height with a standard PDF.
Evaluation of consequences of a decision Database: 50 max built heights for Rosca, 90 heights for competitor For each decision, determine consequences for 5090=4,500 competitions Max possible number of decisions = Number of fitting datasets =
3. Results and lessons learnt • Rosca has all data in database
Observations • Little difference in performance of probability and possibility • Fitting error offsets natural advantage of probability
Effect of inflation of distributions • Epistemic uncertainty, inflate distribution
Observations • Inflation decreases likelihood of success • Opposite effect on optimum decision of two methods. • Increase uncertainty in Competitor performance: probabilistic optimum decreases, possibilistic optimum increases • Difference due to non additivity of possibility
Decision scenario accentuating difference in optimum decisions of two methods
Observations • Optimality condition: • Probability; equal derivatives of probability of Rosca’s success and Competitor’s failure. Focuses on hazard easier to control (Rosca fails to deliver guaranteed height). • Possibility; equal possibilities of Rosca’s success and Competitor’s failure. Non additivity of possibility responsible for difference. • Possibility yields counterintuitive results when uncertainty in Competitor’s performance is very high. Optimum dominated by largest uncertainty (Competitor’s performance).
Results and lessons learntRosca has 5 data points Rosca still knows that normal and Gamma distributions fit all data well.
Observations • Methods have practically same performance • Probability: type of probability distribution that fits all data is known • Possibility: decision maker cannot directly use information about type of probability distribution • Hybrid method (uncertainty in maximum built heights modeled using probability distributions; uncertainty in distribution parameters modeled using possibility distributions) could do better
4. Conclusion • Efficient testing approach for methods for decision under uncertainty • Uses real life data • Allows testing methods on many decisions and evaluate consequences for many outcomes of uncertain events • Abundant easily accessible databases from different fields • Can learn useful lessons from tests • Inflation of probability distributions counterproductive. Opposite effect on probabilistic and possibilistic optimum decision. • Identified weakness in possibilistic approach by accentuating difference between probability and possibility. Probability focuses on hazard easier to control; possibility does not. Cause: non additivity of possibility. • Small modeling errors can offset natural advantage of probability that it maximizes right objective function.