Using databases to test methods for decision under uncertainty

114 Views

Download Presentation
## Using databases to test methods for decision under uncertainty

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Using databases to test methods for decision under**uncertainty R. T. Haftka, R. I. Rosca, E. Nikolaidis September 2004**Introduction Decision under uncertainty**Uncertain event Information Outcome Decision Consequence**Need unbiased and efficient testing approach for methods for**decision under uncertainty • Decisions can be sensitive to modeling errors • Look good on paper but disappoint in practice • Tests can reveal weaknesses in methods that are difficult to identify by examining their theoretical foundations**Challenge**• Large number of tests to obtain statistically meaningful results expensive to test methods • Proposed testing approach • Database with real life data • Invent decision scenario in which data represent outcomes of uncertain events/decisions • Use data to make decisions and evaluate payoffs**Database from student competition**Competitor Rosca**Outline**• Testing approach • Demonstration: Comparison of probability and possibility • Results and lessons learnt • Conclusion**1. Testing approach**A Step A: Find database Evaluate payoff of decision using entire database Step B: Create decision scenario Step C: Select fitting* (learning) subset No Completed all replications? Fitting subset No: Select another fitting subset Yes Step D: Estimate probability of winning (or expected utility), draw conclusions Model uncertainties, make decision A * Fitting subset contains incomplete information available to decision maker**A. Select database**Uncertain variables: Maximum built heights of Rosca and Competitor**B. Create decision scenarioVariables in database represent**uncertainties Consequences Competitor's tower collapses Rosca wins Rosca builds stable tower with height, nguar nguar, Rosca losses Decision: Rosca chooses guarantee height, nguar Competitor builds stable tower with height nguar + handicap+1 Rosca losses Rosca's tower collapses Rorca must select guaranteed height and get a handicap in return. She wins if she builds the tower she promised and is at least as high as the competitor’s minus the handicap.**Testing approach: Steps C and D**A Step A: Find database Find if Rosca wins or losses all combinations of heights in database Step B: Create decision scenario No Step C: Select fitting subset Completed all replications? No: Select another fitting subset Yes Step D: Estimate probability of winning, draw conclusions Model uncertainties, make decision A**Abundant databases -- different decision problems**• Student academic records • Student design projects in undergraduate class on mechanisms**Probabilistic formulation**Select nguar To max Gamma and normal PDF for max built height Possibilistic formulation Select nguar To max Triangular possibility distribution 2. Demonstration: Comparison of probability and possibility Probability maximizes right objective function: probability of winning. Fitting error in approximating discrete PDF of max built height with a standard PDF.**Evaluation of consequences of a decision**Database: 50 max built heights for Rosca, 90 heights for competitor For each decision, determine consequences for 5090=4,500 competitions Max possible number of decisions = Number of fitting datasets =**3. Results and lessons learnt**• Rosca has all data in database**Observations**• Little difference in performance of probability and possibility • Fitting error offsets natural advantage of probability**Effect of inflation of distributions**• Epistemic uncertainty, inflate distribution**Observations**• Inflation decreases likelihood of success • Opposite effect on optimum decision of two methods. • Increase uncertainty in Competitor performance: probabilistic optimum decreases, possibilistic optimum increases • Difference due to non additivity of possibility**Decision scenario accentuating difference in optimum**decisions of two methods**Observations**• Optimality condition: • Probability; equal derivatives of probability of Rosca’s success and Competitor’s failure. Focuses on hazard easier to control (Rosca fails to deliver guaranteed height). • Possibility; equal possibilities of Rosca’s success and Competitor’s failure. Non additivity of possibility responsible for difference. • Possibility yields counterintuitive results when uncertainty in Competitor’s performance is very high. Optimum dominated by largest uncertainty (Competitor’s performance).**Results and lessons learntRosca has 5 data points**Rosca still knows that normal and Gamma distributions fit all data well.**Observations**• Methods have practically same performance • Probability: type of probability distribution that fits all data is known • Possibility: decision maker cannot directly use information about type of probability distribution • Hybrid method (uncertainty in maximum built heights modeled using probability distributions; uncertainty in distribution parameters modeled using possibility distributions) could do better**4. Conclusion**• Efficient testing approach for methods for decision under uncertainty • Uses real life data • Allows testing methods on many decisions and evaluate consequences for many outcomes of uncertain events • Abundant easily accessible databases from different fields • Can learn useful lessons from tests • Inflation of probability distributions counterproductive. Opposite effect on probabilistic and possibilistic optimum decision. • Identified weakness in possibilistic approach by accentuating difference between probability and possibility. Probability focuses on hazard easier to control; possibility does not. Cause: non additivity of possibility. • Small modeling errors can offset natural advantage of probability that it maximizes right objective function.