AMBIGUITY

1. UNCERTAINTY Pastiche from 2 day course on Expert Judgment AMBIGUITY Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology Dec 14, 2009 http://dutiosc.twi.tudelft.nl/~risk/ INDECISION

2. UNCERTAINTY AMBIGUITY INDECISION “Uncertainty from random sampling ...omits important sources of uncertainty” NRC(2003) All cause mortality, percent increase per 1 μg/m3 increase in PM2.5 (RESS-PM25.pdf)

3. UNCERTAINTY AMBIGUITY Very Different Guidelines: The story you hear today is NOT the only story INDECISION History Structured Expert Judgment in Risk Analysis • WASH 1400 (Rasmussen Report, 1975) • IEEE Std 500 (1977) • Canvey Island (1978) • NUREG 1150 (1989) • T-book (Swedish Reliability Data Base 1994) • USNRC-EU (1995-1997) • Guidance on Uncertainty and Use of Experts. NUREG/CR-6372, 1997 • Procedures Guide for Structured Expert Judgment, EUR 18820EN, 2000

4. UNCERTAINTY AMBIGUITY INDECISION Goals of an EJ study • Census • Political consensus • Rational consensus EJCoursenotes_review-EJ-literature.doc

5. UNCERTAINTY AMBIGUITY INDECISION EJ for RATIONAL CONSENSUS:RESS-TUDdatabase.pdf Parties pre-commit to a method which satisfies necessary conditions for scientific method: Traceability/accountability Neutrality (don’t encourage untruthfulness) Fairness (ab initio, all experts equal) Empirical control (performance meas’t) Withdrawal post hoc incurs burden of proof. Goal: comply with principals and combine experts’ judgments to get a Good Probability Assessor “Classical Model for EJ”

6. UNCERTAINTY AMBIGUITY INDECISION What is a GOOD subjective probability assessor? • Calibration, statistical likelihood • Are the expert’s probability statements statistically accurate? P-value of statistical test • Informativeness • Probability mass concentrated in a small region, relative to background measure • Nominal values near truth • ?

7. UNCERTAINTY AMBIGUITY INDECISION Combined ScoreLong run strictly proper scoring rule • Calibration  information  cutoff Requires that experts assess uncertainty for variables from their field for which we (will) know the true values: Calibration / performance / seed variables any expert, or combination of experts, can be regarded as a statistical hypothesis

8. The Ill Advised Bayesian

9. UNCERTAINTY AMBIGUITY INDECISION Calibration questions for PM2.5RESS-PM25.pdf In London 2000, weekly average PM10 was18.4 μg/m3. What is the ratio: # non-accidental deaths in the week with the highest average PM10 concentration (33.4 μg/m3)  Weekly average # non-accidental deaths. 5% :_______ 25%:_______ 50% :_______ 75%:________95%:________

10. EJ Bumperstickers

11. UNCERTAINTY AMBIGUITY INDECISION Expert Judgment is NOT Knowledge Observations – NOT EJ methods - produces agreement among experts EJ is for quantifying ....not removing..... uncertainty.

12. UNCERTAINTY AMBIGUITY INDECISION Experts CAN quantify uncertainty as subjective probability

13. UNCERTAINTY AMBIGUITY INDECISION Including 6700 calibration variables

14. UNCERTAINTY AMBIGUITY INDECISION Studies since 2006

15. UNCERTAINTY AMBIGUITY INDECISION Experts confidence does NOT predict performance Very informative assessors may be statistically least accurate PM25-Range-graphs.doc

16. EJ is Not (just) an academic exercise Invasive Speciesin Great Lakes (EPA, NOAA) • Commercial Fish Landings • Sport Fishing Expenditures • Raw Water User Costs • Wildlife Watching Effort

17. Treat Experts as Statistical Hypotheses P-Value Too Low for Acceptance

18. Raw Water Nuclear Power Plants: 118K /facility/year Other facilities: 30K /facility/year Wildlife Watching 0.8% 2006 Median Percent Reductions & Costs 13% Sport Fishing 35% Commercial Fishing 23% 33% 13% 18% 21% 18% 15% 11%

19. UNCERTAINTY AMBIGUITY INDECISION Experts are sometimes well calibrated AMS-OPTION-TRADERS-RANGE-GRAPHS.doc realestate-range graphs.doc Sometimes not GL-invasive-species-range-graphs.doc Campy-range-graphs.doc

20. UNCERTAINTY AMBIGUITY INDECISION We CAN do better than equal weightingRESS-TUDdatabase.pdf

21. Uncertainty Ambiguity Indecision 3 Out-of-sample ValidationRESS_response2comments.pdf 13 studies with  14 seed vbls, split, initialize on one half, predict other half

22. UNCERTAINTY AMBIGUITY INDECISION Post hoc validationreal estate risk

23. UNCERTAINTY AMBIGUITY INDECISION Experts like performance assessment Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf , Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf Separate scientific assessment of uncertainty from decision making

24. Univariate distributions is so 90’s Now its all about Dependence

25. Elicit (conditional) copulae

26. Elicit Dependence Structure

27. IQ and fish consumption

28. Dioxin and dentition

29. Causal Model for Air Transport Safety1400 nodes, functional and probabilistic ESDs Critical Accident Human Accidents types

30. Current Research Agenda • Dependence elicitation • Dependence modeling • Computing large networks (eg wo normal copula) • Emerging features, • micro  macro correlation • Tail dependence

31. Thank YouWilly AspinallTim BedfordJan van NoortwijkTom MazzuchiDorota KurowickaDavid LodgeRamanan LaxminarayanAbby ColsonHarry Joemany studentsgod knows who else

32. Tail dependence and aggregation

33. UNCERTAINTY AMBIGUITY INDECISION EJ studies since 2006

34. Uncertainty Ambiguity Indecision 4 FAQ’s(1) • From an expert: I don't know that • Response: No one knows, if someone knew we would not need to do an expert judgment exercise. We are tying to capture your uncertainty about this variable. If you are very uncertain then you should choose very wide confidence bounds. • From an expert: I can't assess that unless you give me more information. • Response: The information given corresponds with the assumptions of the study. We are trying to get your uncertainty conditional on the assumptions of the study. If you prefer to think of uncertainty conditional on other factors, then you must try to unconditionalize and fold the uncertainty over these other factors into your assessment. • From an expert: I am not the best expert for that. • Response: We don't know who are the best experts. Sometimes the people with the most detailed knowledge are not the best at quantifying their uncertainty. • From an expert:Does that answer look OK? • Response: You are the expert, not me. • From the problem owner:So you are going to score these experts like school children? • Response: If this is not a serious matter for you, then forget it. If it is serious, then we must take the quantification of uncertainty seriously. Without scoring we can never validate our experts or the combination of their assessments.

35. Uncertainty Ambiguity Indecision 4 FAQ’s(2) • From the problem owner:The experts will never stand for it. • Response We've done it many times, the experts actually like it. • From the problem owner:Expert number 4 gave crazy assessments, who was that guy? • Response: You are paying for the study, you own the data, and if you really want to know I will tell you. But you don't need to know, and knowing will not make things easier for you. Reflect first whether you really want to know this. • From the problem owner: How can I give an expert weight zero? • Response:Zero weight does not mean zero value. It simply means that this expert's knowledge was already contributed by other experts and adding this expert would only add a bit of noise. The value of unweighted experts is seen in the robustness of our answers against loss of experts. Everyone understands this when it is properly explained. • From the problem owner:How can I give weight one to a single expert? • Response: By giving all the others weight zero, see previous response. • From the problem owner: I prefer to use the equal weight combination. • Response: So long as the calibration of the equal weight combination is acceptable, there is no scientific objection to doing this. Our job as analyst is to indicate the best combination, according to the performance criteria, and to say what other combinations are scientifically acceptable.

36. Uncertainty Ambiguity Indecision 3 “ In the first few weeks of the Montserrat crisis there was perhaps, at times, some unwarranted scientific dogmatism about what might or might not happen at the volcano, especially in terms of it turning magmatic and explosive. The confounding effects of these diverging, categorical stances were then compounded for a short while by an overall diminution in communication between scientists and the various civil authorities. The result was a dip in the confidence of the authorities in the MVO team and, with it, some loss of public credibility; this was not fully restored until later, when a consensual approach was achieved. “ Aspinall et al The Montserrat Volcano Observatory: its evolution, organization, rôle and activities.

37. UNCERTAINTY AMBIGUITY INDECISION 1 Using Uncertainty to Manage Vulcano risk responseAspinall et al Geol Soc _.pdf

38. UNCERTAINTY AMBIGUITY INDECISION British Airways Pilots study Range graphs

39. UNCERTAINTY What Is? AMBIGUITY What means? INDECISION What’s best?

40. UNCERTAINTY Experts’ job AMBIGUITY Analysts’ job INDECISION Stakeholder/problem owners’ job