- 341 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'AMBIGUITY' - Olivia

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Pastiche from

2 day course on Expert Judgment

AMBIGUITY

Roger Cooke

Resources for the Future

Dept. Math, Delft Univ. of Technology

Dec 14, 2009

http://dutiosc.twi.tudelft.nl/~risk/

INDECISION

AMBIGUITY

INDECISION

“Uncertainty from random sampling ...omits important sources of uncertainty” NRC(2003)All cause mortality, percent increase per 1 μg/m3 increase in PM2.5 (RESS-PM25.pdf)

AMBIGUITY

Very Different Guidelines:

The story you hear today is NOT the only story

INDECISION

History Structured Expert Judgment in Risk Analysis- WASH 1400 (Rasmussen Report, 1975)
- IEEE Std 500 (1977)
- Canvey Island (1978)
- NUREG 1150 (1989)
- T-book (Swedish Reliability Data Base 1994)
- USNRC-EU (1995-1997)
- Guidance on Uncertainty and Use of Experts. NUREG/CR-6372, 1997
- Procedures Guide for Structured Expert Judgment, EUR 18820EN, 2000

AMBIGUITY

INDECISION

Goals of an EJ study- Census
- Political consensus
- Rational consensus

EJCoursenotes_review-EJ-literature.doc

AMBIGUITY

INDECISION

EJ for RATIONAL CONSENSUS:RESS-TUDdatabase.pdfParties pre-commit to a method which satisfies necessary conditions for scientific method:

Traceability/accountability

Neutrality (don’t encourage untruthfulness)

Fairness (ab initio, all experts equal)

Empirical control (performance meas’t)

Withdrawal post hoc incurs burden of proof.

Goal: comply with principals and combine experts’ judgments to get a Good Probability Assessor

“Classical Model for EJ”

AMBIGUITY

INDECISION

What is a GOOD subjective probability assessor?- Calibration, statistical likelihood
- Are the expert’s probability statements statistically accurate? P-value of statistical test
- Informativeness
- Probability mass concentrated in a small region, relative to background measure
- Nominal values near truth
- ?

AMBIGUITY

INDECISION

Combined ScoreLong run strictly proper scoring rule- Calibration information cutoff

Requires that experts assess uncertainty for variables from their field for which we (will) know the true values:

Calibration / performance / seed variables

any expert, or combination of experts, can be regarded as a statistical hypothesis

AMBIGUITY

INDECISION

Calibration questions for PM2.5RESS-PM25.pdfIn London 2000, weekly average PM10 was18.4 μg/m3. What is the ratio:

# non-accidental deaths in the week with the highest average PM10 concentration (33.4 μg/m3)

Weekly average # non-accidental deaths.

5% :_______ 25%:_______ 50% :_______ 75%:________95%:________

AMBIGUITY

INDECISION

Expert Judgment is NOT KnowledgeObservations – NOT EJ methods - produces agreement among experts

EJ is for quantifying ....not removing..... uncertainty.

AMBIGUITY

INDECISION

Experts confidence does NOT predict performanceVery informative assessors may be statistically least accurate

PM25-Range-graphs.doc

EJ is Not (just) an academic exercise

Invasive Speciesin Great Lakes (EPA, NOAA)- Commercial Fish Landings
- Sport Fishing Expenditures
- Raw Water User Costs
- Wildlife Watching Effort

Treat Experts as Statistical Hypotheses

P-Value Too Low for Acceptance

Nuclear Power Plants: 118K

/facility/year

Other facilities: 30K

/facility/year

Wildlife Watching

0.8%

2006 Median Percent Reductions & Costs13%

Sport Fishing

35%

Commercial Fishing

23%

33%

13%

18%

21%

18%

15%

11%

AMBIGUITY

INDECISION

Experts are sometimes well calibratedAMS-OPTION-TRADERS-RANGE-GRAPHS.doc

realestate-range graphs.doc

Sometimes not

GL-invasive-species-range-graphs.doc

Campy-range-graphs.doc

Ambiguity

Indecision

3

Out-of-sample ValidationRESS_response2comments.pdf13 studies with 14 seed vbls, split, initialize on one half, predict other half

AMBIGUITY

INDECISION

Experts like performance assessmentAsk them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf , Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf

Separate scientific assessment of uncertainty from decision making

Univariate distributions is so 90’s

Now its all about Dependence

Causal Model for Air Transport Safety1400 nodes, functional and probabilistic

ESDs

Critical Accident

Human

Accidents types

Current Research Agenda

- Dependence elicitation
- Dependence modeling
- Computing large networks (eg wo normal copula)
- Emerging features,
- micro macro correlation
- Tail dependence

Thank YouWilly AspinallTim BedfordJan van NoortwijkTom MazzuchiDorota KurowickaDavid LodgeRamanan LaxminarayanAbby ColsonHarry Joemany studentsgod knows who else

Ambiguity

Indecision

4

FAQ’s(1)- From an expert: I don't know that
- Response: No one knows, if someone knew we would not need to do an expert judgment exercise. We are tying to capture your uncertainty about this variable. If you are very uncertain then you should choose very wide confidence bounds.
- From an expert: I can't assess that unless you give me more information.
- Response: The information given corresponds with the assumptions of the study. We are trying to get your uncertainty conditional on the assumptions of the study. If you prefer to think of uncertainty conditional on other factors, then you must try to unconditionalize and fold the uncertainty over these other factors into your assessment.
- From an expert: I am not the best expert for that.
- Response: We don't know who are the best experts. Sometimes the people with the most detailed knowledge are not the best at quantifying their uncertainty.
- From an expert:Does that answer look OK?
- Response: You are the expert, not me.
- From the problem owner:So you are going to score these experts like school children?
- Response: If this is not a serious matter for you, then forget it. If it is serious, then we must take the quantification of uncertainty seriously. Without scoring we can never validate our experts or the combination of their assessments.

Ambiguity

Indecision

4

FAQ’s(2)- From the problem owner:The experts will never stand for it.
- Response We've done it many times, the experts actually like it.
- From the problem owner:Expert number 4 gave crazy assessments, who was that guy?
- Response: You are paying for the study, you own the data, and if you really want to know I will tell you. But you don't need to know, and knowing will not make things easier for you. Reflect first whether you really want to know this.
- From the problem owner: How can I give an expert weight zero?
- Response:Zero weight does not mean zero value. It simply means that this expert's knowledge was already contributed by other experts and adding this expert would only add a bit of noise. The value of unweighted experts is seen in the robustness of our answers against loss of experts. Everyone understands this when it is properly explained.
- From the problem owner:How can I give weight one to a single expert?
- Response: By giving all the others weight zero, see previous response.
- From the problem owner: I prefer to use the equal weight combination.
- Response: So long as the calibration of the equal weight combination is acceptable, there is no scientific objection to doing this. Our job as analyst is to indicate the best combination, according to the performance criteria, and to say what other combinations are scientifically acceptable.

Ambiguity

Indecision

3

“ In the first few weeks of the Montserrat crisis there was perhaps, at times, some unwarranted scientific dogmatism about what might or might not happen at the volcano, especially in terms of it turning magmatic and explosive. The confounding effects of these diverging, categorical stances were then compounded for a short while by an overall diminution in communication between scientists and the various civil authorities. The result was a dip in the confidence of the authorities in the MVO team and, with it, some loss of public credibility; this was not fully restored until later, when a consensual approach was achieved. “

Aspinall et al The Montserrat Volcano Observatory: its evolution, organization, rôle and activities.

Download Presentation

Connecting to Server..