slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
AMBIGUITY PowerPoint Presentation
Download Presentation
AMBIGUITY

Loading in 2 Seconds...

play fullscreen
1 / 41

AMBIGUITY - PowerPoint PPT Presentation


  • 341 Views
  • Uploaded on

UNCERTAINTY. Pastiche from 2 day course on Expert Judgment. AMBIGUITY. Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology Dec 14, 2009 http://dutiosc.twi.tudelft.nl/~risk/. INDECISION. UNCERTAINTY. AMBIGUITY. INDECISION.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'AMBIGUITY' - Olivia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

UNCERTAINTY

Pastiche from

2 day course on Expert Judgment

AMBIGUITY

Roger Cooke

Resources for the Future

Dept. Math, Delft Univ. of Technology

Dec 14, 2009

http://dutiosc.twi.tudelft.nl/~risk/

INDECISION

uncertainty from random sampling omits important sources of uncertainty nrc 2003

UNCERTAINTY

AMBIGUITY

INDECISION

“Uncertainty from random sampling ...omits important sources of uncertainty” NRC(2003)

All cause mortality, percent increase per 1 μg/m3 increase in PM2.5 (RESS-PM25.pdf)

history structured expert judgment in risk analysis

UNCERTAINTY

AMBIGUITY

Very Different Guidelines:

The story you hear today is NOT the only story

INDECISION

History Structured Expert Judgment in Risk Analysis
  • WASH 1400 (Rasmussen Report, 1975)
  • IEEE Std 500 (1977)
  • Canvey Island (1978)
  • NUREG 1150 (1989)
  • T-book (Swedish Reliability Data Base 1994)
  • USNRC-EU (1995-1997)
  • Guidance on Uncertainty and Use of Experts. NUREG/CR-6372, 1997
  • Procedures Guide for Structured Expert Judgment, EUR 18820EN, 2000
goals of an ej study

UNCERTAINTY

AMBIGUITY

INDECISION

Goals of an EJ study
  • Census
  • Political consensus
  • Rational consensus

EJCoursenotes_review-EJ-literature.doc

ej for rational consensus ress tuddatabase pdf

UNCERTAINTY

AMBIGUITY

INDECISION

EJ for RATIONAL CONSENSUS:RESS-TUDdatabase.pdf

Parties pre-commit to a method which satisfies necessary conditions for scientific method:

Traceability/accountability

Neutrality (don’t encourage untruthfulness)

Fairness (ab initio, all experts equal)

Empirical control (performance meas’t)

Withdrawal post hoc incurs burden of proof.

Goal: comply with principals and combine experts’ judgments to get a Good Probability Assessor

“Classical Model for EJ”

what is a good subjective probability assessor

UNCERTAINTY

AMBIGUITY

INDECISION

What is a GOOD subjective probability assessor?
  • Calibration, statistical likelihood
    • Are the expert’s probability statements statistically accurate? P-value of statistical test
  • Informativeness
    • Probability mass concentrated in a small region, relative to background measure
  • Nominal values near truth
  • ?
combined score long run strictly proper scoring rule

UNCERTAINTY

AMBIGUITY

INDECISION

Combined ScoreLong run strictly proper scoring rule
  • Calibration  information  cutoff

Requires that experts assess uncertainty for variables from their field for which we (will) know the true values:

Calibration / performance / seed variables

any expert, or combination of experts, can be regarded as a statistical hypothesis

calibration questions for pm 2 5 ress pm25 pdf

UNCERTAINTY

AMBIGUITY

INDECISION

Calibration questions for PM2.5RESS-PM25.pdf

In London 2000, weekly average PM10 was18.4 μg/m3. What is the ratio:

# non-accidental deaths in the week with the highest average PM10 concentration (33.4 μg/m3)



Weekly average # non-accidental deaths.

5% :_______ 25%:_______ 50% :_______ 75%:________95%:________

exp er t judgment is not knowledge

UNCERTAINTY

AMBIGUITY

INDECISION

Expert Judgment is NOT Knowledge

Observations – NOT EJ methods - produces agreement among experts

EJ is for quantifying ....not removing..... uncertainty.

experts confidence does not predict performance

UNCERTAINTY

AMBIGUITY

INDECISION

Experts confidence does NOT predict performance

Very informative assessors may be statistically least accurate

PM25-Range-graphs.doc

i nvasive species in great lakes epa noaa

EJ is Not (just) an academic exercise

Invasive Speciesin Great Lakes (EPA, NOAA)
  • Commercial Fish Landings
  • Sport Fishing Expenditures
  • Raw Water User Costs
  • Wildlife Watching Effort
treat experts as statistical hypotheses
Treat Experts as Statistical Hypotheses

P-Value Too Low for Acceptance

2006 median percent reductions costs

Raw Water

Nuclear Power Plants: 118K

/facility/year

Other facilities: 30K

/facility/year

Wildlife Watching

0.8%

2006 Median Percent Reductions & Costs

13%

Sport Fishing

35%

Commercial Fishing

23%

33%

13%

18%

21%

18%

15%

11%

experts are sometimes well calibrated

UNCERTAINTY

AMBIGUITY

INDECISION

Experts are sometimes well calibrated

AMS-OPTION-TRADERS-RANGE-GRAPHS.doc

realestate-range graphs.doc

Sometimes not

GL-invasive-species-range-graphs.doc

Campy-range-graphs.doc

out of sample validation ress response2comments pdf

Uncertainty

Ambiguity

Indecision

3

Out-of-sample ValidationRESS_response2comments.pdf

13 studies with  14 seed vbls, split, initialize on one half, predict other half

experts like performance assessment

UNCERTAINTY

AMBIGUITY

INDECISION

Experts like performance assessment

Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf , Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf

Separate scientific assessment of uncertainty from decision making

univariate distributions is so 90 s
Univariate distributions is so 90’s

Now its all about Dependence

causal model for air transport safety 1400 nodes functional and probabilistic
Causal Model for Air Transport Safety1400 nodes, functional and probabilistic

ESDs

Critical Accident

Human

Accidents types

current research agenda
Current Research Agenda
  • Dependence elicitation
  • Dependence modeling
  • Computing large networks (eg wo normal copula)
  • Emerging features,
    • micro  macro correlation
    • Tail dependence
slide31

Thank YouWilly AspinallTim BedfordJan van NoortwijkTom MazzuchiDorota KurowickaDavid LodgeRamanan LaxminarayanAbby ColsonHarry Joemany studentsgod knows who else

faq s 1

Uncertainty

Ambiguity

Indecision

4

FAQ’s(1)
  • From an expert: I don't know that
    • Response: No one knows, if someone knew we would not need to do an expert judgment exercise. We are tying to capture your uncertainty about this variable. If you are very uncertain then you should choose very wide confidence bounds.
  • From an expert: I can't assess that unless you give me more information.
    • Response: The information given corresponds with the assumptions of the study. We are trying to get your uncertainty conditional on the assumptions of the study. If you prefer to think of uncertainty conditional on other factors, then you must try to unconditionalize and fold the uncertainty over these other factors into your assessment.
  • From an expert: I am not the best expert for that.
    • Response: We don't know who are the best experts. Sometimes the people with the most detailed knowledge are not the best at quantifying their uncertainty.
  • From an expert:Does that answer look OK?
    • Response: You are the expert, not me.
  • From the problem owner:So you are going to score these experts like school children?
    • Response: If this is not a serious matter for you, then forget it. If it is serious, then we must take the quantification of uncertainty seriously. Without scoring we can never validate our experts or the combination of their assessments.
faq s 2

Uncertainty

Ambiguity

Indecision

4

FAQ’s(2)
  • From the problem owner:The experts will never stand for it.
    • Response We've done it many times, the experts actually like it.
  • From the problem owner:Expert number 4 gave crazy assessments, who was that guy?
    • Response: You are paying for the study, you own the data, and if you really want to know I will tell you. But you don't need to know, and knowing will not make things easier for you. Reflect first whether you really want to know this.
  • From the problem owner: How can I give an expert weight zero?
    • Response:Zero weight does not mean zero value. It simply means that this expert's knowledge was already contributed by other experts and adding this expert would only add a bit of noise. The value of unweighted experts is seen in the robustness of our answers against loss of experts. Everyone understands this when it is properly explained.
  • From the problem owner:How can I give weight one to a single expert?
    • Response: By giving all the others weight zero, see previous response.
  • From the problem owner: I prefer to use the equal weight combination.
    • Response: So long as the calibration of the equal weight combination is acceptable, there is no scientific objection to doing this. Our job as analyst is to indicate the best combination, according to the performance criteria, and to say what other combinations are scientifically acceptable.
slide36

Uncertainty

Ambiguity

Indecision

3

“ In the first few weeks of the Montserrat crisis there was perhaps, at times, some unwarranted scientific dogmatism about what might or might not happen at the volcano, especially in terms of it turning magmatic and explosive. The confounding effects of these diverging, categorical stances were then compounded for a short while by an overall diminution in communication between scientists and the various civil authorities. The result was a dip in the confidence of the authorities in the MVO team and, with it, some loss of public credibility; this was not fully restored until later, when a consensual approach was achieved. “

Aspinall et al The Montserrat Volcano Observatory: its evolution, organization, rôle and activities.

slide38

UNCERTAINTY

AMBIGUITY

INDECISION

British Airways Pilots study

Range graphs

slide39

UNCERTAINTY

What Is?

AMBIGUITY

What means?

INDECISION

What’s best?

slide40

UNCERTAINTY

Experts’ job

AMBIGUITY

Analysts’ job

INDECISION

Stakeholder/problem owners’

job