Expert opinion elicitation
Download
1 / 21

Expert-Opinion Elicitation - PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on

Expert-Opinion Elicitation. Robert C. Patev North Atlantic Division – Regional Technical Specialist (978) 318-8394. Expert-Opinion Elicitation. Subjective Estimation Elicitation Process Background Expert-Opinion Elicitation (EOE) Process Probability Axioms of Probability

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Expert-Opinion Elicitation' - jerry-johnson


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Expert opinion elicitation

Expert-Opinion Elicitation

Robert C. Patev

North Atlantic Division – Regional Technical Specialist

(978) 318-8394


Expert-Opinion Elicitation

  • Subjective Estimation

  • Elicitation Process

    • Background

    • Expert-Opinion Elicitation (EOE) Process

  • Probability

    • Axioms of Probability

    • Medians and Percentiles

  • Training Example


Subjective estimation
Subjective Estimation

  • Uses of one or more experts to estimate a probability (qualitative or quantitative) for use in engineering risk analysis

    • Good for first estimate of probabilities

    • Quick, cost effective and efficient method

    • Problems:

      • Not a formal elicitation

      • Usually not well documented

      • Probabilities may not be repeatable or defendable

      • Probabilities may be highly subjective and biased

      • Probabilities have larger uncertainties compared to structured elicitation values


Subjective estimation1
Subjective Estimation

  • How good are we at quantifying subjective estimates?

  • Let us see…..


Subjective estimation2
Subjective Estimation

  • How good are we at quantifying subjective estimates?

    • Class Example:

      • How may ships passed through the Panama Canal last year?

        • Give best estimate


Expert-Opinion Elicitation

  • Background

    • Process developed by RAND Corporation in late 1950’s - early 1960’s

      • Delphi Method

      • Scenario Analysis

    • Effects of thermonuclear war

    • Civil Defense strategic planning

      • Examine if U.S. population could survive a nuclear attack


Expert-Opinion Elicitation

  • Background

    • Definition

    • A formal (protocol), heuristic (through discussion) process of obtaining information or answers to specific questions called issues

      • e.g., failure rates or probabilities, and failure consequences


Expert-Opinion Elicitation

  • Background

    • EOE is used for preliminary risk evaluation (screening) is not really intended to replace more complex reliability models

    • EOE has been used by industry and government agencies to develop failure probabilities when there is a lack of failure information


Expert-Opinion Elicitation

  • Drawbacks

    • Subjective process

      • Not consensus building

      • Inherently contains bias and dominance

    • Difficult to process result to determine reliability or hazard rates

      • Assumptions need to be made

  • Current Usage in USACE

    • Supplement to other models

      • Calculate reliability (not for critical components)

      • Event tree probabilities

      • Used in consultation with HQUSACE


Expert-Opinion Elicitation

  • EOE Process

    • Participants

      • Experts

      • Observers

      • Listeners

      • Technical Integrator and Facilitator

      • Peer Reviewers

        • ITR process and results


Expert-Opinion Elicitation

  • EOE Process

    • Identification and Selection of Experts

      • Strong relevant expertise

      • Familiarity and knowledge with issues

      • Willingness to act as impartial evaluators

      • Willingness to participate, prepare, and provide needed input

      • Strong communication skills, interpersonal skills, and ability to generalize


Expert-Opinion Elicitation

  • EOE Process

    • Inform experts of issues

      • “Read ahead” materials

      • Site visits

    • Train experts

    • Elicitation

      • First opinion

      • Discussion among experts

      • Second opinion


Expert-Opinion Elicitation

  • Probability

    • General expressions

      • Percent (1% probability of failure)

      • Fraction (1/100)

      • Relative frequency (1 out of 1000)

    • Axioms of Probability

      • 0 < Pf < 1

      • Sum of probabilities over all possible outcomes must equal 1.

        • This assume events are independent.


Expert-Opinion Elicitation

  • Statistics

    • Median

      • e.g., Median income, median age

      • Rank value

      • For odd n, value with rank of (n+1)/2

      • For even n, average of value with rank n/2 or (n/2) + 1

      • Used to limit extreme values

    • Average

      • Sum of Xi divided by sample size


Median vs average

Sample 1

100

100

200

300

400

Median = 200

Average = 220

Sample 2

100

100

200

300

2000

Median = 200

Average = 540

Median Vs. Average


Expert-Opinion Elicitation

  • Percentiles

    • A p-percentile value (Xp) based on a sample is the value of the parameter such that p% of the data is less than or equal to Xp

      • e.g., The median is the 50th percentile






Expert-Opinion Elicitation

  • Class Example

    • Six experts required

    • Unknown issue given to experts

      • Define assumptions of issue

      • Elicit first values

      • First results

      • Expert Discussion

      • Elicit second values

      • Show final elicitation results


ad