Expert opinion elicitation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Expert-Opinion Elicitation PowerPoint PPT Presentation


  • 74 Views
  • Uploaded on
  • Presentation posted in: General

Expert-Opinion Elicitation. Robert C. Patev North Atlantic Division – Regional Technical Specialist (978) 318-8394. Expert-Opinion Elicitation. Subjective Estimation Elicitation Process Background Expert-Opinion Elicitation (EOE) Process Probability Axioms of Probability

Download Presentation

Expert-Opinion Elicitation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Expert opinion elicitation

Expert-Opinion Elicitation

Robert C. Patev

North Atlantic Division – Regional Technical Specialist

(978) 318-8394


Expert opinion elicitation

Expert-Opinion Elicitation

  • Subjective Estimation

  • Elicitation Process

    • Background

    • Expert-Opinion Elicitation (EOE) Process

  • Probability

    • Axioms of Probability

    • Medians and Percentiles

  • Training Example


Subjective estimation

Subjective Estimation

  • Uses of one or more experts to estimate a probability (qualitative or quantitative) for use in engineering risk analysis

    • Good for first estimate of probabilities

    • Quick, cost effective and efficient method

    • Problems:

      • Not a formal elicitation

      • Usually not well documented

      • Probabilities may not be repeatable or defendable

      • Probabilities may be highly subjective and biased

      • Probabilities have larger uncertainties compared to structured elicitation values


Subjective estimation1

Subjective Estimation

  • How good are we at quantifying subjective estimates?

  • Let us see…..


Subjective estimation2

Subjective Estimation

  • How good are we at quantifying subjective estimates?

    • Class Example:

      • How may ships passed through the Panama Canal last year?

        • Give best estimate


Expert opinion elicitation

Expert-Opinion Elicitation

  • Background

    • Process developed by RAND Corporation in late 1950’s - early 1960’s

      • Delphi Method

      • Scenario Analysis

    • Effects of thermonuclear war

    • Civil Defense strategic planning

      • Examine if U.S. population could survive a nuclear attack


Expert opinion elicitation

Expert-Opinion Elicitation

  • Background

    • Definition

    • A formal (protocol), heuristic (through discussion) process of obtaining information or answers to specific questions called issues

      • e.g., failure rates or probabilities, and failure consequences


Expert opinion elicitation

Expert-Opinion Elicitation

  • Background

    • EOE is used for preliminary risk evaluation (screening) is not really intended to replace more complex reliability models

    • EOE has been used by industry and government agencies to develop failure probabilities when there is a lack of failure information


Expert opinion elicitation

Expert-Opinion Elicitation

  • Drawbacks

    • Subjective process

      • Not consensus building

      • Inherently contains bias and dominance

    • Difficult to process result to determine reliability or hazard rates

      • Assumptions need to be made

  • Current Usage in USACE

    • Supplement to other models

      • Calculate reliability (not for critical components)

      • Event tree probabilities

      • Used in consultation with HQUSACE


Expert opinion elicitation

Expert-Opinion Elicitation

  • EOE Process

    • Participants

      • Experts

      • Observers

      • Listeners

      • Technical Integrator and Facilitator

      • Peer Reviewers

        • ITR process and results


Expert opinion elicitation

Expert-Opinion Elicitation

  • EOE Process

    • Identification and Selection of Experts

      • Strong relevant expertise

      • Familiarity and knowledge with issues

      • Willingness to act as impartial evaluators

      • Willingness to participate, prepare, and provide needed input

      • Strong communication skills, interpersonal skills, and ability to generalize


Expert opinion elicitation

Expert-Opinion Elicitation

  • EOE Process

    • Inform experts of issues

      • “Read ahead” materials

      • Site visits

    • Train experts

    • Elicitation

      • First opinion

      • Discussion among experts

      • Second opinion


Expert opinion elicitation

Expert-Opinion Elicitation

  • Probability

    • General expressions

      • Percent (1% probability of failure)

      • Fraction (1/100)

      • Relative frequency (1 out of 1000)

    • Axioms of Probability

      • 0 < Pf < 1

      • Sum of probabilities over all possible outcomes must equal 1.

        • This assume events are independent.


Expert opinion elicitation

Expert-Opinion Elicitation

  • Statistics

    • Median

      • e.g., Median income, median age

      • Rank value

      • For odd n, value with rank of (n+1)/2

      • For even n, average of value with rank n/2 or (n/2) + 1

      • Used to limit extreme values

    • Average

      • Sum of Xi divided by sample size


Median vs average

Sample 1

100

100

200

300

400

Median = 200

Average = 220

Sample 2

100

100

200

300

2000

Median = 200

Average = 540

Median Vs. Average


Expert opinion elicitation

Expert-Opinion Elicitation

  • Percentiles

    • A p-percentile value (Xp) based on a sample is the value of the parameter such that p% of the data is less than or equal to Xp

      • e.g., The median is the 50th percentile


Chickamauga lock and dam

Chickamauga Lock and Dam


Expert opinion elicitation

Chickamauga Lock and Dam


Expert opinion elicitation

Chickamauga Lock and Dam


Expert opinion elicitation

Chickamauga Lock and Dam


Expert opinion elicitation

Expert-Opinion Elicitation

  • Class Example

    • Six experts required

    • Unknown issue given to experts

      • Define assumptions of issue

      • Elicit first values

      • First results

      • Expert Discussion

      • Elicit second values

      • Show final elicitation results


  • Login