Expert Judgment. EMSE 280 Techniques of Risk Analysis and Management. Expert Judgment. Why Expert Judgement? Risk Analysis deals with events with low intrinsic rates of occurrence not much data available.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
EMSE 280Techniques of Risk Analysis and Management
participating in the study
E[q]=(qL+4q*+qU)/6 Var[q}= [(qUqL)/6]2
Test H0 Expert e Answered Random
Ha Expert e Did Not Answered Randomly
A circular triad is a set of preferences
ai, >> aj , aj >> ak , ak >> ai
Define
c # circular triads in a comparison of n objects and
Nr(i) the number of times that expert r prefers ai to another
object expert data Nr(1), …, Nr(n), r = 1, …, e.
c(r) the number of circular triads in expert r’s preferences
David(1963)
Combining Expert Judgment: Paired Comparison
When n>7, this statistic has (approximately) a chi
squared distribution with df =
Test H0 Experts Agreement is Due to Chance
Ha Experts Agreement is not Due to Chance
Define
N(i,j) denote the number of times ai >> aj.
coefficient of agreement
attains a max of 1 for complete agreement
for small values of n and e under H0
which under H0 has (approx.) a chi squared distribution with .
we want to reject at the 5% level and fail if Pr{2u’}>.05
Define
R (i,r) denote the rank of ai obtained expert r’s responses
coefficient of concordance
Again attains max at 1 for complete agreement
Which is (approx) Chi Squared with df=n1
Again we should reject a the 5% level of significance
vi,r ~N(i, i2) with i= vi and i2 = 2
m1
m2
m3
Probability that 3 beats 2 or 3 is preferred to 2
Think of this as tournament play
vi,r ~N(i, i2) with i= vi and i2 = 2
vi,rvj,r ~N(ij, 22) ~N(i,j, 22) (experts assumed indep)
ai is preferred to aj by expert r with probability
if pi,j is the % of experts that preferred ai to aj then
Then we can establish a set of equations by choosing a scaling constant so that
as this is an over specified system for we solve for i such that
we get and
Mosteler (1951) provides a goodness of fit test based on an approx ChiSquared Value
Thus each paired comparison is the result of a Bernoulli rv for a single expert , a binomial rv for he set of experts
vi are determined up to a constant so we can assume
Define
then vi can be found as the solution to
Iterative solution Ford (1956)
Ford (1957) notes that the estimate obtained is the MLE and that the solution is unique and convergence under the conditions that it is not possible to separate the n objects into two sets where all experts deem that no object in the first set is more preferable to any object in the second set.
Bradley (1957) developed a goodness of fit test based on
(asymptotically) distributed as a chisquare distribution with
df = (n1)(n2)/2
where the parameters μi and σi are selected by the DM to
reflect his\her opinion about the experts’ biases and accuracy
Note: i. the multiplicative model follows the same line of
reasoning but with the lognormal distribution
ii. the DM acts as the e+1st expert, (perhaps uncomfortable)
1
Expert 1
.5
0
ql
q5
q50
q95
qu
1
Expert 2
.5
0
ql
qu
q5
q50
q95
For a weighted combination of expert CDFs take the weighted combination at all break points (i.e. qi values for each expert) and then linearly interpolate
Expert 1
Expert 2
Expert 3
Expert Distribution Break Points
Realization



x




x

x

Var. 1


x





x

x

Var. 2


x


x



x


Var. 3



x




x

x

Var. 4


x



x



x

Var. 5
Expert 1 – Calibrated but not informative
Expert 2 – Informative but not calibrated
Expert 3 – Informative and calibrated
F(x) = [xl]/[hl] l < x < h
F(x) = [ln(x)ln(l)]/[ln(h)ln(l)] l < x < h
Expert Distribution
1
0.5
Uniform Background Measure





max
min
l=min{q5(1),…, q5(m),r} and h =max{q95(1),…, q95(m),r}
si= [# seed variable in interval i]/N is an empirical estimate of
(p1, p2, p3, p4) = (.05, .45, .45, .05)
H0 si = pi for all i vs Ha si pi for some i
Note that this value is always nonnegative and onlytakes the value 0 when si=pifor all i.
to be recalculated
PERFORMANCE BASED WEIGHTS
USER DEFINED WEIGHTS
EQUAL WEIGHTS