1 / 28

How to Use and Abuse MCDA and Consensus Support Methods

How to Use and Abuse MCDA and Consensus Support Methods. Dave Mason Operational Research Division National Defence Headquarters Ottawa, Canada. Outline. Definitions and applications Cardinal utility methods When they are applicable, desired attributes How to abuse

stella-finch
Download Presentation

How to Use and Abuse MCDA and Consensus Support Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How to Use and Abuse MCDA and Consensus Support Methods Dave Mason Operational Research Division National Defence Headquarters Ottawa, Canada

  2. Outline • Definitions and applications • Cardinal utility methods • When they are applicable, desired attributes • How to abuse • Summary: How To Use Cardinal Utility Methods • Ordinal methods • Fundamental concepts (Arrow’s Theorem, Voter’s Paradox) • How to abuse common ordinal methods • Heading towards a solution to the Consensus Ranking Problem • Solution concept: Maximization of rank correlation • Existing statistics: Kendall’s τb, Kemeny/Snell distance • Unified theories, and a new rank correlation statistic: τX • Summary: How to Use Ordinal Methods

  3. MCDA andConsensus Support Methods • Multi-Criteria Decision Analysis Methods • Generally, any scheme that evaluates a list of alternatives across a set of assessment criteria to determine a winner (or final ordering) • Weighted sum methods are the most common form • Consensus Support Methods • Any methodology that attempts to resolve value judgements of a group of individuals into a single collective set of judgements

  4. Applicable Decision Environments • Applicable types of decision environments: • Evaluation of contract bids • Ranking and selection of personnel • Prioritization of an organization’s work program • Inevitable stream of short-fuse ‘Tiger Team’ approaches • … • Generally, involves one of following types of decisions: • 1. Single Choice - selecting the winning alternative • 2. Prioritization - put the list of alternatives in a rank ordering • (3. Resource Allocation – assign resources to each alternative) • Methods are applied after the conduct of, not in lieu of a comprehensive cost-effectiveness analyses

  5. A Single Overall Measure of Utility? • Key question the analyst must ask: “Is the development of a single overall measure of utility reasonable in this case?” • A single measure may defy definition • Is a potentially dangerous over-simplification • But may be a necessary evil to enable an answer to be reached • YES. Then a cardinal utility method may be useful • NO. Then an ordinal methods approach may be a useful alternative

  6. CARDINAL UTILITYMETHODS • Generally the broad class of weighted sum methods are most commonly employed • Assign weights to key criteria • Score the alternatives on each criteria • Arithmetically calculate the weighted sum • Numerous approaches with enhanced mathematical complexity exist • Analytical Hierarchy Process (AHP) • …

  7. Desirable Attributes: 1. Transparency of Method Qualitative and Quantitative Inputs X “Black Box” Method (Clear glass doors) ?? Final winner or ordering

  8. Desirable Attributes: 2. Reasonable Fidelity • Complex enough to adequately represent the key dimensions of overall utility, • “KISS” is fine, but don’t go too far • More Sophisticated Mathematics = A Better Solution? • Not necessarily • Oversold obscure methods with gratuitous mathematical complexity = Snake Oil? • You bet!

  9. How To Abuse WeightedSum Methods Add extra criteria that boost your alternative Reliability Versatility Shave the weights of criteria that hurt your alternative Style Fun Cost ($50K) ($30K) ($35K) ($40K) Weight: Sports Car Sedan Mini-Van SUV Total 3 4 5 10 8 2 6 10 8 7 3 10 3 2 7 2 10 4 4 6 2 6 10 8 7 86 72 75 85 Compress the scale of criteria that hurt your alternative

  10. How To Abuse WeightedSum Methods Reliability Versatility Style Fun Weight: Sports Car Sedan Mini-Van SUV Utility/K$ 3 4 5 10 8 2 6 10 8 7 3 10 3 2 7 2 10 4 4 6 74 52 60 71 /50 = /30 = /35 = /40 = 1.48 1.73 1.71 1.78

  11. The Value Of Weighted Sum Methods • Question: In a weighted sum application, can I fiddle with the weights and scores in subtle ways to make just about any alternative the winner? • Answer: Yeah, pretty much … • Question: I guess that means weighted sum methods are not very useful? • Answer: On the contrary, they are transparent and flexible … just make sure you use them properly

  12. Summary: How to UseCardinal Utility Methods • Simple weighted sum, multi-criteria utility methods are fine • MCDA methods are not a substitute for comprehensive cost-effectiveness analysis • Conduct a (pairwise) sensibility check on criteria utility tradeoffs • Don’t treat cost as another criterion, but as an overall utility divisor • Do a complete sensitivity analysis on both weights and scores to locate the decision cross-over points • Add mathematical complexity if warranted, but there is no positive value to using unnecessarily complex methods • For example, AHP calculates weights using a prescribed, vague qualitative scale in a laborious pair-wise assessment process where the DM is permitted to be inconsistent • Horses for courses: Fit the method to your problem

  13. ORDINAL METHODS • If development of a single overall utility scale is not sensible, then ordinal methods may be a viable option • If there is more than one DM (or more than one criterion being assessed) the problem is one of ‘social choice’, depending on the societal structure assumed • Nobel laureate Kenneth Arrow’s research from 1953 forms the logic foundation • “It … makes no sense to add the utility of one individual, a psychic magnitude in his mind, with the utility of another individual.” • Coke or Pepsi? … Strength of preference, or simply preference? • “A utility function whose significance lies entirely in its ordinal properties is superfluous. If we are concerned with ordinal properties it seems better to represent these directly.”

  14. Kenneth Arrow andSocial Welfare Functions • A ‘Social Welfare Function’ gives the ordering of alternative social states given the orderings by the individuals in the society • Arrow imposed several ‘apparently reasonable’ conditions: • Universality – Individuals can express any possible ordering • ‘Welfare’ Not ‘Illfare’ - social ordering must respond positively (or at least non-negatively) to alterations in individual values • Independence From Irrelevant Alternatives – The societal choice amongst a set of alternatives should be unaffected if other alternatives are added or removed • Citizens’ Sovereignty – the social welfare function is not to be imposed (all outcomes are possible) • Non-Dictatorship – whenever the dictator prefers x over y, so does society • Arrow’s Theorem: You can’t have all five

  15. Arrow’s Third Condition & The Voter’s Paradox • The Voter’s Paradox: • Voter 1: a b c • Voter 2: b c a • Voter 3: c a b • In this example no winner is declarable, yet if you remove any alternative from consideration a clear winner emerges • Upshot: One cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”.

  16. How to Abuse Ordinal Methods • Borda Count (minimum sum of rank values) is commonly employed • Simple example: three rankers, two alternatives • Ranker A: 1. Coke, 2. Pepsi • Ranker B: 1. Coke, 2. Pepsi • Ranker C: 1. Pepsi, 2. Coke • Coke wins … any and every rational ranking scheme will agree • Ranker C insists on including all contending colas (RC, Kik, …) • Ranker C tactically and artificially ranks Coke low to benefit his choice: • Ranker A: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola • Ranker B: 1. Coke, 2. Pepsi, T3. Kik Cola & RC Cola • Ranker C: 1. Pepsi, T2. Kik Cola & RC Cola, 4. Coke • The Borda Count winner now … Pepsi!! • The Problem: Borda count fails any reasonable interpretation of the ‘independence from irrelevant alternatives’ condition • The Lesson: Don’t conduct arithmetic operations on rank values

  17. A Proposed Solution Concept to the Consensus Ranking Problem • “Given m rankings of n objects, what ranking best represents the consensus opinion?” • In its most general form: • Ties are permitted • Incomplete rankings are permitted • Relative importance of rankers is reflected (usually in form of numerical weights) • Solution Concept Employed: Select a measure of agreement between pairs of rankings and select that ranking(s) which maximizes overall average agreement. • Possible measures: • Rank correlation coefficient • Distance measure operating on the space of all rankings of n objects

  18. One Measure of Agreement:Kendall’s Tau-b (1948) 1 if object i ranked ahead of object j aij = -1 if i ranked behind j 0 if i ranked tied with j or if i=j • Strengths: • If ties are not allowed, it is equivalent to the number of interchanges of adjacent objects required to convert one ranking into the other. • Weaknesses: • Not well defined when ties are allowed • The equivalent distance measure fails the triangle inequality test.

  19. Another Agreement Measure:Kemeny-Snell Distance (1962) • K&S defined a set of axioms that any reasonable distance measure operating on the set of all orderings of n objects should satisfy … Axiom 1.1. d(A,B) ≥ 0, with equality holding if and only if A and B are the same ranking. Axiom 1.2. d(A,B) = d(B,A). Axiom 1.3. d(A,B) + d(B,C) ≥ d(A,C), and equality holds if and only if B is ‘between’ A and C. Axiom 2. If A' results from A by a permutation of the objects, and B' results from B by the same permutation, then d(A,B) = d(A',B'). Axiom 3. If A and B agree except for a set S of k objects, which is a segment of both, then d(A,B) may be computed as if these k objects were the only objects being ranked. Axiom 4. The minimum positive distance is 1. • They proved that there is a unique distance metric that satsifies these axioms, and that the following is an implemen-tation of that unique measure (using Kendall’s def’n for aij)

  20. Two Other Proposed Measures of Agreement • The Tau-X Rank Correlation Coefficient: 1 if object i ranked ahead of OR TIED WITH object j aij = -1 if i ranked behind j 0 if i=j • The Half-Flips Metric: • Define Elemental Transformations of any ordering: • 1. The Merge: Any two adjacently ranked groups of tied objects of size p and q can be merged with pq simultaneous half-flips. • 2. The Split: Split any group of tied objects into two subgroups of adjacently ranked objects (of size r and s) with rs simultaneous half-flips. • The Half-Flips Metric is defined as the minimum number of half-flips required to migrate from ranking A to B

  21. A Single Unified Theory • Lemma 1. The Half-Flips Measure satisifes the Kemeny-Snell axioms. • Lemma 2. The Kemeny-Snell metric formulation is equivalent to τX. • THEOREM. The Half-Flips Metric, the Kemeny-Snell Metric, and the τX rank correlation coefficient are equivalent representations of the unique measure that satisfies the Kemeny-Snell axioms.

  22. MaxτX – A Simple Example • 5 Rankers (V, W, X, Y, Z ) and 4 Objects (a, b, c, d) • Rankings (all equally weighted) • V: c b a d • W: a-c d b • X: a-d b-c • Y: d b a c • Z: c-d a-b • “Combined Input Matrix”: • 1*V + 1*W + 1*X + 1*Y + 1*Z = 0 -1 -1 1 1 0 -1 1 1 1 0 1 -1 -1 -1 0 Ranking Matrix . 0 1 1 1 1 0 -1 -3 1 3 0 1 1 3 1 0 0 ? ? ? ? 0 ? ? ? ? 0 ? ? ? ? 0 Max Value? • How do we find the solution ranking(s)? • If sign matrix represents a ranking then it is the unique solution (rare!) • Could search ranking space using brute force

  23. Our Simple Example a,d,b,c a,d,c-b a-d,b,c a-d,c-b Input Rankings (5) a,d,c,b d,a,b,c a-d,c,b d,a,c-b Solution Ranking (1) a-c-d, b 3 d,a,c,b d,a-b,c a,c-d,b d,a-c,b d,b,a,c a,c,d,b 7 a-c-d,b d,a-c-b d-b,a,c d,c,a,b b,d,a,c 2 a-c,d,b d,b,a-c d-b,a-c c-d,a,b d,c,a-b d,b,c,a 3 b,d,a-c c,a,d,b c,a-d,b d,c-b,a d-b,c,a c,d,a,b c-d,a-b Minimum Total Distance = Minimum Voter Unhappiness c,a,b-d d,c,b,a b,d,c,a c,a,b,d c,d,a-b c-d,b,a c,a-d-b d-c-b,a b,d-c,a c,a-b,d c,d,b,a 7 c,b,a,d c,b-d,a b,c,d,a c-b,d,a c,b,a-d c,b,d,a

  24. How Feasible IsThe Brute Force Approach? • Number of rankings of n objects with ties, Q(n) • Q(2) = 3 rankings • Q(3) = 13 rankings • Q(4) = 75 rankings • Q(5) = 541 rankings • … • Q(n) = no. of protons on earth for what value of n? Ans. • O.A. Gross (1962) showed: 39

  25. A Cleverer Way? • Branch-and-bound algorithm devised • When implemented in FORTRAN on Pentium PC will solve for all solutions • Up to 12 or so alternatives (any number of rankers): instantly • Up to 15 or 20: seconds to minutes • Up to 20 or 25: hours to days • Over 30: basically hopeless • Heuristic algorithm developed for larger problems • Progressively applies B&B algorithm over a sliding window of 11 objects until it stabilizes • Generates solutions correctly in virtually all cases tested

  26. DISCUSSION • Extension to Incomplete Rankings • CI Matrix doesn’t change if one adds (off diagonal) zeros – this naturally represents the ‘no information’ situation between object pair • Accommodates various preferential voting schemes where objects can be ranked against other blocks of objects but are left ‘unranked’ against each other (as opposed to ‘tied’) • Difference Between Ties and Indifference • A tie is a positive statement of agreement (‘strong’ ties), not a declaration of indifference (‘weak’ ties) • Max τX method handles both • The Advantage of Dual Theories • Extension in one theory don’t necessarily have a parallel in the other • e.g. Incomplete rankings extension in τX domain has no parallel in Half-Flips Metric domain

  27. Summary: How to UseOrdinal Methods • If development of single overall utility scale does not make sense, then you an ordinal technique may be a suitable choice • Avoid methods that involve arithmetic operations on the rank values themselves • Avoid methods that deal in strength of preference • ‘Independence from irrelevant alternatives’ is the criterion most ordinal methods run afoul of • Arrow proved that one cannot construct an ordinal solution using the argument: “If more individuals place a ahead of b, then the consensus or social choice will place a ahead of b”. • Determining consensus rankings by maximizing rank correlation using the τX statistic is a robust method • The problem-situation must be accepting of multiple solutions • It is computationally intensive to apply

More Related