Rank based sensitivity analysis of multiattribute value models
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

Rank-Based Sensitivity Analysis of Multiattribute Value Models PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Rank-Based Sensitivity Analysis of Multiattribute Value Models. Antti Punkka and Ahti Salo Systems Analysis Laboratory Helsinki University of Technology P.O. Box 1100, 02015 TKK, Finland http://www.sal.tkk.fi/ [email protected] Additive Multiattribute Value Model.

Download Presentation

Rank-Based Sensitivity Analysis of Multiattribute Value Models

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Rank based sensitivity analysis of multiattribute value models

Rank-Based Sensitivity Analysis of Multiattribute Value Models

Antti Punkka and Ahti Salo

Systems Analysis Laboratory

Helsinki University of Technology

P.O. Box 1100, 02015 TKK, Finland


[email protected]

Additive multiattribute value model

Additive Multiattribute Value Model

  • Provides a complete rank-ordering for the alternatives

    • Selection of the best alternative

    • Rank-ordering of e.g. universities (Liu and Cheng 2005) or graduate programs (Keeney et al. 2006)

    • Prioritization of project proposals or innovation ideas (e.g. Könnölä et al. 2007)

  • Methods for global sensitivity analysis on weights and scores

    • Focus only on the selection of the best alternative

      • Ex post: Sensitivity of the decision recommendation to parameter variation

      • Ex ante: Computation of viable decision candidates subject to incomplete information about the parameter values

        (e.g., Rios Insua and French 1991, Butler et al. 1997, Mustajoki et al. 2006)

Sensitivity analysis of rankings

Sensitivity Analysis of Rankings

  • Consider the full rank-ordering instead of the most preferred alternative

    • How ’sensitive’ is the rank-ordering

    • How to compare two rank-orderings? How to communicate differences?

  • We compute the attainable rankings for each alternative subject to global variation in weights and scores

    • How sensitive is the ranking of an alternative subject to parameter variation?

    • Is the ranking of university X sensitive to the attribute weights applied?

    • What is the best / worst attainable ranking of project proposal Y?

Incomplete information

Incomplete Information

  • Model parameter uncertainty before computation

    • Relax complete specification of parameters

      • ”Error coefficients” on the statements, e.g. weight ratios

      • E.g. Mustajoki et al. (2006)

    • Directly elicit and apply incomplete information

      • Incompletely defined weight ratios: 2 ≤ w3/ w2 ≤ 3

      • Ordinal information about weights: w1≤ w3

      • Score intervals: 0.4≤ v1(x12) ≤ 0.6

      • E.g., Kirkwood and Sarin (1985),

        Salo and Hämäläinen (1992), Liesiö et al. (2007)

  • Set of feasible weights and scores (S)

Attainable rankings

Attainable Rankings

  • Existing output concepts of ex ante sensitivity analysis do not consider the full rank-ordering of alternative set X

    • Value intervals focus on 1 alternative at a time

    • Dominance relations are essentially pairwise comparisons

    • Potential optimality focuses on the ranking 1

  • Alternative xk can attain ranking r, if exists feasible parameters such that the number of alternatives with higher value is r-1

Attainable rankings example

ranking 1 is attainable for x3

ranking 4 is attainable for x1


ranking 1 is attainable for x2

ranking 3 is attainable for x3


Attainable rankings

Attainable Rankings: Example

  • 2 attributes, 4 alternatives with fixed scores, w1 [0.4, 0.7]










Computation of attainable rankings

Computation of Attainable Rankings

  • Application of incomplete information  set of feasible weights and scores (S)

  • If S is convex, all rankings between the best and the worst attainable rankings are attainable

    • Best ranking of xk:

    • Worst ranking of xk:

  • MILP model to obtain the best / worst ranking of each xk

    • V(x) expressed in non-normalized form (linear in w and v)

    • # of binary variables = |X| - 1

Example shangai rank ordering of universities

Example: Shangai Rank-Ordering of Universities

  • Shanghai Jiao Tong University ranks the world universities annually

  • Example data from 2007

    • http://ed.sjtu.edu.cn/ranking2007.htm

    • 508 universities

  • Additive model for rank-ordering of the universities



Table adopted from http://ed.sjtu.edu.cn/ranking2007.htm

Rank based sensitivity analysis of multiattribute value models


Sensitivity analysis

Sensitivity Analysis

  • How sensitive are the rankings to weight variation?

    • What if different weights were applied?

    • Relax point estimate weighting

      1. Relative intervals around the point estimates

    • E.g. =20 %, wi*=0.20:

      2. Incomplete ordinal information

    • Attributes with wi*=0.20 cannot be less important than those with wi*=0.10

    • All weights lower-bounded by 0.02

Results rank sensitivity of top universities

Unsensitive rankings

”Different weighting would

likely yield a better ranking”

Results: Rank-Sensitivity of Top Universities

exact weights

20 % interval

30 % interval


incompl. ordinal

no information






  • A model to compute attainable rankings

    • Sufficiently efficient even with hundreds of alternatives and several attributes

  • Attainable rankings communicate sensitivity of rank-orderings

    • Conceptually easy to understand

    • Holistic view of global sensitivity at a glance independently of the # of attributes

  • Applicable output in Preference Programming framework

    • Additional information leads to fewer attainable rankings

  • Connections to project prioritization

    • Initial screening of project proposals for e.g. portfolio-level analysis

    • Supports identification of ’clear decisions’ (cf. Liesiö et al. 2007)

      • ”Select the ones ’surely’ in top 50”

      • ”Discard the ones ’surely’ outside top 50”



  • Butler, J., Jia, J., Dyer, J. (1997). Simulation Techniques for the Sensitivity Analysis of Multi-Criteria Decision Models. EJOR 103, 531-546.

  • Keeney, R.L., See, K.E., von Winterfeldt, D. (2006). Evaluating Academic Programs: With Applications to U.S. Graduate Decision Science Programs. Oper. Res. 54, 813-828.

  • Kirkwood, G., Sarin R. (1985). Ranking with Partial Information: A Method and an Application. Oper. Res. 33, 38-48

  • Könnölä, T., Brummer, V., Salo A. (2007). Diversity in Foresight: Insights from the Fostering of Innovation Ideas. Technologial Forecasting & Social Change 74, 608-626.

  • Liesiö, J., Mild, P., Salo, A., (2007). Preference Programming for Robust Portfolio Modeling and Project Selection. EJOR 181, 1488-1505.

  • Liu, N.C., Cheng, Y. (2005). The Academic Ranking of World Universities. Higher Education in Europe 30, 127-136

  • Mustajoki, J., Hämäläinen, R.P., Lindstedt, M.R.K. (2006). Using intervals for Global Sensitivity and Worst Case Analyses in Multiattribute Value Trees. EJOR 174, 278-292.

  • Rios Insua, D., French, S. (1991). A Framework for Sensitivity Analysis in Discrete Multi-Objective Decision-Making. EJOR 54, 176-190.

  • Salo, A., Hämäläinen R.P. (1992). Preference assessment by imprecise ratio statements. Oper. Res. 40, 1053-1061.

  • Login