Public Procurement Evaluation by Evidence-based Multiple Criteria Decision Analysis — From conventional scoring to syste

1. Public Procurement Evaluation byEvidence-based Multiple Criteria Decision Analysis— From conventional scoring to systematic profiling Professor Jian-Bo Yang Director of Decision and Cognitive Sciences Research Centre Manchester Business School (MBS) The University of Manchester Tel: 0161 306 3427 (Ext: 63427), 07715 175 723 (O2) Email: jian-bo.yang@mbs.ac.uk Web: www.mbs.ac.uk/dsrc, www.personal.mbs.ac.uk/jbyang Public Procurement Evaluation and MCDA by J B Yang of MBS

2. Outline of This Presentation Public procurement evaluation and Multiple Criteria Decision Analysis (MCDA) Typical MCDA models Decision matrix – Scoring Pairwise comparison decision matrix - Rating Belief decision matrix – Profiling or grading Evaluation aggregation based on scores Linear aggregation or weighted sum Reference point approach using nonlinear distance measures Evaluation aggregation based on beliefs (IDS) Evidence collection, mapping and grading Evidential reasoning for generating bidder profiles Expected utility as score for ranking Sensitivity analysis for testing the robustness of ranking Communication based on both bidder profiles and scores Public Procurement Evaluation and MCDA by J B Yang of MBS

3. Public Procurement Evaluation and MCDA Procurement evaluation criteria (weight) Contractor’s organisation (0.1) Financial considerations (0.3) Management resources (0.2) Past experience (0.2) Past performance (0.2) Failure of a contract (0.25) Overruns: time (0.25) Overruns: cost (0.25) Actual quality achieved (0.25) …… Public Procurement Evaluation and MCDA by J B Yang of MBS

4. Multiple Criteria Decision AnalysisUnder uncertainty –Summary of main features • A hierarchy of performance or risk criteria • Quantitative and qualitative criteria • Precise data and uncertain numbers • Subjective judgements with uncertainty • Possible absence of data • Non-commensurability among criteria • Conflict among criteria • Ranking may not be precise Public Procurement Evaluation and MCDA by J B Yang of MBS

5. Transparency and fairness via knowledge sharing • Objectivity via data collection and management • Systematic analysis via information aggregation • Panoramic view of bidder profile • Sensitivity analysis for uncertainty clarification • Consistency in evaluation • Simulation for improvement and feedback • Communication with evidence (original / aggregated) Modelling for Procurement Evaluation Public Procurement Evaluation and MCDA by J B Yang of MBS

6. MCDA Models for Procurement Evaluation Scoring based model – decision matrix MCDA problem with numbers:Decision maker is faced with assessing and ranking several alternatives with all attributes being considered simultaneously, with no attribute being absolutely more important than others. The problem can be represented as follows Decision Matrix (Table) How should the assessment and ranking be made ? Public Procurement Evaluation and MCDA by J B Yang of MBS

7. Scoring-based Decision Matrix– Job evaluation Decision Matrix for Job Evaluation Public Procurement Evaluation and MCDA by J B Yang of MBS

8. Pairwise Comparison MatrixCompare each pair of job offers on a criterion Pairwise Comparison Matrixfor Job Evaluation Job 1 is judged (rated) twice as good as Job 2 in terms of “Quality of life” (Interval comparison ?) Public Procurement Evaluation and MCDA by J B Yang of MBS

9. Evidence-based Belief Decision Matrix – Take into accountjudgmental information MCDA problem with both numbers and judgements: Belief Decision Matrix Belief distribution: Sij={(H1, βij1), (H2, βij2), …… , (HN, βijN)} Public Procurement Evaluation and MCDA by J B Yang of MBS

10. Belief Decision MatrixAssessment based on evidence collected Belief is generated from the assessment of evidence Public Procurement Evaluation and MCDA by J B Yang of MBS

11. Belief Decision MatrixAssessment based on evidence collected and mapped Assessing the Location of House 1 in Altrinchamusing the collectedevidence against the agreed assessment standards (mapping) Public Procurement Evaluation and MCDA by J B Yang of MBS

12. Procurement Evaluation Aggregation–Weighted sum or Multiple Attribute Value Function General form of an additive (linear) value function is given by: • Conditions for use of Additive MAVF: • Satisfaction of preferential independence among any groups of attributes. This is only a necessary condition. • Satisfaction of the corresponding trade-off, or Thomsen condition. • Intervalscale property for constructing marginal value function. • Weights of attributes need to be assessed as scaling constants (trade-offs), or swing weights, not necessarily relative importance. • Linear & complete compensation among criteria without any limit. Public Procurement Evaluation and MCDA by J B Yang of MBS

13. MCDA – Value Measurement Theory– Preferential independence violation example Chinese Restaurant Menu: Combination of soup and main dish Attribute 1: Choose soup Attribute 2: Choose main dish Are you preferentially independent when choosing soup and main dish? If you are preferentially independent in choosing soup and main dish, you would ask for a main dish without considering what soup you have taken. However, is this the case for you? Would you really choose both Mixed veg & bean curd as soup and Bean curdas main dish? Public Procurement Evaluation and MCDA by J B Yang of MBS

14. Limitation or Bias of Additive MAVF Efficient frontier: A, B,D, E, F, G Efficient convex hull: A, E, G Additive MAVF cannot find B or F as the most preferred solution ωsvs+ωpvp=v Public Procurement Evaluation and MCDA by J B Yang of MBS

15. Distance-based AggregationIdeal point models (minimax distance) Ideal point models: Set an ideal reference point and find an alternative closest to the ideal point in certain distance measure. Set criterion weights Ideal point Reference point Public Procurement Evaluation and MCDA by J B Yang of MBS

16. Evidential Reasoning MCDAAssessment distribution by a belief structure ER Example 1:A qualitative assessment that the quality yq of a bidder A is assessed to be “Good” or “Excellent” by an equal number of assessors, with no assessment below “Average”, can be described by the following distribution S(yq(A))={(Bad, 0), (Average, 0), (Good, 0.5), (Excellent, 0.5)} which is termed as a belief distribution of assessment, with “Bad”, “Average”, “Good” and “Excellent” defined as “assessment grade” and 0 (0%) and 0.5 (50%) as “belief degree” (frequency to which “Good” or“Excellent” is ticked by the assessors). The above distribution shows the quality profile of the bidder. Public Procurement Evaluation and MCDA by J B Yang of MBS

17. Evidential Reasoning MCDAAssessment Using ER – What’s different Traditionally, only scores are used ER uses both scores and belief degrees 6.1 Give examples of STRATEGIC Partnering, Alliances and Collaborative Working Bidders Bidder 1 Score 76% ({Best, 28%}, {Good, 51%}, {Average,17%}, {Poor, 4%}, {Worst, 0%} ({Best, 46%}, {Good, 29%}, {Average,15%}, {Poor, 3%}, {Worst, 7%} Bidder 2 Score 76% Public Procurement Evaluation and MCDA by J B Yang of MBS

18. Assessment Using a Decision Support System–Intelligent Decision System (IDS) • IDS is supported by the Evidential Reasoning (ER) approach • ER has been developed over a period of over 15 years • ER results from multi-discipline research • Decision Sciences • Artificial Intelligence • Statistical Analysis • Fuzzy Sets • ER addresses subjectivity and uncertainties • ER can handle heterogeneous information • ER guarantees to generate rational results • ER is gaining popularity in both academia and industry Public Procurement Evaluation and MCDA by J B Yang of MBS

19. Assessment Using IDS – Advantages • Structured and natural No modification needed in IDS for procurement evaluation modelling • Flexible in modellingModel can be modify, attributes changed, added and deleted easily • Improved consistency and efficiencyThrough knowledge management and using an systematic evidence mapping process Public Procurement Evaluation and MCDA by J B Yang of MBS

20. Assessment Using IDS – Advantages • No unnecessary assumptionNo need to use scores for subjective judgement No need to assume missing data • TransparentCandidates compared on any attribute at any level Weaknesses and strengths of each candidate identified • Rational, convincing and informativeExamine impact of changes in any factor on decisions easily so that the decisions are made in a more rational, convincing and informative way Public Procurement Evaluation and MCDA by J B Yang of MBS

21. Assessment Using IDS – Modelling Build an evaluation criteria hierarchy Public Procurement Evaluation and MCDA by J B Yang of MBS

22. Number of grades can be changed Assessment Using IDS – ModellingDefine qualitative attribute: Public Procurement Evaluation and MCDA by J B Yang of MBS

23. Wording of grades can be changed Preference value of grades can be changed Assessment Using IDS – Modelling Define grades to assess a qualitative attribute: Public Procurement Evaluation and MCDA by J B Yang of MBS

24. This can be used as guidelines to help improve consistency in assessment Assessment Using IDS – Modelling Define grade standard to assess a qualitative attribute: Public Procurement Evaluation and MCDA by J B Yang of MBS

25. Assessment Using IDS – Modelling Define quantitative attribute: Public Procurement Evaluation and MCDA by J B Yang of MBS

26. Drag and drop to change weight Or type weight here Assessment Using IDS – Modelling Assign weights Public Procurement Evaluation and MCDA by J B Yang of MBS

27. Assessment Using IDS – Review DocumentEvidence classified and recorded Public Procurement Evaluation and MCDA by J B Yang of MBS

28. Assessment Using IDS – Mange Knowledge Evidence examined and comments provided Public Procurement Evaluation and MCDA by J B Yang of MBS

29. Grade guidelines entered earlier More than one grades may be selected Optional Assessment Using IDS – Make assessmentEvidence mapped and belief degrees assigned to grades Public Procurement Evaluation and MCDA by J B Yang of MBS

30. Can be any attribute in the hierarchy Unknown element due to lack of data in Economic Test & Interview Assessment Using IDS – View ResultsPerformance distribution (profile) of bidder generated Public Procurement Evaluation and MCDA by J B Yang of MBS

31. Assessment Using IDS – View ResultsPerformance scores – when there is missing data Public Procurement Evaluation and MCDA by J B Yang of MBS

32. Assessment Using IDS – View ResultsCompare candidates on multiple criteria – by scores Public Procurement Evaluation and MCDA by J B Yang of MBS

33. Assessment Using IDS – View ResultsCompare candidates – by performance profile Public Procurement Evaluation and MCDA by J B Yang of MBS

34. Other Applications - Siemens UKSupplier pre-qualification assessment Public Procurement Evaluation and MCDA by J B Yang of MBS

35. Other Applications • Product design and evaluation car, motorcycle, ship, aircraft, computer, … • Safety and risk assessment • Quality management • Supply chain management • Environmental management • Financial services and investment • Customer satisfaction survey • Web based survey • Data collection only – remote or onsite audit Public Procurement Evaluation and MCDA by J B Yang of MBS

36. Summary and Conclusions • Public procurement evaluation and multiple criteria decision analysis (MCDA) • Typical MCDA procurement evaluation models • Decision matrix – Scoring • Pairwise comparison decision matrix - Rating • Belief decision matrix – Profiling or grading • Evaluation aggregation based on scores • Linear aggregation or weighted sum • Reference point approach using distance measures • Evaluation aggregation based on beliefs (IDS) • Evidence collection and mapping or grading • Evidential reasoning for generating bidder profile • Expected utility as score for ranking • Sensitivity analysis for testing the robustness of ranking • Communication based on both bidder profile and score Public Procurement Evaluation and MCDA by J B Yang of MBS