World Bank Policy on the Selection and Employment of Consultants: Study of its Effectiveness

1. Operational Policy & Country Services Procurement World Bank Policy on the Selection and Employment of Consultants: Study of its Effectiveness Presentation of Key Findings and Recommendations Giovanni Casartelli

2. Rationale and Objective of the Study • Bank Clients complain they don’t get the best out of their procurement of consulting services. Consultants complain of Clients’ lack of capacity to observe quality and conduct a fair selection process. • This study evaluates the effectiveness of present guidance in bringing by efficient contract allocation by considering quality, efficiency, economy, fair competition, transparency and the development of consulting capacity in Borrowers’ countries. • Complexity and specialization have lead to an overload of rules. Compliance sacrifices the efficiency and economy of the selection process and finding of the most suitable consultant.

3. Rationale and Objective of the Study • In setting selection parameters and making procurement decisions inexperienced clients often ignore the complexity and risk profile of the assignment. Treating all contract as if they where simple and they disregard quality and impacts. • Given this range of problems, consultants, including some of the best often abstain from participating. They complain to a” race to the bottom” that makes consulting frustrating as a profession and unsustainable as a business proposition. • Study suggests that the effectiveness of selections will increase if the guidance that Clients are asked to follow is clarified and improved. This must come with improved capacity of those entrusted with diligent process execution.

4. Finding 1 The selection process takes too long and is too expensive • The selection process duration takes in average 17 months, spent by Clients preparing short lists and evaluating proposals amounting to 400 to 700 pages. • Consultants are discouraged by transaction costs and long decision time. The validity of their technical proposals and their commitment vanish as CVs and work plans need to be changed before starting the assignment.

5. Finding 2 The Guidelines are vague or confusing in giving guidance on key aspects of the selection process Example: A consulting assignment’s TOR should lead the choice of the selection parameters such as selection method and short list, but the Guidelines (para 2.3) do not give guidance on this. As a result Clients disregard the TOR when setting up the selection process which causes their RFP’s to function as imprecise discovery mechanism. If Clients are expected to do a better job in following the guidance and obtain quality, the meaning and purpose of the Guidelines’ six considerations (para 1.4) and key issues must be clarified or explained.

6. Finding 3 QCBS is used in compulsive manner in Bank projects despite the Guidelines offering six different selection methods: Clients adopt QCBS in 92% of the sample cases even when trading quality for price can amount to ill advice. The location and wording of the reference to price contributes to the uncritical application of QCBS when consideration for the assignment’s ToR, project risks and Client’s preferences should be called for in first place.

7. Finding 4 The current Guidelines provisions on short list are rigid and cause short lists to be unbalanced or include unqualified consultants The call for short lists of “six consultants with a wide geographic spread and not more than two firms from the same country but at least one from a developing country” are generally met. However, the requirement often makes Clients’ overlook participants qualifications and their own justified preferences, thus compromising the integrity of the selection.

8. Finding 5 Bank Clients do not assign qualified professionals on Evaluation Committees (EC) • Only 38% of EC members are specialists in the field of the assignment, the rest are executing agency directors and higher authority delegates • The level of decisive disagreement by individual EC members (19%) and the 131 days elapsing between proposal submission and transmission of the Technical Evaluation Report to the Bank, suggest there is generally a lot a room for improvement with regards to the EC capacity to evaluate quality correctly

9. Finding 6 Weak correlation between quality and price EC’s assign similar technical scores to proposals quite differently priced, and the study shows no significant correlation between quality and price. The mean technical score dispersion is 1.12 while the mean price dispersion is 1.79. As a result the lowest financial proposal is awarded the contract in 67% of cases, and in 37% of cases the highest technical score is also the lowest priced proposal. Conversely the highest technical score is awarded the contract in 64% of cases.

10. Finding 7 The scoring formula for QCBS is flawed The scoring formula for QCBS:S =  St + (1 – )Sfwhere Sf=pmin/p *100is flawed in the following ways: • It favors high priced proposals • It depends on the lowest price, pmin, in a way that makes it impossible for consultants to predict how the rule rewards them for quality • pminalso makes it difficult for the Client to calibrate the scoring rule to the specifics of his/her assignment. • It allows irrelevant proposals to change the selection outcome in a way that invites shill bidding These characteristics would not be so troublesome if the mean price dispersion was smaller, but in 28% of cases it is above 2.

11. Finding 8 Participation of National Consultants The study sample cases shows that Bank provisions promoting national consultancy have contributed to an increased participation by national consultants. However the quality of services provided has not increased at the same pace. The consulting industries are plagued by poor regulations, low demand and widespread public indifference. The recurrent use of QCBS combined with prevalently price-based domestic regulations has lead to disregard for quality and low remuneration levels that hurt the sustainability of domestic consultants. The Bank should consider if it is justified to maintain the present preferential provisions of the Guidelines, or promote more strongly the quality and sustainability of independent national consulting as part of their knowledge economy.

12. Finding 9 Transparency The Guidelines provisions on transparency aim at preventing discrimination of consultants, promote access and verifiability. The study results show the Guidelines to be effective in this regard. However, Clients would also benefit if the Guidelines explained how transparency improves outcome quality, credibility and efficiency of the evaluation process itself, by codifying the appointment of EC members and following protocols on how to score, compare, discuss and agree on contract award.

13. Recommendations The study proposes 8 Key recommendations that should help Clients obtain suitable quality efficiently, fairly and transparently by turning the Guidelines into a more effectiv for decisions based on professional judgment, and less a set of loose mechanical prescriptions.

14. Key Recommendation 1 Ensure higher Efficiency and Economy Following measures could be considered: • Aggregating compatible and staged assignments wherever justified by technical and administrative circumstances, or by incumbent’s observed good performance • Unifying numerous requests for EOI under one • Encourage use of available simple selection methods (CQS) and simplified technical proposals (STP) • Induce selectivity and “sincerity” by consultants when expressing interest

15. Key Recommendation 2 Strengthen consideration for quality This can be achieved by the following measures: • Rephrasing the Guidelines to dispel the current misconception that selection methods are recommended without regard for the assignment Terms of Reference and related risk factors (i.e. complexity and impact of the assignment). • Reduce complexity of RFP calibration and proposal evaluation by adopting Simplified Technical Proposal and CQS as default solution for set $ amounts along lines adopted by the ADB.

16. Key Recommendation 3 Ensuring better and balanced short list and choosing the right selection method The importance of the “qualifications for the assignment” can be strengthened by: • Relaxing the mandatory short list requirement of “six firms with a wide geographic spread” (para 2.6) and, • Reformulating para 2.8 to better emphasize the concept of homogeneous vs mixed short lists.

17. Key Recommendation 4 Composition of Evaluation Committees The Guidelines will contribute to improving quality and credibility of the selection by introducing a provision stressing the importance of appointing a capable and honest Evaluation Committee composed of professional officers capable of evaluating proposal quality.

18. Key Recommendation 5 A new Scoring Formula in QCBS The currently used QCBS formula can compromise an evaluation, particularly when the dispersion of prices is high A new scoring formula under QCBS aSt +300 – (1 – a)ln(p)100will eliminate the flaws of the currently used formula, provided that a higher weight (i.e. 90% is given to quality) While the proposed new scoring formula needs further testing, it is recommended that the current QCBS formula be used with great prudence (ie only when short lists are homogeneous and proposals comparable so that price can constitute an efficient and transparent evaluation criterion

19. Key Recommendation 6 Promote National Consultant Growth and Participation • Update the contract value threshold limited to national competition, stressing freedom of including international consultants as subs • Use CQS, QBS and FBS as default methods, and QCBS only when justified by the specific characteristics of the assignment’ TOR and a balanced and fair short list

20. Key Recommendation 7 For more Transparency Guidelines should indicate to Clients how transparent procedures can improve credibility and outcome of the evaluation process by: • Appointing EC members on sound of criteria • Have them follow protocols to manage and time the process, evaluate and score the proposals individually, discuss scoring and agree on award

21. Key Recommendation 8 Action Plan