Quality of Non-Lending Technical Assistance A QAG Assessment October 25, 2005

1. Quality of Non-Lending Technical AssistanceA QAG AssessmentOctober 25, 2005 Quality Assurance Group

2. Table Of Contents • Executive Summary 3 • Introduction 13 • Findings 21 • Conclusions 34 • Recommendations 35

3. I. EXECUTIVE SUMMARY

4. Summary of Findings • Assessment results show that NLTA has been a highly effective knowledge transfer instrument • But…improper coding of tasks under NLTA is high and calls for remedial actions

5. WHY ASSESS NLTA? • Management concern over rapidly growing share of NLTA in AAA program, almost doubling in dollar terms (from $21m to $42m) and in number (from 175 tasks to 303 tasks) between FY01 and FY04 • NLTA quality never systematically reviewed by QAG

6. WHAT IS NLTA? • The definition of free-standing NLTA applicable to the FY04 review period was “human and technological inputs to the development of advisory services and instructional activities”. NLTA often took the following form: • Workshops and conferences:Vehicles to share international “best practice”, exchange knowledge, or build consensus • Technical advice:Practical assistance to support implementation of policies, programs or projects • Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects • The distinction between ESW and TA was whether or not the activity aims to produce “original diagnostic/analytic content for the purpose of influencing the client’s policies”

7. THE SAMPLE • The FY04 NLTA universe includes 303 tasks for a total cost of $41 million 1/ • The NLTA sample includes 752/ randomly selected tasks for a total cost of $19 million, representing 25% of all tasks by number and 46% by cost3/ • Tasks costing less than US$ 20,000 were excluded from the sample • Sample yields robust results at the Bank-wide level only 1/ At the time of sampling. This amount has since increased to $42.4 million 2/ The original sample included 85 NLTA tasks delivered to the client during FY04, of which 10 were Reimbursable TA tasks. It was subsequently decided that Reimbursable TA should be subjected to a separate QAG review. The sample size was reduced accordingly 3/ The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%)

8. MAIN FINDINGS • 48% of tasks sampled (364/ out of 75), accounting for 53% of the sample cost ($10.1 million), were improperly coded as NLTA. Improper coding is primarily caused by a weak overall governance framework (unclear product definition, low SAP user friendliness, insufficient guidance/support to TTLs, unclear accountability) • Only 52% of tasks sampled were assessed and rated (39 tasks5/ out of 75 with a $ amount of 8.9 million or 47% of the total sample cost) as they were appropriately coded as NLTA6/ 4/ Including 5 PPIAF funded tasks or 7% of the number of tasks sampled, accounting for 6% of the sample cost and representing 30% of the number of PPIAF tasks sampled and 26% of their cost 5/ Including 10 PPIAF funded tasks or 13% of the number of tasks sampled, accounting for 17% of the sample cost and representing 70% of the number of PPIAF tasks sampled and 74% of their cost 6/ The balance of 36 tasks could not be assessed and rated, using the NLTA guidance questionnaire, as they were not NLTA

9. MAIN FINDINGS…Continued • At 99% overall Moderately Satisfactory or better (SAT) for both the weighted number of tasks and their weighted value, the quality of FY04 NLTA is strong. This compares well with the 94% overall Satisfactory or better rating for the FY04 NLTA tasks reviewed under the country AAA assessment • A high number (29%) of tasks were rated Highly Satisfactory (HS) overall • Strategic Relevance and Timeliness, Internal Quality and Dialogue and Dissemination were found particularly impressive by the panels as these dimensions were rated 98% SAT overall with only 4%, 5% and 8% of tasks rated Moderately Satisfactory, respectively

10. MAIN FINDINGS…Continued • Close client involvement and solid consultant and Bank inputs resulted in both high quality content generation and significant knowledge transfer as indicated by the 99% and 96% SAT overall rating for Internal Quality and Likely Impact, respectively • Managerial attention over the implementation of PPIAF funded tasks has often been limited, resulting in missed opportunities for better dialogue and dissemination and greater impact • Although resources were found to have been very effectively used, panels noted a number of instances where task costs had not been fully recorded thus understating the cost of about a quarter of tasks assessed

11. KEY RECOMMENDATIONS OPCS should: • Simplify the presentation of the ESW and TA Decision Tree (eliminate overlapping definitions, clearly define the terms “client”, “training”, “internal order” and clarify the trigger for task creation) • Prepare and launch an NLTA portal, similar to the ESW portal currently under development, to provide better guidance and support to TTLs • Given SAP’s low user friendliness and the extent of improper coding, allow all NLTA related entries (including the original task coding) into SAP to be made through the NLTA portal • Explore, jointly with SFR and TFO, ways to ensure full accounting of tasks’ costs (including TFs and other external contributions)

12. KEY RECOMMENDATIONS… continued • The Regions and Sector Anchors should monitor the use of the TA product code by their staff to reduce the incidence of improper coding

13. II. INTRODUCTION

14. NLTA DEFINITION 1/ The definition of free-standing NLTA applicable to the FY04 review period was “human and technological inputs to the development of advisory services and instructional activities”. NLTA often took the following form: • Workshops and conferences: Vehicles to share international “best practice”, exchange knowledge, or build consensus • Technical advice: Practical assistance to support implementation of policies, programs or projects • Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects. 1/ The key distinction between ESW and NLTA was that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge

15. THE NEW NLTA DEFINITION • The new (July 04) NLTA definition2/ includes activities3/ that are: • Aimed at enabling an external client to implement reforms or strengthen its institutions; • Free standing; and • Linked to a Bank unit accountable for the services provided (exclude client-executed grant-funded activities) 2/The key distinction between ESW and NLTA remains that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge 3/ With the following SAP output types: (a) institutional development plan; (b) “how-to” guidance;(c) model/survey; (d) client document review; and (e) knowledge-sharing forum

16. ASSESSMENT OBJECTIVES • Promote accountability for quality by providing indicators on quality of NLTA work on a Bank-wide basis: • Encourage systemic change through: - Improved understanding of key determinants of NLTA quality - Disseminating assessment findings to appropriate Bank units

17. METHODOLOGY • Methodology and Guidance Questionnaire were developed by reviewing the NLTA governance framework as well as from lessons learned from a recent assessment of the quality of OESW • Tasks costing between US$ 20,000-$50,000 were reviewed by a “triage panel” and were rated using a simplified version of the Guidance Questionnaire4/. All other tasks were rated by a customized panel • The overall quality of each task was assessed on five quality dimensions (Strategic Relevance and Timeliness, Internal Quality, Dialogue and Dissemination, Likely Impact and Bank Inputs and Processes) • 110 panelists conducted the assessment • 22 Bank staff participated in the assessments as observers 4/ The simplified questionnaire provides and overall rating and a summary rating for each quality dimension but does not include ratings for individual questions

18. METHODOLOGY…Continued • Panelists used QAG’s new six point scale: - Highly Satisfactory (1): Best practice in several respects and no significant deficiencies - Satisfactory (2): Satisfactory or better on all aspects - Moderately Satisfactory (3): Satisfactory on all key aspects but significant missed opportunities - Moderately Unsatisfactory (4): Significant deficiencies in a few key aspects - Unsatisfactory (5): Significant deficiencies in several key aspects - Highly Unsatisfactory (6): A broad pattern of deficiencies

19. SAMPLE • The formal trigger for entry into the universe was the standard SAP milestone “Delivered to the Client” • The FY04 task universe included 303 tasks for a total cost of $41 million 5/ • All tasks with a cost of less than $20,000 were excluded from the universe • The sample was stratified by cost ($1m+, $400-$999k, $200-$399k, 50-199k and $20,-$50k) with over sampling of large tasks 5/At the time of sampling. This amount has since increased to $42.4 million

20. SAMPLE ROBUSTNESS • The sample includes 756/ randomly selected tasks with a total cost of $19 million, representing 25% of the universe and 46% of its cost 7/ • Statistical Robustness: The sample is only representative at the Bank-wide level (95% confidence level with a 9% sampling margin of error) 8/ • Results in this report are presented on a weighted basis to adjust for over/under representation in the sample because of stratification and sample design 6/ The original sample included 85 tasks, of which 10 were Reimbursable TA (RTA) tasks. It was subsequently decided that RTA should be subjected to a separate QAG review. The sample size was reduced accordingly 7/ The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%) 8/ Results are less robust for Regional and Network cohorts given their small size

21. III. FINDINGS

22. CODING • 48% of tasks sampled were miscoded (369/ tasks out of 75 sampled) representing 53% of the $ amount ($10.2 million out of $19.1 million) • Miscoding was found across all Regions and Networks • There is no significant correlation between tasks’ cost and miscoding. The average cost of miscoded tasks is $283,000 while the average cost of all tasks is $255,000 9/ Including 5 PPIAF funded tasks or 7% of the number of tasks sampled, accounting for 6% of the sample cost and representing 30% of the number of PPIAF tasks sampled and 26% of their cost

23. CODING …Continued • SAP not sufficiently interactive and user friendly as it does not provide adequate guidance to users to prevent task miscoding • Preparation of the ESW and TA decision tree a good initiative but: • Several TTLs found it overly complex, confusing, needing simplification • Some of its definitions are overlapping and could lead to miscoding • Some basic definitions are missing (who is a “client”, what is a legitimate trigger for task creation, what is training, what is an internal order etc…) • A number of Panelists as well as TTLs wondered why training was not considered part of TA for coding purposes

24. CODING …Continued • Of the 36 Miscoded tasks: • 8 were Partnership activities (PT) 10/ • 7 should not have been coded as NLTA since they were conducted for the Bank’s benefit - 5 should have been coded as ESW given their original analytical content - 4 should have been coded as project preparation/appraisal and 4 were not yet delivered to the client as work was under way - 3 were not separate tasks (created simply to separately account for a funding source) - 2 were a combination of several activities (ESW, preparation, IO etc…) - 1 was External Training, 1 was project supervision and 1 should have been cancelled as it never started implementation 10/ Clear definition and processing guidance for this product line have not yet been issued

25. RATINGS • 52% of tasks sampled were assessed and rated (39 tasks11/ out of 75) as they were appropriately coded as NLTA (with a $ amount of 8.9 million or 47% of the total sample $ amount)12/ • Overall quality of FY04 NLTA is strong (99% SAT13/ by number of task and by value) and above the 91% satisfactory or better rating for NLTA tasks assessed under the country AAA assessment 14/ • The most frequently stated development objective of NLTA is Institutional Development/Capacity Building 11/ Including 10 PPIAF funded tasks or 13% of the number of tasks sampled, accounting for 17% of the sample cost and representing 70% of the number of PPIAF tasks sampled and 74% of their cost 12/ The balance of 36 tasks could not be assessed and rated as they were not NLTA 13/ SAT = Moderately Satisfactory or Better 14/ The country AAA assessment reviewed a total of 64 NLTA tasks (of which 1 was delivered in FY01, 9 in FY01, 20 in FY02, 23 in FY03, 10 in FY04 and 1 in FY05). The FY04 NLTA cohort was rated 94% satisfactory or better

26. RATINGS …Continued • Distribution of Ratings:

27. OPERATIONS RATED HS

28. OPERATIONS RATED HS • 11 tasks or 29% of both the weighted number of tasks assessed and their total $ amount ($2.3 million) were rated Highly Satisfactory (HS) • The percentage of HS NLTA tasks (29%) is above the OESW and the FY02 ESW HS levels of 18% and 20%, respectively and comparable to the 29% SAT rating for NLTA tasks assessed under the Quality of Country AAA assessment • The 11 HS tasks were distributed among seven Networks (PREM, HDN, PSD, INF, ESSD, OPCS and FIN) and 4 Regions (ECA, SAR, LCR and AFR)

29. OPERATIONS RATED HS …Continued • Outstanding aspects of HS tasks include (by order of importance): - Strategic relevance, particularly the consistency of the task objectives with the CAS - Internal quality, resulting from either high quality direct task team contributions or the preparation of thorough consultant TORs, a well managed consultant selection process leading to the hiring of highly qualified consultants and high quality supervision of their work by the Bank - Strong client involvement and participation and high quality interaction with the client

30. Percent of Tasks in Each Rating Category Quality Dimension HS S MS SAT <----------------Percent---------------> Overall Assessment 29 56 13 99 Strategic Relevance & Timeliness 38 57 4 100 Internal Quality 29 65 4 99 Dialogue and Dissemination 28 62 7 98 Likely Impact 14 77 6 96 Bank Inputs and Processes 16 54 22 91 RESULTS BY QUALITY DIMENSION

31. STRENGHTS • Assessed tasks were implemented within a well defined strategic framework • Task objectives were clearly stated • For workshops/conferences the consultative process and high degree of client involvement were usually very instrumental in defining content, structure, venue and participants. High degree of client involvement noted for other tasks as well • Extensive and relevant knowledge from both inside as well as outside the Bank was drawn upon to generate high quality content

32. STRENGHTS…Continued • Both the quality of partnership arrangements with other donors and dissemination arrangements were very appropriate • For a number of tasks, client impact was evident very shortly following delivery (shift in policy positions, setting up of a regulatory body, introduction of competitive bidding process, modeling of pension options etc…) • Solid task team contributions often with help from highly qualified consultants, well supervised by the bank • Many panels noted that task implementation had resulted in significant knowledge transfer

33. WEAKNESSES • Panels noted that for PPIAF funded tasks, beyond the selection of qualified TTLs, managerial attention had often been limited during task implementation, resulting in missed opportunities for better dialogue and dissemination and greater impact • Costs had not been properly recorded for about a quarter of tasks assessed: • Tasks often cross subsidized from other budgets/task codes • TF contribution sometimes not accounted for • Other contributions (from other donors, NGOs etc…) not accounted for • With about 54% of tasks delivered to the client in June, the Bank’s internal processes, more than client needs, seem to drive the timing of delivery

34. IV. Conclusions • Results show that NLTA is a valuable knowledge transfer tool that has been used very effectively by the Bank to engage clients in a timely manner and to deal with strategically important issues with appropriate expertise • But…improper recording of tasks under NLTA is high and calls for remedial actions

35. V. RECOMMENDATIONS

36. OPCS should: - Simplify the presentation of the ESW and TA DecisionTree (avoid overlapping definitions, clearly define the terms “client”, “training”, “internal order” and the trigger for task creation) • In order to provide better guidance and support to TTLs, finalize and launch an NLTA portal similar to the ESW portal currently under development - Given SAP’s low user friendliness, allow all NLTA entries (coding, milestones, AIS etc…) into SAP to be made through the NLTA portal • Explore, jointly with SFR and TFO, ways to ensure full accounting of all task costs (including TFs and other external contributions) • The Regions and Sector Anchors should monitor the use of the TA product code by their staff to reduce the incidence of mis-coding