FM Sector Quality and Results

1. FM Sector Quality and Results

2. FM Quality and Results Objectives of the session:- To build a common understanding of • FM Quality and Results Framework • Current Issues and Initiatives • The way forward to enhance quality and results

3. FM Quality and Results The results we seek:- • Improved borrowing countries’ financial management performance (client focus/ development objective) • Acceptable assurance on the use of Bank loan proceeds (corporate focus/ fiduciary objective)

4. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework

5. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework

6. FM Quality and Results • Financial Management Practices Manual • FM Practices Manual (FMPM) provides implementation guidance on OP/BP 10.02 • FMPM is currently under review and will be covered on other sessions at the Forum

7. QUALITY and RESULTS COMMITTEE- Revision of FM Practices Manual Step 1 LAUNCH Step 2 Discussion and Feedback at FID. FORUM Step 3 Drafting Step 4 Finalize AND DISSEMINATE STATUS AS AT MARCH 24, 2008 PRE-FORUM ACTIVITIES STEPS 1 & 2 January February March April May/June Work Step & End Date • LAUNCH • Initial discussions at Q&R Committee Done • Prepare Survey Questions and Committee Done • Review • Confirm broad approach with FMSB Done • Heads up and alert to FM Group Merged • Firm up with IT objectives and survey tool Done • Arrange Regional Consultations Done • Review of survey questions by FMSB Done • Launch Survey Done DISCUSSION AND FEEDBACK • Convene Regional meetings Done R • Distill messages and feedback collected from user survey, IDA14 FM , QAG, Legal Done • Distill messages from Consultations Done • Prepare draft outline of main messages Rev. Done • Validate outline with Q&R Committee Done • Validate outline with FMSB Done • FIDUCIARY FORUM DISCUSSION AND FEEDBACK • Circulate 2-3 page synopsis of proposed revisions Done • Discuss and obtain feedback during the forum Mar. 28 • Distill feedback and messages Mar. 28 • Q&R Committee Discussion Mar. 28

8. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework

9. FM Quality and Results • FM work supported by Tools • Audit Reporting Compliance System (ARCS) • PRIMA • IRIS

10. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework

11. FM Quality and Results • Regional Quality Assurance Arrangements • Ensure that FM work is carried out in accordance with provisions of the FM Manual • Supplemented by region specific business procedures as called for by FMPM • Have been implemented across all regions.

12. FM Quality and Results • Para. 82 of FMPM:- Joint CSR/FM Network Evaluation of Quality Arrangements • Compliance :- Objective is to determine whether the Quality Assurance for FM work is sufficiently robust, implemented as designed and in accordance with the FMPM. • Quality:- whether judgments and conclusions of FM work are being properly supported by evidence • Efficiency:- work is compliant and of good quality, sufficient to be relied upon by the Controller as a key pillar in disbursement simplification reform.

13. FM Quality and Results • Joint CSR/FM Network Evaluation of Quality Arrangements • Detailed Implementation Arrangements are being defined, however here are the broad basic principles:- • The reviews will be Annual • Anticipated start date: later this calendar year • Arrangements being developed by the FMSB, OPCVP and CSRVP, through the Quality and Results Committee. • Risk-based approach envisages two-phase process

14. FM Quality and Results • Joint CSR/FM Network Evaluation of Quality Arrangements • Phase I initial high level confirmation of degree of implementation of Regional QA • Phase II, compliance and substantive testing, grounded in risk based approach and based on accepted peer review principles

15. FM Quality and Results • Joint CSR/FM Network Evaluation of Quality Arrangements • To reduce oversight fatigue, the reviews will seek to create synergies between IDA 14 review work, QAG reviews and work performed by IAD • Team members will be drawn from the Network, Regions and CSR • At end of one year pilot the roles, responsibilities, methodology will be re-evaluated

16. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework

17. FM Quality and Results • Other institutional arrangements to enhance quality assurance • Quality Assurance Group (QAG) • IDA 14 Controls review • Internal Audit Department (IAD) • Independent Evaluation Group (IEG)

18. FM Quality and Results • QSA 7 Compared with 94% in QSA6 Compared with 91% in QSA6 Compared with 94% in QSA6 Compared with 89% in QSA6

19. FM Quality and Results QEA 8 conclusions & recommendations: • The relatively high risks involved in decentralized projects and the diversity of measures taken to mitigate these risks, suggests that quality could be enhanced by wider dissemination of good practice examples and updating the FM/CDD guidelines. • The continued high level of performance of DPOs can be attributed to the much wider availability of country diagnostics (CFAAs, PERs, ROSCs, PFM reviews); better integration of FM staff into the teams; FM staff having gained more experience with these kind of operations; as well as the clarity of the new OP8.60 policy and accompanying guidelines. • While in general documentation is of reasonable quality, it remains one of the areas that need further improvement. There is still an inconsistency in the quality of preparation and comprehensiveness of documentation.

20. FM Quality and Results QEA 8 The QAG Process – Going Forward • As indicated in previous QEAs, the sustained good performance of FM quality at entry suggests that this is an area that no longer requires specialist assessment in future QEA reviews and that it is timely to consider how the FM network, and the Bank, could maximize benefits from quality review processes in the future. In this regard a number of issues may be considered, including: • Impact of the Implementation of the Joint Evaluation Mechanisms • Supervision work should be linked with the updated risk assessment framework to follow up on the operational implications of the risk mitigation measure identified during the preparation phase of the project.

21. FM Quality and Results IDA 14 Internal Control Review • Source • IDA 14 Replenishment Report • Required management to “carry out an independent [through IEG], comprehensive assessment of IDA’s internal control framework, including internal controls over IDA operations and compliance with its charter and policies.” • Responsible Groups • Management (CSR and OPCS lead and coordinate “self-assessment”) • IAD reviews management findings/conclusions • IEG evaluates both • Methodology • COSO framework adapted to address control framework over operations

22. FM Quality and Results IDA 14 Internal Control Review • Part I: Transaction-level controls • Detailed identification of processes, key controls and attributes that govern IDA’s primary instruments (CAS, DPL and IL) • Compliance testing of 29 business processes and 115 controls (6 FM controls, 10 procurement, and 13 disbursement) • Part I completed in FY07; results pointed to the need to: • Reform and rationalize policies, procedures, and controls that apply to Investment Lending (IL) • Automate work-flow arrangements in the context of IDA operations to improve efficiency and document accessibility • Review, during Part II, the quality aspects of fiduciary controls (FM and procurement)

23. FM Quality and Results IDA 14 Internal Control Review • Part I Findings (FM): • Big problem with timely accessibility of relevant documents (generally) • Variances in regional implementation of FM manual • FM control failures • Due to unavailability of documents, unable to determine whether RMFM (or delegate) reviewed consistency of FMA with PAD • Majority of FM Supervision reports prepared after November 2005 were not in accordance with the FM manual • In about 40% of projects reviewed, there was no documentation evidencing communication by the FMS to the TTL of the FM rating for the ISR • In one case, the FM rating provided by the FMS was different from the FM rating in the final ISR; no explanation was attached

24. FM Quality and Results IDA 14 Internal Control Review • Part II: Effectiveness, efficiency and entity-level controls • Focus on identifying, mapping, and assessing entity-level controls along the 5 COSO components • Address issues deferred from Part I (including AAA, Country Offices, Information Technology, Fraud and Corruption Risk) and follow-on work on exceptions identified under Part I (quality of fiduciary controls) • Link the findings and results of Parts I and II and report on overall findings and conclusions on internal controls framework over effectiveness and efficiency of IDA’s operations • Part II work completed; report in drafting stage

25. Step 1 Identification of FM Issues Step 2 Assessment of Regions Step 3 Results Analysis Step 4 Reporting Communication Work Step & Time Frame • Identification of FM Issues • Prepare fiduciary issues tracker • Prepare consolidated issues matrix • Analyze common themes • Discuss review methodology and objectives with FM Sector Board (FMSB) • Develop draft questionnaire to interview regional managers, OPCFM & FMSB • Assessment of Regions • Finalize questionnaire • Conduct interviews with RMFM and team • Review and assess quality controls established in each region • Obtain evidence to support compliance with these controls where applicable • Results Analysis • Identify variances and gaps • Evaluate compensating controls where they exist • Evaluate impact of variances and gaps • Reporting • Present summary findings to FMSB for validation • Prepare draft report and recommendations • Request management responses • Finalize report and action plan

26. FM Quality and Results IDA 14 Internal Control Review • Summary of Part II FM findings: • Quality Assurance (QA) arrangements • All Regions have QA arrangements that have been documented and disseminated to staff • Difficult to ascertain in some Regions if QA arrangements for projects under implementation are completely in compliance with the FM manual • There is a range of supplementary guidance for FM staff • Projects under preparation • QA arrangements are operating effectively • Variances among Regions appear to be reasonable and generally responsive to Regional structure & organization

27. FM Quality and Results IDA 14 Internal Control Review • Projects during implementation • Some Regions are not in full compliance with their QA arrangements (e.g., monitoring of interim financial reports, recording in ARCS of FMS review of audit reports) • Some gaps in the FM Network QA arrangements are not addressed in the Regional QA arrangements (e.g., FM conditions and legal covenants, audit exemptions) • There are Regional variances in QA arrangements – many make sense, but in some Regions these variances represent risks and possible control gaps (e.g., documentation of supervision, timing of quality interventions) • Documentation and communication • Regional variances in level of detail and content of documentation of FM work • No uniform system of filing (e.g., naming conventions)

28. FM Quality and Results IDA 14 Internal Control Review • Part II concluded that deficiencies exist in the following areas: • Documentation of review of audited financial statements by an FMS • Management oversight of Audit Report Compliance Systems (ARCS) data • Documentation of FM supervision • Timing of quality interventions during supervision • Identification and monitoring of risky projects.

29. FM Quality and Results IDA 14 Internal Control Review • Part II Action Plan agreed with the FMSB • Regions to consolidate and update QA arrangements • FM manual to be reviewed and updated – to consider IDA 14 findings • Joint CSR/OPCS Evaluation of FM Network Quality Arrangements to begin – draft methodology to be updated for IDA 14 findings • ARCS data entry to be brought up to date • FMSB to monitor ARCS regularly to ensure reliability of this institutional monitoring system • Data entry of baseline information (risk ratings and interim financial reports) in PRIMA to be completed

30. FM Quality and Results IDA 14 Internal Control Review • Joint CSR/OPCS Evaluation of FM Network Quality Arrangements to specifically monitor improvements to QA arrangements over FM supervision work • Pending establishment of the institutional Enterprise Content and Document Management system, Q&RC to review existing good practices and develop guidance for staff • Fiduciary Forum to be used to remind staff about the importance of: • Documenting FM supervision work • Monitoring interim financial reports • Retention and timely accessibility of FM documents

31. FM Quality and Results The results we seek:- • Improved borrowing countries’ financial management performance (client focus/ development objective) • Acceptable assurance on the use of Bank loan proceeds (corporate focus/ fiduciary objective)

32. FM Quality and Results Objectives of the session:- To build a common understanding of • FM Quality and Results Framework • Current Issues and Initiatives • The way forward to enhance quality and results

33. Regional Quality Arrangements & Joint Evaluation FM Practices Manual Independent Reviews (e.g. QAG, IAD, IEG, IDA 14 Controls Review) Tools (e.g. ARCS, PRIMA, IRIS) FM Quality and Results Framework