The Quality Assessment, Monitoring and Evaluation Framework

1. The Quality Assessment, Monitoring and Evaluation Framework

2. The 16th APEC Ministerial meeting:“Ministers approved a new Quality Assessment Framework, which will replace the ECOTECH Weightings Matrix, as a tool to foster good quality project proposals and an efficient allocation of APEC resources.” Initially commenced for Operational Account and ASF projects, at BMC1 2007, the BMC approved the use of the QAF for ALL projects. The History of the QAF

3. The Assessment Framework Will the project contribute to APEC’s mission? The Monitoring Framework Is the project on track to achieving its objective? The Evaluation Framework Have the objectives been achieved? The Quality Assessment, Monitoring and Evaluation Frameworks (AM&E)

4. Each fora establishes a small group to assess project proposals 17 design criteria to be assessed Each design criterion is scored from 3 to 0. -Good practice (3) -Satisfactory (2) -Marginally satisfactory (1) -Weak (0) -Not applicable (0) All criteria are of equal value, there is no weighting system How to use the Quality Assessment Framework (QAF)

5. Comments made against each criterion can be a simple yes or no. However, comments are encouraged as they help the PO improve the project. Add scores together to get a final score. Lead shepherds and Chairs oversee the process and ensure the framework is objectively filled in and are either kept informed or participate in the group. The Chair or the QAF group should send the filled out QAF to the proposal proponent and project overseers to allow them to make revisions if required. The PO submits the final project proposal, with QAF attached, to the fora for final approval before submitting to the Secretariat for submission to BMC. This will clearly indicate the strengths and weaknesses of the proposal to the Committee.

6. 1. Does the proposal respond to a priority identified by Leaders or Ministers? The proposal should be directly relevant to one specific priority set by APEC Ministers and Leaders Comments (if any) Assessment Yes 2 Examples

7. 9. Is the implementation methodology described in the proposal both clear and effective? Have the main risks to implementation been identified? Explanation: Implementation should be well-planned and logical, and based on a breakdown of functions over time. Main risks or assumptions should be identified and practical strategies prepared to manage them. Comments Assessment No, needs more detail 1

8. 13. Does the proposal have a strategy to make project benefits sustainable? Explanation: Project benefits should last well beyond project completion. For example, regular up-dates of documents and training, ensuring that manuals are in local language, etc. Comments Assessment Yes 3

9. 16.Final statement of suitability for APEC funding If implemented, is the project likely to make a sufficient contribution to APEC’s mission to justify its funding? Considers all relevant factors plus any others specific to a fora. Comments Assessment Yes, high priority 3

10. The starting point is good quality project proposals strengthened by the quality assessment framework From here we may need to monitor progress to ensure projects are on track to achieve their stated objectives

11. The Monitoring Framework is a health check for the project as it is being implemented. Benefits: Ensures projects on track, helps achieve the desired outcomes, enhances APEC credibility and supports regional development Stage 2: Monitoring Framework

12. Identify reasons why a project may not be on track to succeed Identify potential problems so that corrective actions can be taken Provide data for the outcome evaluation A Health Check can:

13. 10 to 20% of APEC projects probably do: Multi-year Have a follow-up or an extension Perform well and have lessons Not perform well and have lessons Innovative or pilot Interesting and controversial Who needs a Health Check?

14. How? - Use the Monitoring Framework Who identifies a project for the health check? - The BMC on approval, or the PD or fora should flag candidate projects Who then reviews the project? - Members of the fora (the Small Group on Evaluation), with the PO as a resource person, and the Co-sponsor involved

15. Why do we bother to evaluate? To be accountable for APEC funds Learn lessons that strengthen future projects Provide lessons about policies, strategies and processes within which projects operate Stage 3. Outcome Evaluations

16. The Evaluation Framework started in 2006 and replaced the former Evaluation report One Framework is used for all projects (finalised BMC1 2007) Outcome Evaluation

17. Procedures are similar to the QAF. The PO completes the first section ( a statement on the outcomes of the project) and sends to the fora. The fora, or a small group, then completes the questions in the framework and sends to the Chair. The Chair comments and signs off on the framework. The PD may comment, and then send the framework to the BMC Procedures

18. The QAF process can be completed on the PDB. For OA and ASF projects, at the draft stage the PO clicks on ‘submit for qaf review’. Members, or the small group, will receive an email with a link to the qaf template. For TILF projects, please complete manually and send to the PO. The PO can upload as supporting documentation. (The PDB will be updated to include TILF at a later stage). The Monitoring and Evaluation Framework can also be submitted online at the private project site. QAFs and the PDB

19. Lead Shepherds and Fora Chairs should ensure all projects being put to the BMC have been assessed using the Quality Assessment Framework (QAF) and that evaluations are completed using the new Evaluation Framework Chairs should identify projects in need of monitoring and monitor them as required. Next steps for projects

20. For any further advice on the AM&E Frameworks please contact: The Project Management Unit pmu@apec.org Contacts