1 / 57

Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project

Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project . Honolulu, Hawaii 18-20 July, 2011. Day 1 Session 2-2: Output/Outcome/Impact & Key Performance Indicators. Norman H. Okamura, Ph.D., Faculty Specialist Christina Higa, Associate Director

slone
Download Presentation

Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project Honolulu, Hawaii 18-20 July, 2011 Day 1 Session 2-2: Output/Outcome/Impact & Key Performance Indicators Norman H. Okamura, Ph.D., Faculty Specialist Christina Higa, Associate Director Telecommunications and Information Policy Group Social Science Research Institute University of Hawaii at Manoa

  2. Understanding Key Components of Policy, Program, and Project Monitoring and Evaluation STEPSOUTPUT, OUTCOME, IMPACT AND KEY PERFORMANCE INDICATORS

  3. Overview of Policy, Plan and Process

  4. Theoria and Praxis World Bank – Building a Results-Based Monitoring and Evaluation System

  5. Program Logic Model and Implementation Implementation with Lots of Resources and Integrity of Planned Activities M&E enables corrective actions Summative Post-Hoc Implementation Could Lead to a Not Desired State Implementation Ex-Ante Future State Actual State Desired State Future State Monitoring Formative Evaluations Today Future

  6. Program Logic Model and Implementation World Bank – Building a Results-Based Monitoring and Evaluation System

  7. Monitoring and Evaluation Basic Steps

  8. 10 Monitoring and Evaluation Steps 3 Select Key Indicators to Monitor Outcomes 5 Planning For Improvement Selecting Results Target 9 Using Findings 7 Role of Evaluations 1 Readiness Assessment 6 Monitoring for Results 4 Baseline Data on Indicators – Where are we today? 2 Agreeing on Outcomes to Monitor and Evaluate 10 Sustaining the M&E Framework 6 7 5 4 8 2 1 3 9 10 8 Reporting Findings

  9. Basic Steps – Simplified Version • Formulate outcomes and goals (Completed in Plan) • Select outcome indicators to monitor (Not Completed …) • Gather baseline information on the current condition • Set specific targets to reach and dates for reaching them • Regularly collect data to assess whether targets are being met • Analyze and report the results. World Bank

  10. Monitoring and Evaluation Role of Indicators (Crude and Finely Meshed)

  11. How do we Measure? • Measurement is done in M&E through collecting and analyzing data on indicators! • Indicator Indicator - a number or ratio (a value on a scale of measurement) derived from a series of observed facts; can reveal relative changes as a function of time or A measurable characteristic or variable, which represents a policy/program/project progress. or An objective, which is time-limited and can be measured. Targets are set for a baseline in successive measurement over a period of time.

  12. Types of Indicators in M&E Framework

  13. Process, Output, Outcome, Impacts

  14. To Do Monitoring …. • Description of the problem or situation before the intervention • Benchmarks for activities and immediate outputs • Data collection on inputs, activities, and immediate outputs • Systematic reporting on provision of inputs • Systematic reporting on production of outputs • Directly linked to discrete intervention (or series of interventions) • Designed to provide information on administrative, implementation, and management issues as opposed to broader development effectiveness issues

  15. Elements of Results Monitoring • Baseline data to describe the problem or situation before the intervention • Indicators for outcomes • Data collection on outputs and how they contribute toward achievement of outcomes • More focus on perceptions of change among stakeholders • Systematic reporting with more qualitative and quantitative information on the progress toward outcomes • Done in conjunction with strategic partners • Captures information on success or failure of partnership strategy in achieving desired outcomes Fukuda-Parr, Lopes, and Malik 2002, p. 11.

  16. 10 Steps to Results-Based M&E System • Conducting a Readiness Assessment • Agreeing on Outcomes to Monitor and Evaluate • Selecting Key Performance Indicators (KPIs) to Monitor Outcomes • Baseline Data on Indicators – Where Are We Today? • Planning for Improvement-Selecting Results Targets • Monitoring for Results • The Role of Evaluations • Reporting Findings • Using Findings • Sustaining the M&E System within the Organization World Bank – Building a Results-Based Monitoring and Evaluation System

  17. Monitoring and Evaluation Indicators and Measurement (Crude and Finely Meshed)

  18. How do we Measure? • Measurement is done in M&E through collecting and analyzing data on indicators! • Indicator Indicator - a number or ratio (a value on a scale of measurement) derived from a series of observed facts; can reveal relative changes as a function of time or A measurable characteristic or variable, which represents a policy/program/project progress. or An objective, which is time-limited and can be measured. Targets are set for a baseline in successive measurement over a period of time.

  19. Defining ICT Indicators/Measurements Plane of Theory Goal or Objective Affordable Phone Key Performance Indicator Plane of Concept Pop w Phones Price Plane of Measurement # of Phones Popu- lation $

  20. Representation of World Space Cartesian Representation of X and Y Variables (World Space) The “world” is classified and defined. Y s y _ y _ t X x x Crudely and Finely Meshed

  21. Program Logic Model and Implementation The Plan in the Plane of Theory Computers in Libraries will improve student achievement Resources Budgets Personnel Work Effort Computer in Schools No Change in Student Performance Conceptual Indicators $50,000 .5 FTE Procurement Installation 50 Computers in Library Level Examinations Plane of Measurement FTE = Full Time Equivalent

  22. SMART Indicators Specific Measurable Achievable Relevant Time-bound

  23. CREAM Indicators Clear Precise and unambiguous Relevant Appropriate to the subject at hand Economic Available at a reasonable cost Adequate Provide a sufficient basis to assess performance Monitorable Amenable to independent validation

  24. CREAM and SMART Indicators Clear Precise and unambiguous Relevant Appropriate to the subject at hand Economic Available at a reasonable cost Adequate Provide a sufficient basis to assess performance Monitorable Amenable to independent validation Specific Measurable Achievable Relevant Time-bound

  25. Indicators • The choice of core indicators in ICT4D is the key to determining the outcomes and impact of planned national policies, programs, projects on development. • In order to understand the outputs of any program, inputs must also be measured, such as ICT resources. • Outputs should be measured against these same variables as well as costs. • Data should be collected throughout the program’s implementation, and in sufficient breadth and depth such that conclusions have credibility with the consumer of the study.

  26. Indicators • Indicators need to be context-specific; • Ideally indicators assess the direct issue. Most of the time, however, an indirect indicator is more feasible and therefore more reliably monitored; • Indicators are expressions of the values of those who chose the indicator. Both the selection and acceptance of an indicator depends on values; • Indicators often work best and sometimes only in combination - a single indicator does not necessarily tell you enough, and; • Indicators are transitory or sometimes seasonal – they need to be reviewed and adjusted;

  27. Indicators • Over the course of the program, conditions change, objectives are altered or better indicators [may be] discovered. In these instances, it may be appropriate to change the indicators you are monitoring. Adapted from Wieman3; Sander4.

  28. Key Indicators in National ICT? • What are some Key Indicators in National ICT4D? UNDP, UNESCO

  29. Indicators in National ICT and Telecommunications Plans • Presence of a national ICT policy • Presence of a master plan with a timeline • National expenditure on ICT • Ratio of students to computers • Integration and availability of computer networks in schools, libraries, health care organizations • Telecommunications used more by people • Telecommunications availability to people • Percentage of households with: electricity, radio, television, computer, Internet access • Percentage of population that: is covered by mobile access, uses a computer, has Internet access UNDP, UNESCO

  30. Monitoring and Evaluation Wow! That’s a lot of things we can look at!

  31. Monitoring and Evaluation Data and Data Collection

  32. Data • What are the sources of data? • What are the data collection methods? • Who will collect the data? • How often will the data be collected? • What is the cost and difficulty to collect the data? • Who will analyze the data? • Who will report the data? • Who will use the data? World Bank – Building a Results-Based Monitoring and Evaluation System

  33. National ICT Data • What mechanism and authority exists within a country to routinely collect data regarding the progress of National ICT planning? • Do the regulatory agencies within the country have the authority to collect data from telecommunications carriers? • Could the National ICT planners requests that the regulatory agency collect information on core indicators? • Does the Pacific island country have the resources to conduct annual meetings to collect data and other information regarding the implementation of the National ICT Plan?

  34. National ICT Data • Formulate outcomes and goals • Select outcome indicators to monitor • Gather baseline information on the current condition • Set specific targets to reach and dates for reaching them • Regularly collect data to assess whether the targets are being met • Analyze and report the results.

  35. Monitoring and Evaluation Data Issues

  36. Monitoring and Evaluation Application to National ICT Plans

  37. Transforming Goals/Objectives into Outcomes • What does a strong outcome statement look like? • Expresses what change will look like after a period of time • Describes the end-state not a process or ver • Is time-bound • Contains one idea and not many • Is not expressed as a indicator or include a “target” • Should allow for measurement • Should not overlap with other outcomes World Bank

  38. Outcome: weak * strong examples Weak: • To enable Internet access for the schools • To obtain computers for students • To …. (Process) Strong: • Each Village school has 5 Mbps of Internet Access with an QoS of 95% throughput to Australia • Each student shall have a laptop or netbook in the school • Result World Bank

  39. For Each Goal or Objective • Develop Outcomes • Performance SMART and CREAM Indicators for each • Establish the Targets • Develop Evaluation Questions • Assess the Assumptions and Risks World Bank

  40. Outcomes – Contentious Point • There are other outcomes that are also important not just intended outcome • e.g. Internet access provides access to pornography • We don’t always know what the outcome, change and indicators will look like at the start • e.g. Telecommunications in the States – Called great experiments for U.S. • Development is dynamic and interventions must be flexible • What is important can not always be measured World Bank

  41. Example of a Strong Outcome Statement for National ICT Plans • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

  42. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

  43. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

  44. Key Steps in Evaluation • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

  45. Build an Example of a Strong Outcome Statement National ICT Plan

  46. Measuring Outcomes:Monitoring or Evaluation? • If outcomes are important and worthwhile, the preference should be for monitoring over evaluation • Only evaluate when the cost and resource required for systematic monitoring is too high • What is evaluated should from the intended outcomes and the monitoring and evaluation f/work • A strong Framework will also identify the key research questions for evaluation World Bank

  47. Be Mindful • Organizational (management and governance) commitment is essential • M&E is a tool, not the answer • M&E works best when it is used to support conversations and not as hard and fast ‘rules’ • Information collection (resources) should be scaled to importance • Focus on measuring the achievement and trust has been demonstrated • Consulting and involving stakeholders will improve the “reality” of the final product, and help buy-in World Bank

  48. Large Scale Evaluations! • Prepare a draft terms of reference – purpose, objectives/questions, and methods • Put together the evaluation team • Brief and support the team before and during fieldwork • Critically assess the draft report World Bank

  49. Key Messages During the Planning Phase • Allow sufficient time to include stakeholders • Allow sufficient time for appropriate methods in the field, including participatory approaches • If possible agree the key questions for the evaluation when designing the M&E F/work • Use monitoring information and past evaluations to help answer the evaluation questions • Consider and agree involvement (who) and ownership together – sharing power World Bank

  50. Defining the Purpose • This process takes time….. • May require several rounds of consultation involving stakeholders • Formulate the purpose in a way that indicates how information provided will be used • If difficult, may signal evaluation not sufficiently useful to justify investment; consider postponing or whether needed at all • Timing is important – if evaluation can not be completed before is to be used to inform decisions; redesign or reconsider World Bank

More Related