International engagement: m & e meeting

International engagement: m & e meeting Monitoring & Evaluation: an introduction for practitioners Liz Allen

monitoring & evaluation can play a vital role in the management & improvement of an activity, organisation or process Source: Subsetum; http://www.flickr.com/photos/subsetsum/

Monitoring & evaluation: an introduction for practitioners my talk • Monitoring & evaluation: why do it? • Theory & practice • M & E: challenges • Keeping it real: practical & proportionality

Monitoring & evaluation: rationale rationale my funder requires it … • accountability & validation • management of funding initiatives • strategy & planning • policy & advocacy • learning I want my programme to work well … I want to do more in the future … I want to tell people about what we do … & show the benefits … I want to find out what works …

Theory

M & E framework • Monitoring • Set up review • Process review • Formative review • Summative review Source: Subsetum; http://www.flickr.com/photos/subsetsum/

M & E framework • Monitoring • Set up review • Process review • Formative review • Summative review • Inputs/activities • outputs • outcomes • … impact? • (longer term impact) Source: Subsetum; http://www.flickr.com/photos/subsetsum/

M & E framework • Monitoring • Set up review • Process review • Formative review • Summative review Things to do & things to ‘measure/track’ • inputs/activities • outputs • outcomes • … impact? • (longer term impact) Source: Subsetum; http://www.flickr.com/photos/subsetsum/

Theory & formal frameworks • Logical framework • Outcome mapping • Payback framework • Theories of change • Results-based management

Monitoring & evaluation: rationale M & E in project cycle Learning & lessons for next time Project start ‘ M & E Framework’ design Taking stock OUTPUTS/OUTCOMES Set-up/process review INPUTS Monitoring & review points INPUTS & OUTPUTS

Monitoring & evaluation: rationale Frameworks good for • keeping outcomes in mind • ‘forces’ definition of: • project objectives • indicators • appropriate data collection method & timings • organisation • simplification

Monitoring & evaluation: rationale Frameworks less good • keeping outcomes in mind • ‘forces’ definition of: • project objectives • indicators • appropriate data collection method & timings • organisation • simplification • accommodating externalities & context • dealing with the unexpected • understanding process

Practice doing m & e

Monitoring & evaluation: doing m & e (my) basic principles • involvement of stakeholders • definition of objectives/outcomes & associated indicators & method • integrated into project plan from start • properly resourced • practical (usable) & proportionate

Introduction to LFA: defining indicators an indicator “quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement.” “ something that helps us to understand where we are, where we are going and how far we are from the goal ... They are bits of information that summarize the characteristics of systems or highlight what is happening in a system. “

Introduction to LFA: defining indicators SMART & SPICED indicators SPICED Subjective Participatory Interpreted/communicable Cross-checked Empowering Diverse & disaggregated SMART Specific Measurable Achievable Relevant Timebound

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”attributed to Albert Einsteintheoretical physicist, philosopher & author 1879-1955

Quantitative • access & participation data (‘penetration’) • attendance & visitor data (‘reach’) • ‘audience’ characteristics (e.g. demographics) • consumer characteristics • scientific output assessment (e.g. publications) • ‘alt metrics’ • amount raised/ follow-on funding • structured opinion & feedback

Qualitative • interviews • open questions (in questionnaire) • focus groups • ethnography • participatory research • observation • comment/bulletin boards • visitor books

Monitoring & evaluation: an introduction for practitioners my talk • Monitoring & evaluation: rationale & overview • Theory & practice • M & E: challenges • Keeping it real: practical & proportionality

the elusive ‘impact’ time frame involved serendipity attribution & contribution ‘ripple effects’ counter-factual value of ‘negative’ findings

and closer to home … getting agreement on goals, objectives, indicators & method

and resourcing m & e properly … adequately, systematically, proportionately…

Monitoring & evaluation: an introduction for practitioners my talk • Monitoring & evaluation: rationale & overview • Theory & practice • M & E: challenges • Keeping it real: practical & proportionality

Monitoring & evaluation: keeping it real Summary • Understand stakeholder & audience requirements • Be prospective: build m & e in from the start • Chooseright method & tailor • Resourcing: ensure access to key data & information & someone to manage this • Consider option for trends & benchmarks • Keep it real: be proportionate & practical – measures can evolve • Be flexible – learning is iterative & should be part of the process • Beware over-monitoring & evaluation!

Wellcome Trust’s Indicators of Progress

Questions l.allen@wellcome.ac.uk

Suggested resources Ahmed, S. & Palermo, A-G (2010) Community Engagement in Research: Frameworks for Education and Peer Review. American Journal of Public Health, 100, 8: 1380-1387 Matthew, B. & Ross, L. (2010) Research Methods: a practical guide for the social sciences. Pearson Education Limited NORAD (2008) Results Management in Norwegian Development Cooperation: a practical guide.http://www.norad.no/en/tools-and-publications/publications/publication?key=109837 Stein, D. & Valters, C. (2012) Understanding ‘Theory of Change’ in international al development : a review of existing knowledge http://www2.lse.ac.uk/internationalDevelopment/research/JSRP/downloads/ToC_Lit_Review.pdf Vogel, I. (2012) Review of the use of ‘Theory of Change’ in international development. For DfID UK Web Center for Social Research methods http://socialresearchmethods.net

International engagement: m & e meeting