Making evaluations more meaningful to local stakeholders

1. Making evaluations more meaningful to local stakeholders Presented by Peter Kiwumulo Founder Chairman Uganda Association for Social Economic Progress

2. Outline • What is evaluation? Different types • Why evaluation? • When evaluation? • Common characteristics of project evaluation – common assumptions and mistakes • The Ideal – principals and attitudes • The Ideal – methods • Common perceptions of local stakeholders of project evaluations • Making evaluations more meaningful to the local organizations and beneficiaries • Benefits • Challenges • References

3. What is project evaluation? • Complex – defining it poses difficulties as it takes some different forms – process and purpose • Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object www.socialscience.net (1-9-14) • Evaluation is a process that involves the systematic collection of information about activities, characteristics, and outcomes of an activity or action, in order to determine its worth or merit (Coastal Planning and Management Manual, 1998, pp 5.2) • The generic goal of most evaluations is to provide "useful feedback" to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. Most often, feedback is perceived as "useful" if it aids in decision-making – ibid, pp 5.3 • It’s a major part of learning, and can provide a wealth of useful information on the outcomes of a project or action and the dynamics of those who undertook the outcomes

4. Key questions to ask when embarking on evaluations • What is being evaluated? • Why? • How? • For whom? • Who? NB/ guiding questions should be able to inform the details of the evaluation exercise rather than standardized approaches • It calls for creativity and innovation !

5. Why evaluation? • Accountability • Relevance • Effectiveness • Efficiency – process • Impact • Sustainability – build capacity • Value for money • Replicability • Learning – decision-making • Proper exit – debrief to stakeholders • Empower participants • Build teams

6. When evaluation? • Any time –to be determined by reason and nature – need to be in built in the design, not spontaneous • Ex-ante, post-ante, mid-term • Initial, ongoing or at agreed milestone • Formative – summative

7. Common characteristics of evaluations • Mainly outcome-oriented – accountability driven • Donor driven – commissioned and financed • Mainly end line • Minimal control on the part of frontline stakeholders • Minimal participation • Top down – linear • Technocratic – expert led – scientific methods • Limited feedback

8. Common characteristics – cont. • Perceived as police-like by the frontline stakeholders – antagonistic • Too short – tourism • Expensive compared to the overall budget of the projects • Standardized – two weeks – two consultants • Largely ineffective in terms of causing change

9. The ideal ! • Pluralistic – inclusive • Participatory • Evolutionary • Innovative • Site specific • Reflective • Iterative • Insightful

10. The Ideal – cont. • Quotation: “A participatory or collaborative evaluation approach provides the partners and stakeholders involved with hands-on practical, experiential training in monitoring and evaluation techniques. It can contribute to the institutionalization and use of information for project improvement by local actors.” (Judi Aubel, 1999, pp 5)

11. Recommended methods • Be mindful of literacy levels and lack of knowledge of the value – apathy • Ethnography • Drama • Brainstorming – focus group discussions • Stakeholder analysis • Causal linkage diagramming • Social mapping • Gender analysis

12. Recommendations • Make evaluations friendly and useful to frontline stakeholders, i.e. implementers – local organizations’ staff, beneficiaries, local authorities – how? • Involve them effectively in planning, executing and evaluating the evaluation process • Use more user-friendly methods, e.g. storytelling • Change your mind set – exercise academic humility

13. Recommendations – cont. • Be innovative and consult them continuously • Prepare more – learn more, listen more (they know what affects them more) • Avoid biases – do reconnaissance – control ego • Understand the power relations and the varied stakeholder interests – avoid rumours • Give it more in terms of time – build it in and allow for internal evaluations on top • Use the vantage point of local experts • Provide feedback continuously to avoid suspicion

14. Recommendations – cont. • Engender the process • Promote minority interests • Contextualize working environment and level of institutional growth • Understand organizational power relations and existing struggles • Be positive – and empathetic • Be open and rational

15. Recommendations – cont. • Do avoid doing harm • Be guided by the bigger picture – mission • NB/ the locals are the biggest stakeholders • Their interests are the major reason why the project and the evaluation is being done • To be effective requires a change in mind-set (humility) and mobilizing for an evaluation culture within the local organizations and among the locals

16. Benefits • Empowerment • Increased utility • More reliable data • Sustainability • Capacity building • Team building • Harmony • Accountability • Corrective actions – enduring change

17. Challenges • Time • Resources • Mind set of donors and experts • Skills • Vested interests

18. References • Chambers, Robert (2007) ‘Who Counts? The Quiet Revolution of Participation and numbers’ IDS Working Paper 296. Falmer: Institute of Development Studies • Designing impact evaluations: different perspectives. Robert Chambers, Dean Kerlan, Martin Ravallion and Patricia Rogers, July 2009, international working paper 4

19. Thank you for listening