Approaching the intangibles e-VAL – including different perspectives into evaluation

1. Approaching the intangiblese-VAL – including different perspectives into evaluation Presentation during LenCD Meeting Nairobi, October 2006 Karsten Posse, GTZ Evaluation Department

2. Overview • Understanding of Capacity Development • Challenges for Evaluation • Methodology and Interview • Delivered Results (example)

3. Understanding of Capacity Development • DefinitionThe process of strengthening the abilities or capacities of individuals, organisations and societies … in order to achieve their own goals on a sustainable basis. *1 • for people transfer of knowledge, experiences, skills and values, behavioural changes, … • for organisationsestablishment of management and incentive systems, development of organisational culture, improvement of work processes, … • for institutions and policiesimprovement of legal framework, gearing of policies to development, promotion of cooperation between state, private sector and civil society *1 GTZ Policy Paper No. 1 – Capacity Development for sustainable development, 2003

4. Understanding of Capacity Development Stability known coastline Goal What does that mean for Evaluation? good Planning knowledge of present position calculation of delta Use of objective data Use of predefined evaluation methods

5. Understanding of Capacity Development Instability unknown waters Goal adjustedGoal What does that mean for Evaluation? Understanding of Process (process costs) Moving step by step – no direct line to follow

6. Understanding of Capacity Development • Approach Purely technical approaches mostly don't work . CD requires complex change processes on individual, organisational social and political level in order to ensure sustainable development. • Assesment of success Proxy Indicators might help to measure some results of CD, but vital factors are intangible:ownership, participation, political willingness, appropriateness of concepts in culture and tradition, … *1 GTZ Policy Paper No. 1 – Capacity Development for sustainable development, 2003

7. Methodical Challenges (1) Use subjective perceptions - Involve the “right” people The best experts for evaluation of Capacity Development are the people that are involved and close to the process.

8. Methodical Challenges (2) Triangulate the different perspectives Experts from different important interest groups (Donors, Partner Organisations, Intermediaries, Civil Society representatives) have to be included to ensure balanced and comprehensive results (triangulation).

9. Methodical Challenges (3) Use content sensitive instruments that allow comparism Questionnaires • Useful for simple and describing questions • Appropriate for representative studies (big numbers) • Allows easy aggregation • Less sensitive for cultural differences • Prompting issues and answers • Problem of relevance of questions • Understanding of questions Qualitative interviews • Using and showing expertise of the interviewee • Only personal relevant issues covered • time consuming to analyse • hard to compare Challenge: to use the best of both approaches

10. Methodology and Interview • e-VAL is based on qualitative structured interviews*1 conducted by specially trained neutral interviewers. • e-VAL does not provide defined answers. Each interviewee raises the issues which are personally important (original phrasing taken) • The interviewee rates progress of CD, cohesion of interests, quality of contributions and appropriateness of concept regarding his/her issues • The interview is been directly recorded in a specially developed software which allows instant mathematical procession • Immediately after the interview a model of the personal perception is calculated, discussed and authorized by the interviewee *1 based on the „Repertory Grid“ by G.A. Kelly, 1955

11. Methodology and Interview

12. Delivered results • The individual models are aggregated on a central server by stakeholder groups within minutes • Figures allow a quick and easy overview over general tendencies • Possibility for deeper analysis showing strengths and weaknesses in original phrasing of the interviewees • Detailed analysis of CD process and results • SWOT Analysis (strengths, weaknesses, opportunities, threats) by different stakeholders • Overview over cohesion of concepts • Assessment of stakeholders contributions in the process (stakeholder analysis) • Assessment of quality of the process planning and concept

13. e-VAL application “PRS assessment Armenia” Background information Date of evaluation: • May 2005 (1 ¾ years after begin of process) Interviewees • 6 donors • 4 Armenian Government representatives • 7 representatives of Civil Societies (NGO)

14. e-VAL application “PRS assessment Armenia” • e-VAL results • Very big differences in assessment of progress reached so far • Government representatives: 64%, • Donors: 50%, • Civil Society representatives: 14% • Donors and Government shared the same ideas about an appropriate strategy and believed that NGOs think in same. In fact, NGOs ideas regarding an appropriate strategy were very different • Donors und NGO described lack of participation and decentralisation in the process

15. e-VAL application “PRS assessment Armenia” Interpretation of results (original phrasing) Criticism of Civil Society representatives - too much focus on economic growth- no focus at all on education and health - PRS not anchored at local level - Donors dictate their ideas Criticism of Donors- strategy not balanced, still lack of planning skills- no vision of government of how local development can be fostered- no exit strategy (self criticism by donors) Use of results Common workshop with all different stakeholder groups - transparent discussion of the evaluation results - common agreement on areas where action is necessary - adoption of strategy for the coming year

16. Thank you for your attention

17. Appendix 1

18. optional specify Must How was the situation two years ago? How do you assess the CD Process at present? Beginning of project Project today What development do you expect within the next two years? Project in a year optional What do you realistically expect to be the future impact of this Capacity Development? Future project effects Must Elements – Guiding Questions So called “elements” are structuring the interview questions: 1. Process analysis – Strength, Weaknesses and Opportunities

19. Elements – Guiding Questions 2. Comparison of Success concepts of all stakeholders What characterizes for you the ideal CD Process? Project success must must What characterizes for you a (virtual) failure of the process?When would you say it has failed? Failure Project success from client view Which criteria,do you think, do the different donors have? What characterizes success for them? specify Project success from counterpart view must specify What do you think are the criteria guiding the partner/s? Project success from target group(s) view mustspecify. What do you think are the criteriaguiding the target groug/s?

20. Elements – Guiding Questions 3. Assessment of Contributions of all stakeholders (stakeholder assessment) How do you assess the German contribution to the process? What was helpful? What should have been different? German contribution (GTZ) Must How do you assess the contribution of the Partner/s to the process? What was helpful? What should have been different? Must specify Counterpart contribution How do you assess the contributions of other stakeholders to the process? What was helpful? What should have been different? Contribution of other stakeholders optional specify What are the frame conditions furthering or hindering the CD process? Framework conditions Must

21. How do you evaluate the quality of preparations before the process has started? Preparation of project Must How do you evaluate the current objectives of the CD process? Project goal today Must Elements – Guiding Questions So called “elements” are structuring the interview questions: 4. Quality of Process Design

22. Appendix 2

23. Results of a single interview Result of single interview