Phase 2 Evaluation of the PD

1. Phase 2 Evaluation of the PD Key Issues for Evaluation Design Reference Group Workshop/Meeting 11 – 13 February 2009

2. Evaluation Design Many choices must be made when finalising an evaluation design, for example, we have to choose: • The main focus and purpose • The Unit of Analysis – what are the investigations located • Methods and techniques • Information sources • Management and governance etc. All of these are discussed at this workshop

3. Evaluation Design In this session we want to concentrate on two important design choices: • The evaluation questions that should be prioritised in the evaluation • Some propositions (or hypotheses) that follow from the characteristics of the PD itself

4. Evaluation Questions Evaluation questions are at the centre of any evaluation – what are they? At simple level they are questions we want answers to and we are likely to be able to use the results to inform policy or practice Evaluation questions also concern effects and imply judgement – placing a value on these effects

5. Evaluation Questions So an evaluation question : • Concerns an effect or result external to the interventions of the programme or policy: if a question is just about the internal working of the policy it’s better dealt with by monitoring or auditing! For example……………

6. Evaluation Questions An evaluation question also implies a judgement criterion: Some evaluation questions at least should help us understand what constitutes ‘success’, ‘achievement’ or ‘good’ performance For example……………

7. Evaluation Questions Evaluation questions lead to certain types of ‘enquiry’. They can be: • Descriptive: so that the evaluation describes what happens in a situation or over time • Explanatory: so that the evaluation investigates causal relationships • Normative: so that the evaluation focuses on how to bring about desired change For example…………

8. Evaluation Questions In the Approach Paper & the ‘linkages’ study different types of evaluation questions are identified. The main types are: Results questions. For example: How has the adoption of PD principles and commitments improved development results And Counterfactual questions. For example: Are there other ways – not following PD principles – that could lead to similar results more effectively & efficiently

9. Evaluation Questions The emphasis on results follows from the focus of the Phase 2 evaluation. Of course we can ask ‘process’ and implementation questions as well if these processes help us understand results

10. Evaluation Questions But these are examples only – in the group discussions after this presentation you will have the chance to put forward what you think are the most important questions for the evaluation to answer

11. Characteristics of the PD Many evaluation design choices aren’t choices at all – they follow from the character of the policy or programme being evaluated! For example the linkages study identifies some of the special qualities of the PD that make this evaluation very challenging

12. Characteristics of the PD For example we know that: • The PD is closely connected to other aid and development policies: how to separate out PD effects? • Development actors (governments, civil society etc.) will use PD arrangements differently – there won’t be one standard model • The PD is not a single once-only intervention it is a development and learning process • Not all results of the PD will be evident by 2010!

13. Characteristics of the PD • Evaluation will be difficult but possible! The ‘linkages’ study identifies 21 ‘propositions’ (or hypothesis) that research suggests could be tested in the Phase 2 evaluation. These are on Page 16 of the Approach Paper. • These ‘propositions’ come from research rather than practice. This is why it would be useful for people here who are familiar with aid & development policy to consider and review these propositions

14. Discussion Tasks The two tasks for the discussion groups will concentrate on these two design choices: • Evaluation Questions • Evaluation Propositions • But first are there any questions or points of clarification……..?