Organisational conditions for meaningful evaluation

1. Organisational conditions formeaningful evaluation Ian C Davies President – European Evaluation Society Stirling, September 14-15, 2010

2. Evaluation and generally accepted principles for increasing the likelihood of use, tend to focus at the program level and on the “stakeholders” of the programHowever most “programs” operate within an organisational environment that, beyond individual stakeholders, also affects the likelihood of use, often to a significant extent

3. The purpose of this presentation is to identify and discuss some key organisational characteristics that can have an important bearing on evaluation, its use and its meaningfulness.

4. UseTypically the question of use has focussed on the application of evaluation findings by intended users to support, inform, contribute to decision making.Participation in the evaluation process is seen primarily as a means to promote legitimacy and ownership of the evaluation findings and results.

5. The reality of evaluationEyewash evaluationsWhitewash evaluationsSubmarine evaluations Posture evaluationsPostponement evaluations

6. What are some key organisational conditions for meaningful evaluation? Does the organisation allow for ….?

7. In Peter Ustinov’s 1956 play, “Romanoff and Juliet”, a crafty general is the head of a tiny European country being squabbled over by the United States and the Soviet Union, and is determined to play one off against the other. He tells the U.S. Ambassador that the Soviets have broken the Americans’ secret code. “We know they know our code,” the Ambassador, Moulsworth, replies, beaming. “We only give them things we want them to know.” The general pauses, during which, the play’s stage directions say, “he tries to make head or tail of this intelligence.” Then he crosses the street to the Russian Embassy, where he tells the Soviet Ambassador, Romanoff, “They know you know their code.” Romanoff is unfazed: “We have known for some time that they knew we knew their code. We have acted accordingly—by pretending to be duped.” The general returns to the American Embassy and confronts Moulsworth: “They know you know they know you know.” Moulsworth (genuinely alarmed): “What? Are you sure?” (Malcolm Gladwell – Pandora’s Briefcase– The New Yorker – May 10, 2010)

8. ClarityWhy are we really doing this evaluation?The “dual purpose” of learning and accountabilityLeadership, management direction, communication

9. LearningIs the organisation really interested in learning?Organisational knowledge, intelligence, memoryResearch and development, protection of assets

10. ResourcesDoes the organisation really take evaluation seriously?Functional capacity for evaluation, profile, budget, positionOrganisational capacity, quality, monitoring and reporting

11. UseIs the organisation really open to change?Process, participation, authority, quality of managementWorking environment, results based management, evidence based policy, complexity

12. Checking organisational conditions:To undertake evaluationTo develop evaluative thinking

13. It is not because I weigh myself that I lose weight

14. THANK YOU Ian C Davies President – European Evaluation Society