Communicating effectively evaluation findings

1. Communicating effectively evaluation findings GIZ evaluation workshop14-16 October 2014Kampala, UgandaGlenn O’Neil

2. Workshop schedule Day 1: communictiontheory, communication in evaluation, communication plan and the evaluation report Day 2: Promotion, messages, toolsand presentations Day 3: Practicalimplementation

3. Workshop objective Participants improve their know-how and skills in analysis, report writing and promotional tactics in order to communicate evaluating findings effectively

4. Exercise What are the challenges you face in communication and evaluation? Note down keywords on Post-It notes and post them on the whiteboards

5. 1. Definitions and communication theory

6. Evaluation The systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability. (DAC-OECD, 2002)

7. Communication The process of people sharing thoughts, ideas and feelings with each other in commonly understood ways Organisational communication: Activities that are dedicated to the management of communications between an organisation and its publics.

8. Whycommunicateevaluationfindings? Feedback Validate Inform Improve Aid Influence Change

9. Communication theory Organisation Public Message Silver bullet theory

10. Communication theory Minimal effects theory

11. Communication theory win-win Two-way symmetrical communications

12. Communication process Message Decoding Channel Encoding Noise Receiver Sender Feedback Frame of reference Frame of reference Environment

13. Theory to practice What can communication theory and its study tell us? • Environment and sender are important • The more targeted the more effective • Level of interactivity dictates level of influence • What influences understanding is not only the content but the way the message is communicated • To make people aware they need to be reached through different means • Communication rarely works 100%!

14. Exercise Communication challenges in the evaluation process Read the scenario assigned to you and determine what is the communication challenge? Channel, the sender, encoding, etc.?

15. 2. Communicating before, during and after the evaluation

16. What makes an effective evaluation? • Takes into account the context of the evaluation • Identifies the evaluation audiences and involves them early • Communicates frequently and reports interim results • Tailors reporting to audience needs • Reports results in a timely manner and to a variety of audiences • Presents vivid and concrete illustrations of findings • Use clear and jargon-free language Source: Torres, R. T., Preskill, H. S. & Piontek, M. E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Sage Publications.

17. Before, during and after • Communications needs to be planned and thought about in the initial stages of the evaluation • Responsibility for communications is often shared between the commissioner and the evaluator(s) • A lot of communications is internal (i.e. to staff, partners and governing bodies) and not external • Often we think about only communicating findings but we need to communicate before and during the evaluation process Source: Torres, R. T., Preskill, H. S. & Piontek, M. E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Sage Publications.

18. Before, during and after WHY? Source: Torres, R. T., Preskill, H. S. & Piontek, M. E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Sage Publications.

19. Before – key points • A lot of communication before involves planning for during and after – but not only • Communicating before often involves a lot of direct contact and discussion with stakeholders that are part of the evaluation process • Issuing the Terms of Reference is an opportunity to communicate to broader audiences

20. During - key points • The level of communications during the evaluation will vary depending upon the participative nature of the evaluation • As the evaluation team move towards results they need to consider how much they want to share their initial results • Communication during the evaluation needs to be flexible – e.g. a delay in data collection may mean more regular updates are needed

21. During - key points

22. Before, during and after scheme Before During After Standard evaluation products Evaluation report Preliminary findings Inceptionreport Terms of reference 1-1 discussions Infographic Video report Online discussion group Workshop Blog Communication activities(examples) Website text Webinar Email updates Social media page Media articles 1 page snapshot Communication planning steps Consider key messages Refine communication plan Design communication plan of action • Develop ownership • Create awareness • Build relationships • Set expectations • Support access • Facilitate process • Communicate progress • Support validation of findings • Disseminate findings • Create awareness • Encourage use Communication objectives(examples) Main lead for communication Commissioner Evaluator

23. Exercise Before, during and after -Break into groups -Read the case study: -Consider what are 3 opportunities for communications in this evaluation -Consider what are 3 challenges for communications in this evaluation

24. 3. The communication plan

25. The communication plan Start with two questions: • What is particular about this evaluation context? ---- > situation analysis • How can communications support this evaluation? ---- > Objectives

26. The communication plan Stakeholder/audience analysis • Primary: Management, sponsors, donors, programme staff, governance • Secondary:Partners, related-programme staff • Tertiary: Like-minded bodies, potential partners, academics, persons working in sector

27. The communication plan Communication activities • These are not your daily activities to keep the evaluation running; more so activities that are planned systematically to support your objectives • Choice of activities depends upon many factors, including; • Suitability for audiences • Level of interactivity desired • “Depth” of information to be communicated • Budget available

28. Activities - communication plan

29. Activities - communication plan

30. The communication plan - template

31. Exercise The communication plan -Break into groups -Create a communication plan! - Report back to plenary

32. 4. Creating the evaluation report

33. Exercise What are the Do’s and Don’ts of the evaluation report? Note down keywords on Post-It notes and post them on the whiteboards

34. The evaluation report • The report remains a key communication tool • It also serves as a source from which other summaries and tools will be drawn from • We can consider best practices in terms of: • Structure • Content • Design

35. The evaluation report - structure • There is not a set structure but most reports contain: • Executive summary • Introduction • Short description/background of project/programme • Methodology description • Findings • Conclusions and recommendations (lessons learnt – optional) • Annexes • Findings can be organised on the basis of the evaluation questions/criteria, the theory of change, major programme activities or themes that emerge from the evaluation • Regardless of structure you need to respond to the evaluation questions!

36. The evaluation report -content • Write in a logical manner • Avoid compartmentalizing your findings • Link findings to conclusions to recommendations

37. The evaluation report -content Q. Is the Central Registry an effective tool in obtaining information and extracting disaster management information? This review has found that the Central Registry has been limited in its effectiveness in providing disaster management information. The CR has not substantially contributed to users obtaining information and consequently using this information to facilitate the rapid delivery of emergency assistance. The out-of-date nature of the information was raised consistently by survey and interview respondents as being a major impediment to the CR being an effective tool. What is the extent of out-of-date information on the CR? Based on a content analysis of contact data for six CR directories, it was found that on average only 34% of the content has been updated in 2013 and 2014 with wide variation per directory as show below. Further to the existence of other sources, users, potential users and stakeholders questioned if the concept of the CR, an online centralised database, was the most efficient tool for obtaining and extracting information on disaster management…All major donor governments interviewed shared this opinion as summarised by this comment of an emergency management official and CR user: “The use of the CR seems a little bit outdated due to the fact that requests for assistance are not made out of the resources listed in the directory but out of needs in the actual situation and response is offered from where the resources are available.” Conclusion: The out-of-date nature, the incompleteness of the information in CR and the poor user experience it offers (discussed further below) have all impeded the small number of users that sought to use the tool for the purpose for which it was conceived… Consequently, the CR is not an integral or central part of emergency response processes of governments, NGOs and the UN – including OCHA Example of a logical sequence You answer directly the question You provideevidence to suppport your answer Display keydata in agraphic form You expandupon the initialfindings Provide supportingevidence You indicatethe whatthismeans 37

38. The evaluation report -design Use of headings and sub-headings Use of lists & boxes to break up text Contrast betweenhighlight box & text Repetition of colours & fonts Alignment ofinformation Use tables, charts, illustrations to break up the text--- examples on next pages

39. The evaluation report -design Examples: showing project’s activities/progress over time

40. The evaluation report -design Examples: summarising progress

41. The evaluation report -design Examples: scorecards to summarise progress/rate activities

42. The evaluation report -design Examples: illustrating processes

43. Exercise The evaluation report - Break into groups • Review the case materials provided • Carry out exercise • Report to the plenary

44. Summary of day one