MAKING THE BIGGEST DIFFERENCE

1. MAKING THE BIGGEST DIFFERENCE Using evaluative thinking to get the most out of your project

2. What will we do today? • Introduce ourselves • Making the biggest difference: Using evaluative thinking to get the most out of your project (and our role in helping you do this). • Vision: Inspiring and improving practice (activity: creating a vision map) • Evaluation in practice - Tools and techniques(activity: using critical questions to guide your choices of tools and techniques)

3. Point Research

4. Who are Point Research? • Independent research company, (est. 1997). • Focus on social change and innovation • supporting individuals and organisations to use evaluative thinking to make the biggest difference they can • capturing the impact and journey of social innovation (but we also do lots of other things) • Qualitative (stories) and quantitative (numbers) focus (capture hearts and minds

5. Some of our 2012 projects include… • Communities and change • Know Your Neighbours evaluation • Refugee Health Collaborative evaluation • Children, young people & their families and whānau • Amplify action enquiry - research team with children yrs 5-8 • Vodafone World of Difference evaluation • Youth “Voice of NZ” Survey (13-21 year olds) • ASAH – meeting the needs of 13-18 year olds who seek help for sexual violence or abuse • Youthtown i-project – developmental activities and opportunities for 9-13 year olds • Other • ADHB Health Voice • Te TimitangaHou – Homeless Court evaluation & report • Employment of people with disabilities formative evaluation • Plunket Family Partnership impact profiles Our experience with education/schools includes community consultation processes to inform strategic planning, evaluation of outside education programmes (e.g. Keeping ourselves Safe, sports development programmes), PAFT evaluation, education focus groups

6. What we would like you to think about… JRM have supported your project to build better family and whānau engagement in education to support better educational outcomes for disadvantaged children • How do you know you are building better family and whānau engagement? • Openness, honesty & ongoing reflection “what more can we be doing? How can we be doing it better.”

7. Thinking evaluatively Using evaluative thinking to get the most out of your project

8. How can we show that…? • We are building whānau and family engagement in education that supports better educational outcomes for disadvantaged children and; • What we are doing is making a difference to people’s lives

9. Outcome of the workshop • We can understand how to use evaluative thinking and evaluation methods and techniques to: • Support and accelerate the changes we want to make • Find out the difference we are making • Report and share our findings • We understand how what we are doing contributes our overall vision • We understand how what we are doing contributes to the overall vision of the Trust. • We understand the different roles of our evaluators and how they can help us achieve our vision.

10. How does social change happen?

11. Point Research role + support Support organisations to monitor engagement and service delivery Inspire good practice using reflection and evaluative thinking Advance the state of knowledge and practice

12. What is good to understand… • We are not adding another layer of accountability Instead… • We are adding another layer of thinking designed to help you make the most difference you can.

13. Demystifying Evaluation

14. What is evaluation? Aim of evaluation is simply to identify… • What is working well • What is not working well • What needs to change • What difference is it making (impacts and outcomes)

15. Developmental approach

16. Purpose of Developmental Approach A developmental approach means we continually check, reflect, adapt • Looking at what we do • Reflecting • Making changes to what we do We do some of this anyway (e.g. raising children): • No one right way • Formulas don’t work – feed water sunshine grow • Try things (reflect, adapt, check, reflect, adapt)

17. A developmental approach helps to improve and accelerate change • Which parts of what you do are making a difference (or the most difference) • For whom?

18. Aligns with funders’ requirements • What you did and why • What worked • What we can learn from this and what learnings we can share

19. Question that underpins evaluation How do you know?

20. Difference between opinion and evaluation… Evidence

21. Inspiring and Improving Practice Purpose: To improve and accelerate change

22. The importance of VISION • Literature is telling us … don’t lose sight of your vision • Clarity about what you are trying to achieve • Clear vision (dream) Quick Test – How would you explain your vision to someone at a bus stop?

23. Vision is what you are working towards so inevitably shapes what you do Vision 1 Families are engaged in their children’s early education Vision 2 All children have the opportunity to participate in early childhood education

24. What is your vision? Vision is your big picture or dream • What you want to see • Guides or drives what you do (What gets you out of bed in the morning? What keeps you going?) (note: Mission is your purpose or reason for being “Who we are and what we do".)

25. Why have a shared vision? A shared vision can be the foundation of change. • How do you know that your vision is one which is shared by others?

26. Workshop: Vision Mapping • Your vision • What is the change you want to see working towards your vision (outcomes – choose a couple) • What needs to happen to achieve this (2 or 3 key activities that underpin this) • (How do you know those are the right activities?) • How will you know if you have been effective (how will you measure it)

27. YOUR VISION Outcome 2(change towards your vision) Outcome 1 (change towards your vision) Activity Activity Activity Activity Activity Activity Measurement Measurement Measurement Measurement Measurement Measurement

28. Developmental approach • Keep visiting map you are doing • I.e. check, reflect, adapt • What is working well? • What is not working well? • What needs to change? • What difference is it making? (impacts and outcomes) • How do you know?

29. Improving and accelerating change is about … • Being intentional • Being audience or user focused • Thinking about what you are doing • Adapting • Doing what makes a difference • Letting go of what doesn’t

30. Remember… Hold tight to your VISION not your activities

31. Evaluation in Practice Tools and techniques to capture the change we are making

32. Evaluation basics… • What is it you need to find out? • Who will you find this out from? (service users, stakeholders) • How will find this out? • How will you share your findings? • Who will you share with them with?

33. Evaluation toolkit • What tools and methods to you need to measure the change you are making? • Variety of methods (“evaluation trends”) • Never one right (or wrong) method – it is all context dependent • Methods can be adapted and combined • Think about using a mixture of qualitative and quantitative methods

34. Two types of data • Qualitative data (hearts) • Best explain the why and how • Gathered from people, documents, case studies • Can include photos, videos, audio recordings & other non-text data • Quantitative data (minds) • Capture the what, who and when • Statistical representation (can be counted)

35. Common qualitative methods(and our favourites) • Interviews (conversations with a purpose - Alex) • Focus groups (‘conversation groups’, ‘café conversations’) • Anything which involves post it notes • Impact profiles (stories of change) • Online surveys • Observation

36. Common quantitative methods(and our favourites) • Surveys (paper-based and online) (primary data) • Secondary data e.g. • Census • MoE • MoH • Police • Can use creative ways of collecting – e.g. Warehouse “stand on the line”

37. How to choose the right tool • What do you want to measure? • Who is your audience? (tools for children, communities) • Where will you collect the data? • Who else will be involved? (participatory methods e.g. young parents/young parent-led) • What resources do you have? Need? • What are your skills? • Opportunities for collaboration/partnership?

38. Reporting your findings • What do you want to capture/need to report • How are you going to deliver it? (format) • Who are you going to share it with? Don’t forget to report back to your community/partners

39. Evaluation in Practice Workshop: Shaping your Developmental Approach

40. Activity • As a group, complete some (or all) of the critical questions worksheet (You don’t need to be confined to the worksheet, use paper, post its, pictures to help you form ideas/illustrate) You may already be working on a question in your initiative (and have some answers). In this case, we would like you to share what you are doing Or We may need to work out together what you can do. In that case, jot down your ideas as a starting point.

41. Need some help? Go back to the basics… • What is it you need to find out? • Who will you find this out from? (service users, stakeholders) • How will find this out? • How will you share your findings? • Who will you share with them with?

42. Some ideas to get you started? Think about your audience: • Ask! • Online surveys (free online tools survey monkey) • Short paper surveys • Feedback forms • Interviews • Focus groups • Café conversations • Graffiti walls • Continuums • Photo voice • Secondary data collection

43. Reporting Bringing it all together…

44. Writing an evaluation report “Great questions make great reporting”. (Diane Sawyer) • Three main areas: • Content • Look and feel • Usability

45. RECORD what you are doing as you go Record changes that you have made and why e.g. • Brokered a relationship between an agency delivering services to young people, help them relocate to the old school dental clinic at a school • Organised a celebrating success evening

46. Reporting • What do you want to capture/need to report • How are you going to deliver it (format) • Who are you going to share it with Don’t forget to report back to your service users

47. Hearts and minds. Think about… • What would an amazing report: • Contain (has to have what is needed) • Capture (share) • Look like (format) • Think hearts and minds (Might help to think of disappointments and what could be done better)

48. Reporting creatively • Make sure your hard work isn’t shelved. Your report may contain (or may consist of…): • Photographs • Stories/case studies • Websites with stories/videos • Video • Infographics • Slide shows • Quotes Any other ideas?

49. Example One: MPHS • We asked 9 – 13 year olds to tell us • what it was like to live in their community, • their hopes and dreams • how they like to be treated. • We chose a method which would capture their imagination (photos) and which would extend their skills. We also used this to report…

50. Tagging not a work of art just a disgrace to our community.Tagging it doesn’t show your artistic skills, it just shows your (recklessness) throughout the community.