Evaluation demystified: Part 2 How to be confident, credible and creative Karen Gray

1. Evaluation demystified: Part 2 How to be confident, credible and creative Karen Gray Willis Newson

2. Presentation agenda • Background • Academia ↔ Practice • Making it credible • Making it creative Reflective practice drawings by Elizabeth Jane Grosse for Arts at Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health Partnership NHS Trust. Image: Willis Newson.

3. Background: Willis Newson Decorative glazing by Stuart Low in Older Adult Unit, Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health NHS Partnership Trust. Photo; Paul Highnam Specialist arts and health consultancy developing and delivering projects in acute, mental health and community settings Long-term collaboration with Norma Daykin, Professor of Arts and Health at University of West of England UWE led research-based evaluations of Willis Newson arts in mental health programmes: Moving On & Arts @Callington Road Collaborative research and evaluation work culminated in 2 year Knowledge Transfer Partnership (2010/12)

4. UWE-Willis Newson KTP • Improved understanding of academic research and practice-based evaluation • Mapped current state of evaluation knowledge and provision within arts and health practice • Identified barriers to delivery of credible evaluation within arts and health practice • Internal - a sustainable, credible and creative evaluation framework that we use across our own projects • External - an approach to sharing knowledge and expertise with the sector through the delivery of mentored or guided evaluations and CPD

5. Academia ↔ Practice: Bridging the Gap • Evaluation – Research continuum • High quality credible evaluation will help create fertile ground for research • Research may provide the kind of ‘hard’ evidence we need to convince policy makers, commissioners and funders to support our work • The rigour of research can be applied to evaluation to ensure credibility at all levels • The creativity of practice can be applied to research to generate rich, meaningful data about complex interventions

6. Creative and Credible Credible Creative Credible Meaningful Appropriate Analysable • Timing • Appropriate Tools • Avoiding Bias • Accurate analysis and reporting • Ethics

7. Making it Credible : Timing Project development Post-project Implementation

8. Making it Credible : Appropriate Tools • Appropriate for what? • The evaluation question • The stakeholder group or participants • The context • The scale of the project • Available time and budget • Focus groups will not give you quantitative health outcomes data • Validated scales are not appropriate for small projects with fewer than 100 participants • Questionnaires may not be appropriate for people with questionnaire fatigue

9. Making it credible : Bias • If the project artist asks participants what they gained from the arts workshop will they get the same answer as an ‘outside’ evaluator • If the project manager analyses the feedback forms will they include negative as well as positive information in the findings? • If you tell focus group participants that the evaluation findings will help secure funding for next year’s project will they give you the full story? • If you select case study participants because they have a really good story to tell about the project, will it give a true picture of what people got out of the project? • How will you ensure objectivity and safeguard against bias?

10. Making it Credible : Analysis & Reporting Image from a ‘Savouring & Flow’ workshop run by Light Box as part of The Happiness Project. Photo: BS6 Photography. Be careful what you claim! If you have 100 participants for a project and 10 fill in feedback forms, but all 10 say that they enjoyed the project, can you say that “participants enjoyed the project”? If you use a validated scale, but out of 200 participants, you only manage to get beginning, middle and end scores for 30 participants, can you claim that these scores are representative of the project as a whole? A good project and credible evaluation can easily be undermined by over claiming.

11. Making it Credible : Ethics Respect Appropriate and sensitive to the needs and vulnerabilities of participants Participants’ right to decline to take part in an evaluation Participants’ right to withdraw from the evaluation Professionalism Everyone involved fully informed and adheres to ethical code of practice Responsibility Privacy and confidentiality of participants Ensure participants’ anonymity Participants are fully informed before they provide consent to take part Permission obtained to use images, artwork or comments Integrity Report evaluations accurately and transparently

12. Creative and credible Formal / Structured Creative / Innovative “Creativity is not an escape from disciplined thinking, it is an escape with disciplined thinking.” [Jerry Hirshberg]

14. Creative Credible Meaningful Appropriate Analysable

15. Creative Credible Meaningful Appropriate Analysable

16. Creative Monitoring station, The Happiness Project. Image: Light Box. Credible Meaningful Appropriate Analysable

17. Creative Collage created during a focus group exercise designed to explore the impacts of a project on participants. The quote opposite is from the same participant describing her collage Credible Meaningful Appropriate Analysable “When I came along I felt quite a broken person. I’ve realised you don’t have to be perfect - broken bits make a person. Coming on the course let me stand back and see the whole person.”

18. Creative and Credible Credible • Timing • Appropriate Tools • Avoiding Bias • Accurate analysis and reporting • Ethics Creative • Credible • Meaningful • Appropriate • Analysable

19. In summary… • Practice and academia working together and learning from each other • Robust and credible evaluation practice creates fertile ground for research • Research providing the ‘hard’ evidence we all need • The rigour of research can be applied to evaluation to ensure credibility at all levels • The creativity of practice can be applied to research to generate rich, meaningful data about complex interventions

20. Thank youkaren@willisnewson.co.uk