Creating Value, Evaluating Impact

1. Creating Value, Evaluating Impact • ‘Self-evaluation is a vital part of the discipline imposed in undertaking creative work’ (Moriarty, 2002) Kerry Traynor November 2012 What is Evaluation and why do it? About the Project Measuring Impact Theories and Models Next Steps

2. What is evaluation? • Evaluation involves gathering evidence before, during and after a project and using it to make judgements about what happened. • The evidence should prove what happened and why, and what effect it had. • The evidence can also help you to improve what you are doing during the project and what you do next time. (Woolf, 2004 cited in Arts Council, 2011)

3. Why evaluate? ‘Self-evaluation is hard work and time-consuming. The reward is that it can give us the ability to do things beyond the best of our present available knowledge’ (Moriarty, 2002) • evaluation helps with planning, as it makes you think about what you’re aiming to do, how you will do it and how you will know if you’ve succeeded • on-going feedback keeps you on track and helps to avoid disasters • evaluation helps you to adapt/change as you go along • evaluation is a good way of dealing with ‘quality assurance’ – you’re keeping an eye on things to make sure quality is maintained • evaluation helps prove the value of what you are doing • evaluation records your contribution to the field you are working in • your evaluation can help others working in the same field • information you collect can also be used for reporting back to those with an interest in the project (eg participants, funders) and telling others about what you’ve done • the evidence you collect can support future funding applications (Arts Council, 2011)

4. Evaluation helps the arts… • Rigorous evaluation of our work enables us to : • accumulate a collective body of evidence • contributes to the ‘collective practice wisdom of the sector’ • builds a record of our ‘history and achievement’ (Arts Victoria, 2002) • Cannot evaluate all projects in the same way and in the same depth • Can be more or less formal, and more or less detailed • Artistic judgements about process, materials, form and content • Judgements about the results of what you did and what you have produced • Process as well as product - quality and impact of both (Arts Council, 2011)

5. Planning your evaluation: • What kinds of information or evidence you are going to include in your evaluation • e.g. what people say, what they have done (process and finished work), what you have done, how participants/audiences responded? • What questions you are going to ask? • How you plan to answer those questions –what sort of information you need to answer the questions and how you will collect it. • When you should collect the information. • How you will collect the information? • e.g. keeping a register, asking people in a questionnaire, asking them to video their thoughts about a project, keeping a diary, taking photographs, etc. • How you are going to make sense of the information you have collected? • How you are going to present the results of the evaluation? • Who you are going to share it with and how? (Arts Council, 2011)

6. About the Project • Awards for All funding / Nov 12 – Aug 13 • Aims • Understand how impact is currently measured and evaluated in small enterprises in the creative industries • Understand the strengths/weaknesses/opportunities/risks of evaluating impact • Review models of impact evaluation and identify appropriate model(s) for use in creative communities network • Identify needs and deliver support to evaluate impact more effectively • Outcomes • Impact Evaluation Strategy for Safe Productions and the Creative Communities Network – finding a shared approach that can measure the impact of the whole network • One-to-one support for members in implementing impact evaluation • Interactive blog / web archive of useful info http://evaluatingimpact.wordpress.com/

7. Background • Charity Manager in Toxteth for 6 yrs • Director / Chair of Safe Productions for 10 yrs • Grants Assessor for BBC CIN for 10 yrs • Researcher & lecturer at LJMU for 8 yrs • Manager of Community Media Enterprise • MSc in Governance (Creative Industries) • PhD in Impact of Local Media

8. What do we mean by impact? • Inputs + Outputs = Outcomes • Outcomes – What would have happened anyway = Impact • Difference achieved by a project/service • Social, environmental, financial

9. Exercise • Introduce yourself and your organisation • What do you believe are the main impact(s) of your organisation? • What is your experience of impact evaluation? • What has worked well for you? • What hasn’t worked so well? • What would you like to do better? • What do you hope this could achieve? • What are your fears / concerns?

10. Models of Impact Evaluation • The Big Picture • Volunteering Impact Assessment Toolkit • ISO9001:2008 • Global Reporting Index (GRI) • Investors in People (IIP) • Eco-Mapping • EU Eco-Management and Audit Scheme (EMAS) • PQASSO • Social Return on Investment (SROI) • European Foundation for Quality Management (EFQM) Excellence Model • Prove It? • Quality First • Social Enterprise Balanced Scorecard • SIMPLE • 3rd Sector Performance Dashboard • AA1000 Assurance Standard

11. Why SROI? • Robust methodology developed through academic peer review process • Flexible, universally applicable • (Relatively) straightforward • Endorsed by Cabinet Office • Becoming common practise

12. SROI: A simplified example • Inputs : £20,000 • Outputs : 20 unemployed people complete a 6 month training programme and get a qualification • Outcomes : 5 people get a job • But : 2 people would have got a job anyway • Impact : 3 people move into employment • Attach monetary values (returns) to that impact e.g. • Reduced costs to state of paying unemployment benefits • Increased income to the state from employment taxes • Calculate returns over 5 years • Divide total returns by investment e.g. • £100,000 / £20,000 = 5:1 • Or, for every £1 invested, £5 is created in benefit for society

13. A collective approach – find your common impacts • Moving people towards employment / rehousing/ financial independence • Improving the physical environment • Improving people’s health & wellbeing Agree common methodology Measure and evaluate over time Convert into monetary values • Find your individual and collective • Social Return On Investment Rates

14. H&W: A simplified example • WEMWBS 7 or 14 point questionnaire • 5 response codes • For participants aged 13+ • Each participant completes at start/end • Points awarded for each response

16. H&W: A simplified example • Over 1 year, 500 people participate across all member organisations • Results in an average increase of 5 points per person, or • A total increase of 2,500 points across the area • Can be analysed by postcode/age/gender etc.

17. Visualising our collective impact

18. Visualising our collective impact

19. Visualising our collective impact The Health and Wealth of Nations The Decline: The Geography of a Recession

20. But does that really capture impact… ? • Can we really put a price on social impacts? • Will SROI let us tell the whole story? • Or will we need more? Eg. case study, images, quotes, narrative etc. • Creativity and the Creative Industries – arts & health • Community development – what is valuable and sustainable? How do we build strong and resilient communities? • Social Capital theory – our social, economic and cultural stock (Bourdieu, 1984) • The ties that bind us - bonding & bridging (Putnam, 2000) • Decline in social capital = decline in moral and social standards • Increasing social, democratic and civic engagement • Pride, shared identity, shared vision - challenging the negative, promoting the positive • Alternative socio-economic models – 21 hours (nef) • Can we use these theories and models to find a better way to demonstrate our impact?