Multi-strand initiatives: using theory of change evaluations

1. Multi-strand initiatives: using theory of change evaluations Alan Dyson: D.A.Dyson@manchester.ac.uk Kirstin Kerr: Kirstin.Kerr@manchester.ac.uk Karen Laing: k.j.c.laing@newcastle.ac.uk Liz Todd: liz.todd@newcastle.ac.uk

2. Introduction • Aims • The team – Alan Dyson, Karen Laing, Liz Todd • Plan: Introduction to theory of change evaluation Theory of change in action Using evidence Final thoughts

3. Full service extended schools national evaluation Colleen Cummings, Alan Dyson, Ivy Papps, Daniel Muijs, Diana Pearson, Carlo Ruffo, Lucy Tiplady, Liz Todd: Newcastle University, University of Manchester, Tecis Ltd http://www.ncl.ac.uk/cflat/ESnetwork.htm Cummings Dyson & Todd Cummings, Dyson, Muijs, Papps, Pearson, Raffo, Tiplady & Todd Dyson & Kerr

4. Theory of Change A systematic and cumulative study of the links between activities, outcomes and context of the initiative Fullbright-Anderson, Kubisch and Connell, 1998: 16

5. Theories of Change • Popularized in 1990s to capture complex initiatives • Outcomes-based • Causal model • Articulate underlying assumptions

6. What is involved in theory of change…. • Qualitative and quantitative data collected over time • Identify early, intermediate and long-term outcomes • Theorise retrospectively and prospectively

7. The starting situation What it’s like now – and why Strands of action What we are going to do about it Intended outputs What we will need to do differently Steps of change How things will change (for beneficiaries) Intended outcomes How the starting situation will change for CYP

8. Steps of change

9. How a Theory of Change approach can help • Working together • It is dialogical, involving conversation and negotiation • Enables continuous feedback • Can feed into project planning • Provides guidance about data collection methods and self evaluation

12. Asking the right questions • What is the situation you face? • What needs to change? • How will these changes be made? • What actions will you take? • What effect will those actions have? On whom? • How will you know if change is happening? • What will happen for person A, person B etc

18. Presenting problems: Youth unemployment Youth disorder Poor mental and physical health – unfit families Children starting KS3 below level 3 Poor early language development ‘Troubled’ families High benefit claimant rates – disability and JSA A Children’s Community will seek to achieve the following outcomes for the area: Reduction in presenting problems (no noticeable difference to other areas of N. Tyneside) The area has the same opportunities and support structures as more advantaged areas. It provides the same ‘offer’ in terms of life chances (i.e. employment/ parental support/community support). The community supports the realisation of aspirations. It establishes & sustains another way of behaving – i.e.: -the community becomes self-empowered -parents expect better for their children and act on this (services stop substituting for parents) -the community is ‘fit for life’ mentally & physically -there is a positive community identity linked to employment -community members can connect to ‘wider world’ contexts and opportunities There is upward mobility, both in and out of the area The area attracts and sustains employment Presenting as: The perpetuating situation (c.2003) The Children’s Community Suggested core strands of action for the Children’s Community: 1. Getting things right early – e.g. speech, language communication & parenting. Aim to set children on positive pathway rather than addressing deficits later 2. Being fit for life – e.g. making sure children are healthy (physically and mentally) and are able to engage with opportunities 3. Realising aspirations & behaving in new ways – e.g. empowering the community to support children to achieve their aspirations and to do things they have not done before – making the community positive for residents and attractive to others (inc. employers). Past context: Recent context: Resulting in: Underlying problems: In some cases: Parents have low expectations of their own children Families don’t know how to be good parents or find it too hard Families abdicate responsibility to services or refuse to engage with services The hard to reach remain hard to reach Children have aspirations but not the opportunities / capacities / support to act on these Community not supportive of aspirations Opportunities that are available are not recognised and acted on Ship Building (Lost) 100% employment (Lost) Traditional gender roles (Maintained) Self-contained community (Maintained) Sufficient community fabric (Fractured) Lack of jobs / unemployment Inability to connect to local service jobs Reluctance to commute (locally/into Newcastle) Resentment – feeling that something has been taken away from the community (not always sure what) Lost generation of grandparents / parents Loss of area esteem – some fracturing of community (e.g. parts of the area ‘in work’ resented) ‘Unhealthy place’ physically & mentally

19. Getting things right early Good schools capitalise on better prior attainment Better opportunities available in area Better parenting in the home Children with capacity to take advantage of schooling (confidence & resilience) Children doing well through transition • Getting things right early actions: • Transition support • Support for parenting • High quality pre-school provision Access better post-school opportunities (including jobs) Higher attainment in primary schools Higher attainment throughout schooling Higher aspirations Parents with higher aspirations for their children

20. Mentors: • access children in primary schools • identify children who may have problems later but who do not have additional provision because they are coping in primary school (Jan-July) • provide nurturing relationships and establish friendship groups • liaise with parents • offer summer school in secondary school • identify issues secondary school needs to be alerted to Children learn to trust mentors who become ‘the trusted adult’ Parents ‘buy into’ the programme & encourage their children to participate Children become familiar with secondary school & become more confident about transition Secondary school prepares appropriate action Children’s confidence grows • In secondary school (autumn term), mentors: • maintain contact with children (eg through breakfast clubs) • monitor children in school to identify emerging problems • equip children with coping skills (eg anti-bullying strategies) • pass information onto guidance team (& Relate counsellor in Churchill) • wean children off support so they can cope alone Secondary school implement effective support strategies Children develop coping strategies Guidance team implement strategies Children cope with transition with support Children cope with secondary school without support Transition mentors

21. Drawing conclusions • Formative: • ‘Initiative X is triggering changes a.b.c….which are likely/unlikely to lead to outcomes 1, 2, 3….’ • Summative: • ‘Initiative X triggered changes a.b.c….leading to outcomes 1, 2, 3….’ • ‘Initiative X also triggered changes l, m, n….which may yet lead to outcomes 1, 2, 3….’ • But: • ‘Initiative X# may not trigger changes a.b.c….or lead to outcomes 1, 2, 3….’

22. Drawing conclusions • Building a theory: • Initiatives X, X#, X##...triggered similar changes leading to similar outcomes’ • Initiatives of type X are likely to lead to similar outcomes • ToC and ‘what works’ • Using a D & R approach