Using theories of change and logic models to understand and evaluate (educational) interventions

1. Using theories of change and logic models to understand and evaluate(educational) interventions Lauren Wildschut 2 July 2015

2. Workshop overview We will examine the differences between Theories of Change (ToC) and Logic Models (LM) and then examine examples of each of these types of models and discuss how they can be used during the different phases of evaluation

4. Evaluation process and programme life-cycle link Primary focus of session

5. Purpose Clarificative evaluation clarifies the underlying rationale of a program. Program developers use this information to think through and make explicit the logic that supports the program, including assumptions about how its components link to produce the desired outcomes. Whereas clarificative evaluation would usually occur before the implementation of a program, it may also be carried out while a program is operating if it is not clear how it was intended that the program was to be delivered. Therefore, it has a formative evaluation orientation (Brian J. English, Rick Cummings and Ralph G. Straton p 126 http://www.popcenter.org/library/crimeprevention/volume_14/05-English.pdf)

6. Programme design - key questions

7. Theories of change (ToC) and logic models (LM) A logic model or theory of action depicts, usually in graphic form, the connections between program inputs, activities and processes (implementation), outputs, immediate outcomes, and long-term impacts... The only criterion for a logic model is that it be, well, logical, that is, that it portrays a reasonable, defensible, and sequential order from inputs through activities to outputs, outcomes, and impacts. A theory of change, in contrast, bears the burden of specifying and explaining assumed, hypothesized, or tested causal links. Logic models are descriptive. Patton (2002:162-163)

8. Relationship between ToCs and LMs

9. Programme theory and programme logic A program theory is an explicit theory of how an intervention is understood to contribute to its intended or observed outcomes; ideally includes a theory of change and a theory of action ( Funnel S and Rogers P 2011)

10. Theories of change

11. Theory with “a small t” It is an acknowledged embarrassment to our viewpoint that social science theory is not well enough developed that appropriate theoretical frameworks and schema are ordinarily easily available off the shelf. But the absence of fully developed theory should not prevent one from using the best of what is already at hand. Most important of all, it is necessary to think theoretically, that is, to rise above the specific and the particular to develop general understandings of social phenomena (Chen & Rossi 1997:285).

12. Theory of change: definitions

13. Programme theory and programme logic Wyatt Knowlton L and Phillips C.C. (2013:5)The Logic Model Guidebook

15. Single Strategy Do Get Multiple Strategies Do Get Strategy Strategy Results Results Strategy Strategy Strategy Wyatt Knowlton L and Phillips C.C. (2013:18-19)The Logic Model Guidebook

16. Weiss, (1997 :504)

17. Anderson A :4

18. Donaldson and Gooler 2003:358

19. ToCs are derived from evidence and practice Knowledge Research Do Get Believe Assumptions Results Strategy Practice Theory Wyatt Knowlton L and Phillips C.C. (2013:22) The Logic Model Guidebook

20. Elements of a ToC There is consensus on the basic elements that make up the theory of change approach. As a minimum, theory of change is considered to encompass a discussion of the following elements: •Contextfor the initiative , including social, political and environmental conditions, the current state of the problem the project is seeking to influence and other actors able to influence change • Long-term change that the initiative seeks to support and for whose ultimate benefit • Process/sequence of change anticipated to lead to the desired long-term outcome • Assumptionsabout how these changes might happen, as a check on whether the activities and outputs are appropriate for influencing change in the desired direction in this context. • Diagram and narrative summary that captures the outcomes of the discussion. Vogel I (2012: 4)

21. Methods for developing a ToC • Identify a causal mechanism theory and an action theory A theory of change ideally has two components – a causal mechanism theory, which explains how change occurs, and an action theory, which explains how the program activities trigger the change process … • Draw on research-based theories of change Research of various kinds can provide potentially relevant theories about how change occurs for individuals, families, households, and communities… • Use an outcome hierarchy format for the logic model rather than a pipeline A third strategy to make it easier to articulate an actual theory of change is to use a different format for representing it in a diagram. In particular, it is often useful to show it as a chain of outcomes rather than as a pipeline diagram of inputs-processes-outputs-outcomes-impacts… (Rogers, 2013).

22. Key purposes of ToCs 1. Strategic planning: ToC helps organisations practically to map the change process and its expected outcomes and facilitates project implementation. For these purposes, ToC is often used in conjunction with logframe approaches. 2. Monitoring and evaluation: ToCs articulate expected processes and outcomes that can be reviewed over time. This allows organisations to assess their contribution to change and to revise their ToC. 3. Description: ToC allows organisations to communicate their chosen change process to internal and external partners. A simple description of an organisation’s ToC can be understood as minimal way of engaging with ToC. 4. Learning: ToC helps people to clarify and develop the theory behind their organisation or programme. This relates to an understanding of ToC as a thinking tool.

23. Benefits of developing a ToC • Developing a common understanding of the work and surfacing any differences. • Strengthening the clarity, effectiveness and focus of programmes. • Providing a framework for monitoring, evaluation and learning throughout a programme cycle. • Improving partnership by identifying strategic partners and supporting open conversations. • Supporting organisational development in line with core focus and priorities. • Using theory of change to communicate work clearly to others and as a reporting framework. • Empowering people to become more active and involved in programmes. (James C 2011:4)

24. Logic models

26. Options for representing programme logic How can organisations represent the complex processes of change that they are engaged with, at local, national and international levels, along with a host of other actors, many of whom do not share the same objectives? Without adequate representation, it is much more difficult for an organisation to propose, test and improve its “theory of change”, and communicate this refined knowledge to others (Davies R 2004:102)

27. WHAT DOES A LOGIC MODEL LOOK LIKE? • Graphic display of boxes and arrows; vertical or horizontal • Relationships, linkages • Any shape possible • Circular, dynamic • Cultural adaptations; storyboards • Level of detail • Simple • Complex • Multiple models • Multi-level programs • Multi-component programs 27 University of Wisconsin-Extension, Program Development and Evaluation

28. Pipeline logic models Source: United Way. (1996). Measuring program outcomes: A practical approach. http://www.odesinc.org/Measuring_Program_Outcomes-UW.pdf

29. “Families” of models or “nested” models Multiple models may be needed to describe and explain complex systems or initiatives. Bring coherence across an organization • Multi-level: a way to describe and link activities across an organization to depict varying levels such as national-provincial-district levels OR, institution-division-unit levels. • Multi-component programs: A series of models to depict various components (goals, sites, target populations) within a comprehensive initiative. University of Wisconsin-Extension, Program Development and Evaluation 29

30. Multi level - Logic models can be linked to display consistency of purpose and strategy across levels and show how parts work to achieve organizational goals MISSION Each logic model is built with reference to the levels above and below, and in relation to the organization’s or program’s mission. Macro level – Government Department Level Program level 30 University of Wisconsin-Extension, Program Development and Evaluation

31. Multi-component – a way to describe and link different activities within a comprehensive initiative 31 University of Wisconsin-Extension, Program Development and Evaluation

32. Notes: On goals and objectives (1) • These are statements capturing “intent” and “purpose” on the part of the project team • Goals (or aims) are best formulated to refer to the overarching purpose of the whole intervention • Goals are invariably more comprehensive than objectives and therefore often contain more “abstractions” or “constructs”. • Goal and objective statements are best formulated as “To ….. “ statements

33. Notes: On goals and objectives (2) Goal and objective statements must in the final analysis refer to some intended change/benefit/effect/outcome in or with regard to the target group, e.g. Obj 1 = To improve the educators’ understanding NOT – To run workshops Obj 2 = To assist emerging researchers in writing scientific reports NOT – To run scientific writing workshops In both these cases the correct formulation indicates what effect (change/ benefit) with regard to the target group is aimed at. The wrong formulation is a statement (of intent/purpose) about the means to achieve the desired effect. In the Logic Model the latter will be included in the Activities Column (what the programme staff have to DO to realize the stated objectives)

34. Notes: On activities and outputs (1) • The activities column refer to those actions that the programme staff need to take to realize the stated objectives. These actions are often multi-facetted which means that the more precise and concrete the better. • Activity statements are ideally formulated in the present participle tense: Designing (courses/curricula)/ Developing materials/ Organising and presenting workshops/ Recruiting, selecting and appointing staff/ Funding visiting fellows. • Look out for inherent conditionalities and path-dependencies when writing the activity statements. Number these statements in the sequence that these activities is most likely going to occur, e.g. deciding on selection criteria, advertising, recruiting, selecting, appointing, etc…

35. Notes: On outputs and outcomes Perspective continued: OUTCOMES are best understood from the perspective or viewpoint of the target group/target institution/organisation Outcomes refer to the effects of an intervention ON the target group – how they benefit from the intervention Outcomes invariably imply CHANGE (positive or negative): more knowledgeable scholars/ more skilful learners/ change in attitudes or values or beliefs/ increased awareness/ etc. This means that one can (from the project team’s perspective) produce all the promised deliverables (training sessions, manuals, etc.) without this being translated into some benefit to the target group.

36. Notes: Indicators and sources of evidence • Indicators are measures or signs of project impacts, outcomes, outputs, and inputs (for measuring improvement or success) • Indicators organise information in a way that clarifies the relationships between a project's impacts, outcomes, outputs, and inputs. Indicators must be • Understandable – an indicator must be simple and easy for everyone to understand. • Reliable – people must trust the information that an indicator provides. • Measurable – it must be a measurable sign (even if only in an index) • Relevant – an indicator must fit the purpose you have it for – help measure progress toward a goal, raise awareness about a critical issue, or help local decision-making regarding natural resource use, etc.

37. GET AHEAD PROGRAMME THEORY If Get Ahead improves the content knowledge and teaching methodologies of the GET educators in mathematics and science through the workshops, classroom support and additional materials THEN we should get an improvement in the quality of teaching in the classrooms which will lead to improved learner performance in science and mathematics.

38. Target group = Maths educators in primary schools GET AHEAD LOGIC MODEL Goal = To contribute to the professional development of Maths educators by increasing their conceptual understanding of and teaching skills in Maths