Relational evaluation

1. Workshop 2006, Copenhagen, September 27-29 Relational evaluation One more turn to evaluation and evidence Juha Koivisto, Stakes, FinSoc juha.koivisto@stakes.fi The Inter-Centre Network for the Evaluation of social Work Practice

2. Workshop 2006, Copenhagen, September 27-29 Background(Nutley ym. 2002; Solesbury 2001) • the evidence-based practice (EBP) agenda has taken root at the turn of the millenium in the key sector areas of the public sector, such as health care, education, social care and the criminal justice system • different discourses concerning the role of evidence/knowledge in practices: → evidence-based practice → evidence-informed practice → evidence-aware practice → knowledge-based practice

3. Workshop 2006, Copenhagen, September 27-29 • the main issue in the EBP agenda concerns ”effectiveness” • in social sciences there is no analytical consensus of what effectiveness is, how the effects of social work or interventions arise or how the effectiveness of interventions should be studied; the discussion tackles among other things • ontological questions; the relation of intervention and context • the question of causality • the ’right’ sources of knowledge; researchers, practitioners, end-users of services, etc. • research methodology; quantitative versus qualitative • transferability of work and intervention methods

4. Workshop 2006, Copenhagen, September 27-29 Three main research linesRossi & Freeman 1993; Patton 1997; Pawson & Tilley 1997; Robson 2001) • in social sciences three main research lines of effectiveness can be identified on the basis of evaluation literature • Positivistic, experimental research • Social constructivism • Realistic evaluation

5. Workshop 2006, Copenhagen, September 27-29 1) Positivistic, experimental research • research produces objective, universal and cumulative knowledge of the effects of interventions, but only if the contextual factors do not disturb the knowledge production • an intervention method has an inner causal power; it is a necessary and sufficient element to produce the pursued effects • ’goodness’ or ’badness’ is an inner feature of an intervention method • the best methods can be identified by randomized controlled trials (RCT); quantitative methods • good intervention methdos can be transferred to and applied where ever there are subjects with similar problems

6. Workshop 2006, Copenhagen, September 27-29 2) Social constructivism • an intervention process and its effects are principally and in the last hand determined, moulded or guided by the social context and factors • an intervention method does not have an inner causal power; the social context, where the intervention takes place, explains the effects of an intervention • qualitative methods • it is possible to transfer an intervention method to a new social context, providng that the social structures of the new context are similar enough to the previous context • the best intervention methods are the ones that best match with the ready–made social context; ’goodness’ or ’quality’ is relative to the social context

7. Workshop 2006, Copenhagen, September 27-29 3) Realistic evaluation • tries to combine the two former research lines • seeks for answers to the question of ”what works, to whom and in which context” • generative view on causality; a method or intervention in itself does not cause effects, but through mechanisms which can be understood as the favor the intervention activities gain in the subjects of the intervention • studies the contexts, mechanisms and outcomes of interventions; quantitative and qualitative methods • explains the effects of an intervention multi-causally by the intervention activities, by the social context where the intervention takes place, and by the mechanisms • realistic evaluation’s view on transferability is a transferable theory meaning that ”this programme theory works in these respects, for these subjects, in these kinds of situations”

8. Workshop 2006, Copenhagen, September 27-29 Critisism • the three former research lines take for granted ’erroneous’ ontological dualisms, such as objective versus subjective, method versus context, and intervention versus context (see Latour 1992) • it is impossible to say where a method ends and where the context starts • before we can better analyse what is happening in a social intervention and how change and effects are produced and achieved, the firmly established dualisms have to be crossed

9. Workshop 2006, Copenhagen, September 27-29 Ontological dualism of effectiveness studies Intervention/ method Social context Realistic evaluation: Explaining the effects multi-causally by the method and context that produce mechanisms Social constructivism: Expaining the effects by the social context Positivism: Explaining the effects by the intervention method

10. Workshop 2006, Copenhagen, September 27-29 One more turn: Relational evaluation • relational evaluation (RE) is based on the ideas of the new science and technology studies, especially on actor-network theory (Latour 1987; Callon 1991; Law 1994), that study the simultaneous production of technical and social, the co-production of technical and context (see Harbers (ed.) 2005) • RE attempts to cross dualisms, such as method versus context or objective versus subjective • RE studies the simultaneous production of intervention and context

11. Workshop 2006, Copenhagen, September 27-29 Relational ontology • RE bases on relational ontology according to which the social world is studied as materially heterogeneous networks (sociomaterial, sociotechnical) • the networks are constituted by humans, technical artefacts, money, architecture, values, goals, norms, etc. • human action is mediated by non-human elements • the networks are relational effects which the mediated human action continuously perform, produce and reproduce • nothing exists in itself ”out there” as fixed, every entity is constituted in the networks, the entities constitute each other, they are relations

12. Workshop 2006, Copenhagen, September 27-29 • social structures and orders are sociomaterial networks, continuous effects; we take for granted in our action materially heterogeneous elements, such as artefacts, norms, tools, and tacit conventions (the mediators), which have been earlier black-boxed, thus, social structures and orders are not created in a vacuum • facts, technologies or services are not universally true and therefore applicable where ever; they are cosntituted and they function in the sociomaterial networks, a network has to be created for them, similar than their original network, that they function; ’goodness’ or ’quality’ is relational to the network • to transfer a technical artefact or an intervention method means that the artefact or method and its sociomaterial network, the whole hybrid, has to be built in a new place

13. Workshop 2006, Copenhagen, September 27-29 • thus, organizations, work communities, families, life situations etc. can be studied as sociomaterial networks which are continuosly performed and constituted by the human and non-human elements • they are typically produced daily in the same way, but they change more slowly or faster depending on the changes in the elements of the networks

14. Workshop 2006, Copenhagen, September 27-29 What is social work and intervention? • social work and intervention is local activity and interaction of different actors, such as social worker, other professsionals, client, family members or friends, who use any kind of resources (mediators) in their activity; the goals, interests and problems of the client are negotiated and defined in the interactions • the overall goal of the intervention is usually to change the life of the client somehow, to change the sociomaterial networks of the client; this means that the client should learn and start to perfrorm his/her life activities in some other way and maybe mediated by some new resources

15. Workshop 2006, Copenhagen, September 27-29 → life situation of a client is not a context that is around the client, its rather a continuous effect that the client and other actors perform, produce and reproduce in their daily activities • social work and intervention is usually based on and organized according to some method • a method is a hybrid which is contituted by the human and non-human elements; it is not real until the implementation, before that it is only a ”script” (see Akrich 1992) for example in a method book that defines and implies what kind of actors and elements there are in the intervention and what kind of processes and activities the actors have to perform • the actors perform and produce the method real in the concrete and mediated intervention activities, thus it is not something that the actors just use

16. Workshop 2006, Copenhagen, September 27-29 What does RE study? • RE of social work and interventions studies and describes how the sociomaterial networks of a client are changed by the intervention activities; it studies how the different actors produce and achieve change by their collective action in which they use any kind of mediating resoruces • it is an empirical question as to what kind of role the different actors, elements and activities have in an intervention process that produce pursued effects • the sociomaterial networks are followed before, during and after the development activities

17. Workshop 2006, Copenhagen, September 27-29 • the continuous production of social structures and orders (networks) as well as the intervention activities that attempt to change the way they are produced can be analysed empirically by combining different research methods • the study methods are not decided a priori, the study design is rather build individually in every study • to transfere a work method, the whole sociomaterial hynrid, is never fully possible and therefore ”the intervention” is never fully the same • what is actually transferable is a script that defines the actors. elements and activities of an intervention, the sociomateriality of an intervention

18. Workshop 2006, Copenhagen, September 27-29 What is evidence? • evidence is ’knowledge’ which tells by which kind of necessary actors, resources and activities change and pursued effects can be produced and achieved • this kind of knowledge analyses the sociomateriality of social work and interventions; it can be produced by different study methods and by different actors; it is always imperfect

19. Workshop 2006, Copenhagen, September 27-29 Bibliography • Callon, Michel (1991) Techno-economic Networks and Irreversibility. In John Law (ed.) Sociology of Monsters: Essays on Power, Technology and Domination. Routledge, London, 132-161. • Harbers, Hans (ed.) (2005) Inside the Politics of Technology: Agency and Normativity in the Co-Production of Technology and Society. Amsterdam University Press, NBN International Ltd, and University of Chicago Press. • Latour, Bruno (1987) Science in Action. How to follow scientists and engineers through the society. Harvard University Press, Cambridge, MA. • Latour, Bruno (1992) One more turn after social turn. In Ernan McMullin (ed.) The Social Dimensions of Science. University of Notre Dame Press, Indiana, 272-294. • Law, John (1994) Organizing Modernity. MA, Blackwell, Oxford and Cambridge. • Nutley, Sandra, Davies, Huw & Walter, Isabel 2002. Evidence Based Policy and Practice: Cross Sector Lessons From UK. Research Unit for Research Utilization, Department of Management, University of St Andrews. ESRC UK Centre for Evidence Based Policy and Practice: Working Paper 9. http://www.evidencenetwork.org/Documents/wp9b.pdf • Patton, Michael Quinn (1997) Utilization Focused Evaluation. The New Century Text. Sage, Thousand Oaks, London and New Delhi. • Pawson, Ray & Tilley, Nick 1997. Realistic Evaluation. Sage, London. • Robson, Colin (2001) Käytännön arvioinnin perusteet. Opas evaluaation tekijöille ja tilaajille. Tammi, Helsinki. • Rossi, Peter H & Freeman, Howard, E (1993) Evaluation: A systematic approach. Sage, Newbury Park, London and New Delhi. • Solesbury, William 2001. Evidence Based Policy: Whence it Came and Where it's Going. ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary, University of London. ESRC UK Centre for Evidence Based Policy and Practice: Working Paper 1.