WHEN THE RESULTS OF EVALUATION CAN MAKE A DIFFERENCE…

1. WHEN THE RESULTS OF EVALUATION CAN MAKE A DIFFERENCE… Dr. Liljana Rihter, Faculty of Social Work, University of Ljubljana, Slovenia EFTC, June 6th, 2007

2. STRUCTURE OF PRESENTATION What is the understanding of term evaluation? Do we all have the same needs and interests as regard evaluation? Establishing the evaluation system for social protection programmes inSlovenia • Typology of the social protection programmes • Evaluation model • Evaluation criteria • Measuring instruments Proposed scenario of evaluation Conclusions

3. WHAT IS THE UNDERSTANDING OF TERM EVALUATION? • Many definitions of evaluation • In narrow sense (evaluation=estimation): examinations and judgements about accomplishments and effectiveness • In broader sense (evaluation = evaluation research): systematic data collection on potentially broad question sets about programme/project activities, characteristics and effects to be able to judge about programme, to plan future activities. Source: Patton, 1997

4. WHAT IS THE UNDERSTANDING OF TERM EVALUATION? Evaluation is: evaluation OF SOME OBJECT for some (more or less defined) PURPOSE with before known and defined CRITERIA.

5. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? The need to evaluate: • Outside tensions: • EU, UN, WHO,… • Financers of programme (state) • Inside tensions: • Professionals: ethically obliged to do the best and professionally correct work. Money, control of resources, costs, results Control of procedures and processes

6. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? • The goals of financers: • To know which programmes are ‘better/worse’, to allocate the money to the best ones • ‘better’: efficient (comparison of inputs and results), effective(comparison of goals of programme with real outcomes) • Evaluation should enable comparison of outcomes of (rather) similar programmes •  SUMMATIVE EVALUATION (to obtain the judgement on value, purpose, efficiency and effectiveness of programme)

7. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? • The goals of professionals carrying out the programme: • To be informed about procedures and processes in programme • Is concrete procedure implemented in the planned manner; do we perform the work ‘well’? • Evaluation should enable researching procedures and processes, to obtain the information to plan future activities •  FORMATIVE EVALUATION (change the programme)

8. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? • The goals of users of programme: • To receive appropriate services in comparison with their needs • Satisfaction with the services, benefits • Evaluation should enable to monitor the changes • EVALUATION OF GOALS, OUTCOMES (are the goals of users in concordance with programme goals; are they achieved)

9. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? • Problems if evaluation responded only to the needs of one stakeholder: • Financers would not obtain the data on comparison of outcomes of programmes; which render impossible to efficiently plan the politics • For professionals the evaluation would be merely control • The users could think that they are solely ‘means of production’

10. DO WE ALL HAVE THE SAME NEEDS AND INTEREST AS REGARD EVALUATION? • Evaluation should be a response to the needs of as many stakeholders as possible; therefore following is needed: • To create a model, encompassing all main questions • Planning and implementing evaluation in accordance with the scientific criteria of objectivity, reliability and validity. COMPLEX EVALUATION

11. ESTABLISHING THE EVALUATION SYSTEM THE IMPETUS FOR ESTABLISHING THE SYSTEM TO EVALUATE THE IMPLEMENTATION OF SOCIAL PROTECTION PROGRAMMES IN SLOVENIA: - necessary adjustments of systems of social protection in Europe; - Slovenian legislation (enabling the function of an integrated and transparent social protection system) - evaluations have been carried out according to different models, diverse criteria,… THE AIMS OF THE SYSTEM: - to enable a comparison of effects between similar programmes (requested by the Ministry) - to provide the professionals with feedback information to enable them to plan and direct their future work

12. Typology of the social protection programmes REVIEW OF THE SOCIAL PROTECTION PROGRAMMES AND TYPOLOGY OF THE PROGRAMMES (Rode, 2001) momentary (transitory) programme prevention correction (preventive (therapeutic programme) programme) permanent programme TC

13. Typology of the social protection programmes DURATION dimension: - momentary, one-off services - transitional programmes - permanent programmes PREVENTION – CORRECTION dimension - purely informative programmes - combinations of preventive and curative programmes - therapeutic programmes

14. Evaluation model (Yates, 1999) SOURCES: means and personnel needed to carry out programme PROCEDURES: activities and services which contribute to realising the programme’s aims PROCESSES: all changes in the users, reactions, events OUTCOMES: changes in psychological and social statuses caused by the programme The evaluation of LINKAGES between sources, procedures, processes, outcomes sources procedures processes outcomes

15. Evaluation criteria ESTABLISHING THE CRITERIA:necessity of co-operation of all parties involved in the implementation of social protection programmes (users, providers, financier) • The general criteria (to measure the aims from the National Programme of Social Protection): • The quality of life • Strength perspective • Normalisation • Special criteria (encountered by various kinds of social protection programmes) • Functional ability • Therapeutic effects • Specific criteria (formed by individual programmes for self-evaluation)

16. EVALUATION CRITERIA – analysis of criteria in the programmes, co-financedby the Ministry

17. Evaluation criteria

18. Evaluation criteria DEFINTION OF SPECIAL AND SPECIFIC CRITERIA Focus group discussion with: - providers of programmes and services’ users; - direct involvement into establishing the system to evaluate social protection programmes. Purpose: - to obtain insight into concrete aims; to form criteria for measuring quality of work and achievements of aims.

19. Evaluation criteria DEFINTION OF SPECIAL AND SPECIFIC CRITERIA in programmes: therapeutic communities and other programmes which offer accommodation for drug users; counselling and social rehabilitation centres of illegal drug users who need everyday treatment Participants: 4 representatives of 3 different programmes Structure and results of discussion: - services and goals of activities: various, depend on interference into users life (→ separated discussions) • existing methods of self-evaluation; identification of common criteria (→ select one ‘good’ already existing methodology): no minimum agreement on how to proceed

20. Evaluation criteria DEFINTION OF SPECIAL AND SPECIFIC CRITERIA in programmes: therapeutic communities and other programmes which offer accommodation for drug users; counselling and social rehabilitation centres of illegal drug users who need everyday treatment Positive effects of suggestions: saving the time and money (no need to develop new methodology, measuring instruments etc.) Negative effects of suggestions: - professionals from other organizations and programmes might not adopt the ‘forced’ methodology and criteria - users and financiers would not have the right to present their criteria

21. Evaluation criteria DEFINTION OF SPECIAL AND SPECIFIC CRITERIA in programmes: therapeutic communities and other programmes which offer accommodation for drug users; counselling and social rehabilitation centres of illegal drug users who need everyday treatment Necessary steps: • to follow principles of dialogic evaluation • the minimum evaluation criteria should be agreed (each programme can have also its own, additional criteria) Positive experience form other groups of programmes: • agreement on possible criteria was feasible in other programmes, where also next steps have been already done

22. Measuring instruments Prepared on the basis of: - general criteria - criteria set by the focus group discussions. Three questionnaires for measuring general criteria: • Questionnaire for measuring the quality of life (Lancashire Quality of Life Profile) • Questionnaire for measuring the use of normalisation principles • Guidelines for the interview for the use of the strength perspective.

23. THE PROPOSED SCENARIO OF PROGRAMME EVALUATION 1. GENERAL PRINCIPLES • To ensure the highest possible level of objectivity (several evaluators for one programme, comparison of their estimations). • To ensure dialogue between evaluators and the representatives of organisations (to reveal possible reasons for good/bad results). • Two similar programmes are evaluated simultaneously to ensure the comparabilityof evaluation.

24. THE PROPOSED SCENARIO OF PROGRAMME EVALUATION 2. THE SCENARIO OF EVALUATION • Two similar programmes are evaluated by two trained evaluators. • Every evaluator has a collaborator (a service provider). • Each of four participants (two evaluators and the two co-workers) estimates both programmes. • The evaluators discuss the data and estimations, write an evaluation report and present it to both organisations and to the Ministry.

25. PILOT EVALUATION PURPOSE: - to test feasibility of the uniform procedure of the proposed evaluation system - to test measurement instruments

26. PILOT EVALUATION EXAMPLE OF PILOT EVALUATION FROM THE PROGRAMME Institutional Care for the Aged Pilot evaluation proceeded as follows: Introductory meeting (October 2004): - purpose of evaluation, evaluation procedure and method - review of proposed measurement instruments, critical assessment of the feasibility of questionnaire - arrangements about data collection Data collection for users upon admission (October 2004 – January 2005) Second joint meeting (January 2005): - report on data collection procedures, problems - the way of collecting the data with the questionnaire for users upon the evaluation - presentation of the collection of data on resources, procedures, processes and other criteria Two visits in each home (March – May 2005) - review of the documents - interview with the representative of the home - survey of the users - data collection and fill in customised form of Yates (Yates, 1999) Data processing (January – May 2005) and preparation of report Report sent to all four evaluators (beginning of June 2005) Harmonisation meeting (end of June 2005) and final report

27. CONCLUSION • Evaluation scenario is feasible in practice • Problems: measurement instruments, data collection • Constant review of criteria is necessary • More involvement of users • Importance of giving value not only to efficiency but also to effectiveness

28. LITERATURE AND SOURCES D.E. Chambers, K.R. Wedel,M.K. Rodwell (1992), Evaluating Social Programs. Massachusetts: Allyn and Bacon J. Cheetam, R. Fuller, G. McIvor, A. Petch (1992), Evaluating Social Work Effeciveness. Buckingham: Open University Press. L.L. Martin, P.M. Kettner (1996), Measuring the Performance of Human Service Program. London: Sage Publications. M.Q. Patton (1997), Utilization-Focused Evaluation. London: Sage Publications. L. Rihter (2004), Evalvacije na področju socialnega varstva in njihov pomen za prilagajanje sodobnih držav blaginje na izzive globalizacije. Univerza v Ljubljani: Fakulteta za družbene vede (doktorska disertacija). N. Rode, L. Rihter, B. Kobal (2006), Evalvacija programov v socialnem varstvu: model in postopek izvedbe. Ljubljana: Fakulteta za socialno delo, Inštitut RS za socialno varstvo. M. J. Smith (1990), Program Evaluation in the Human Services. New York: Springer Publishing Company. B.T. Yates (1996), Analysing Costs, Procedures, Processes and Outcomes in Human Services. London, New Delhi, Thousand Oaks: Sage Publications.