New Ways to Evaluate Counselling Effectiveness

1. New Ways to Evaluate Counselling Effectiveness Bryan Hiebert Adjunct Professor Department of Educational Psychology & Leadership Studies University of Victoria Professor Emeritus in Education University of Calgary

2. What is counsellingall about? Counselling or Counselling Psychology Clinical Psychology Social Work etc. • Write down the words or phrases that come to your mind • 1 minute

3. Overview Background An alternate to RTCs A framework for demonstrating value Alternate evidence gathering tools Purpose An alternate way of looking at demonstrating the value of human services Intended to stimulate discussion

4. Evaluation is important • But counsellors do not evaluate their work with clients • 1993 study • 40% never evaluated their work with clients • 35% evaluated with client during interview • 2005 study • 33% did not answer • 56% reported data • Client flow, counsellor time use, etc.

5. How to address the problem We need an approach that is: • Comprehensive enough to include what is needed • Simple enough for people to use • Incorporates evaluation into standard practice • Your own practice becomes your data source for predicting client outcomes • Each client is a n = 1 experiment • Across time and across clients, counsellors acquire ability to make predictions linking interventions & outcomes Local Clinical Scientist (Professional Practitioner)

6. Evidence-Based Outcome-Focused Practice InputProcessOutcome Resources Actions Client change Need to link process with outcome

7. Evidence-Based Outcome-Focused Practice InputProcessOutcome • Indicators of client change • 1. Learning outcomes • Knowledge and skills linked to intervention • 2. Personal attribute outcomes • Changes in attitudes • Intrapersonal variables (self-esteem, motivation, independence) • 3. Impact outcomes • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training • Societal, economic, relational impact

8. Outcomes of Counselling • Client learning outcomes • Knowledge • Skills • Impact on client’s life • Client presenting problem • Economic factors • Third party factors +PrecursorsPersonal Attributes

9. Precursors Intervene between learning outcomes & impact outcomes • Attitude • Motivation • Self-esteem • Stress • Internal locus of control • Belief that change is possible Changes counsellors mentioned That were being observed but not reported

10. Evidence-based Outcome-focused Practice InputProcessOutcome • Activities that link to outputs or deliverables • Generic interventions • Working alliance, microskills, etc. • Specific interventions • 1. Interventions used byservice providers • Skills used by service providers • Home practice completed by students • 2.Programs offered by school • 3. Involvement by 3rd parties 4. Quality of service indicators • Stakeholder satisfaction, including students

11. Evidence-based Outcome-focused Practice InputProcessOutcome • Resources available • 1. Staff • Number of staff, level of training, type of training • 2. Funding • Budget • 3.Service guidelines • Agency mandate 4. Facilities 5. Infrastructure 6. Community resources

12. Outcome Focused Evidence-Based Practice InputProcessOutcome Need to link process with outcome • What will I do? • What are the expected client changes? • What do I expect clients to learn? • What sorts of personal attributes do I want my clients to acquire? • What will be the impact on their lives? • How will I tell?

13. Outcome Focused Evidence-Based Practice • Client • Context • Needs • Goals • Resources available • Context: Structure of opportunity • Staff: Number of staff, level of training, type of training • Funding: Budget • Service guidelines • Facilities • Infrastructure • Community resources • Activities that link to outcomes or deliverables • Generic interventions • Working alliance, microskills, etc. • Specific interventions • Strategies linked to specific client problems(stress, grief, depression, career, etc.) • Client home practice • Other • Programs & Workshops • Facilitation guides • Intervention manuals • External Referral PROCESSES INPUTS OUTCOMES • Indicators of client (Learner) change • Learning outcomes • Changes in knowledge and skills linked to the program or intervention used • Progress Indicators End Result Indicators • Personal attribute outcomes • Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc. • Progress Indicators End Result Indicators • Impact Outcomes • Changes in the client’s life resulting from application of learning

14. Evidence gathering • A Framework for evaluation • Learning from the evaluation experts • Gathering evidence of our impact • Documenting what works and why • Developing evidence gathering tools • Need to use informal evidence Some examples

15. Assessment as Decision Making (vs. Judgement) Please use a two-step process • Would you say that your level of mastery of the attribute under considerations is • Then assign the appropriate rating • 0 = really quite poor • 1 = just about OK, but not quite • 2 = OK, but just barely • 4 = really very good • 3 = in between barely OK and really good unacceptable acceptable 0 1 2 3 4 0 4

16. Problem with skill self-assessment • Participants asked to rate their skill (or knowledge) before and after a program • Often, pre-workshop scores are high and post-workshop scores are lower • People find out as a result of the workshop that they knew less than they thought or had less skill than they thought • Based on the new awareness, post-scores are lower • People don’t know what they don’t know • How can we get around this problem?

17. Assessing Learning & Attribute Outcomes unacceptable acceptable 0 1 2 3 4 Post-Pre Assessment • We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now? • Please use a two-step process: • Decide whether the characteristic in question is acceptable or unacceptable, then • assign the appropriate rating 0 4

18. Applied Career Transitions Program(on-line program for unemployed university grads) For Module 1 • All together there were 10 (items) x 29 (participants) = 290 ratings • Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings • Unacceptable Ratings decreased from 50% to 14% • Pre: 6 Exceptional Ratings – Post:130 Exceptional Ratings • Exceptional Ratings increased from 2 to 44% of the participants

19. Results: Impact outcomes Module 1 • 23 out of 29 had found a job • 10 of the jobs lined up well with career vision Module 2 • 4 out of 6 had found a job • 3 of the jobs lined up well with career vision

20. Post-Pre Assessment: Bridging Health Care with Self-Care • Knowledge of stress and stress control • Knowledge of how to manage personal change • Knowledge of nutrition and nutrition control - - - - - - - - - - - - - - • Level of stress • Level of nutrition (high=healthy) • Level of fitness • Confidence in ability to manage personal change

21. Participant Self-Assessed Change

22. Post-Pre ResultsBridging Health Care with Self-CareStress & Stress Control Start of Program End of Program Vertical axis indicates % of participants

23. Post-Pre ResultsBridging Health Care with Self-CareManage Personal Change Start of Program End of Program Vertical axis indicates % of participants

24. Attribution for Change • To what extent would you say that any changes depicted above were a function of participating in our program, and to what extent were they a function of other factors (e.g., job, personal circumstances, etc.)

25. Participant Comments This program made me realize that change is possible for me to make, and that even a small effort can bring significant benefits. When I joined the program, I thought that only a superhuman effort would result in any positive benefits, and that thought itself was a barrier to making a behavioural change. Now I know that “I can do it” and that knowledge itself is half the battle This is a fantastic program. The biggest benefit for me is that I am starting to see myself as being a more active person, rather than a “couch potato”. I appreciated the focus of the program being on life change and wise choices, rather than on diet and weight loss. I

26. Tracking client goal attainment • Clients are different • Live in different contexts • have different goals • require different interventions • How can we consolidate the results?

27. Goal Attainment Scaling Much better than expected A bit better than expected About what I expected A bit worse than expected Much worse than expected ObservableIndicators +2 +1 0 -1 -2

28. Thought Listing Stimulus question • For Clients: • What are the main factors contributing to your presenting problem? • What needs to happen in order for you to achieve your goals? • What are the barriers you need to overcome? • For counsellors: • What is counselling all about? • What factors are responsible for client change? • Write down all the words or phrases that come to mind in response to this question.

29. Cognitive Mapping Task • Complete the thought listing task • Transfer each item to a “Post-it” sticker, 1 item per sticker. • Arrange the stickers on a large sheet of graph paper • Place related concepts close to each other • Draw connecting lines between stickers that represent related concepts • Draw a circle around any clusters of concepts • label each circle

30. Cognitive Mapping-Example What is counselling all about? Scoring scheme: http://cmap.ihmc.us

31. Relaxation Monitoring Sheet Feelings, body sensations, etc. Thoughts, perceptions, imagery, etc. .

32. Informal Evaluation of Headache, Pain, and Related Affective States

33. Self-Monitoring Headache 0 - No headache 1 - Low level, only enters awareness when you think about it 2 - Aware of headache most of the time, but it can be ignored at times 3 - Painful headache, but still able to continue job 4 - Severe headache, difficult to concentrate with demanding tasks 5 - Intense incapacitating headache

34. Headache Monitoring Grid Level 5 4 3 2 1 0 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05 (time of day) Before treatment After treatment

35. Headache: Indicators of Success

36. Self-Monitoring Stress 0 - No stress 1 - Low level, only enters awareness when you think about it 2 - Aware of stress level most of the time,but still able to stay focused on job 3 - Very stressful, beginning to interfere with what you are doing 4 - Severe stress, creates major interference with what you are doing 5 - The most stress you ever feel

37. Self-Monitoring Confidence 0 -No confidence at all in your ability to handle the situation 1 -Low confidence, but you might get lucky and it will turn out OK 2 - Sort of confident, you think you might be able to handle it, but probably not 3 -Pretty confident, you think you have a chance of this working out OK 4 -Very confident, you’ll probably be able to pull this off 5 -Completely confident about what you are doing

38. Self-Monitoring Job Satisfaction 0 -No satisfaction at all with your job 1 -Somewhat satisfied, there are some things about your job that are OK 2 -Satisfied enough – as jobs go, this one is as good as any 3 -Mostly satisfied, occasionally there are things about your job that you actually like 4 -Very satisfied, but there are occasional things bother you 5 -Completely satisfied with your job, feeling a sense of self-fulfillment

39. Self-Monitoring Motivation 5 - Very motivated-it's the most important thing for me to do today 4 - Very motivated, but something might come up to interfere 3 - Quite a bit motivated, I think I will end up doing some of it 2 - I sort of care and I might get around to doing some work today 1 - If I run out of things to do, I'll try looking in some of my books 0 - I couldn't care less if I ever did my school work

40. Using Self-Talk to Control Anger With cognitively-challenged adults in residential settings Hiebert, B., & Malcolm, D. (1988). Cognitive strategies for mentally handicapped clients. In B. O'Byrne, (Ed.). Natcon-14 (pp. 289-298). Toronto, On: Ontario College Counsellor's Association. (Reprinted in Resources In Education, June 1989, ERIC Document Reproduction Service No. Ed 302800)

41. Picture Stimulus: Pushing

42. Picture Stimulus: Hitting

43. Picture Stimulus: Kicking

44. Interview Protocol for Assessment • I am going to show you 3 pictures and ask you some questions about each picture. • Look at this picture. • Tell me how many people are in the picture. • What are the people doing in the picture? • Point to the person that is being pushed/kicked/hit • If you were the person being pushed/kicked/hit, how would you feel? • If you were the person being pushed/kicked/hit, what would you feel? • If you were the person being pushed/kicked/hit, what would you think? • If you were the person being pushed/kicked/hit, what would you do?

45. Interview Scoring Sheet

46. Field Test Results: Picture Stimuli

47. Frequency of Anger Outbursts 10 Mentally Handicapped Residents

48. Monitoring Your Self-Talk Con Pro                         

49. Changing Your Self-Talk Con Pro                

50. Self-monitoring Stress Subjective Units of Disturbance