SWK 7401. Evaluating Social Work Practice

1. SWK 7401. Evaluating Social Work Practice Alvin Mares, PhD, LSW Autumn 2012

2. Week 1 (Aug 22nd & 24th) • Introductions • Instructor • Students and “memorable client” from 1styr field placement • Organization of Textbook • Overview (Chpts. 1-2) • Measurement (Chpts. 3-10) • Evaluation Designs & Decision Making (Chpts. 11-15) • Review of Syllabus • Assignments • Schedule

3. Introductions • Instructor • Practice, program & policy-level interventions for at-risk, TAY (18-29) to prevent chronic homelessness • Emphasis on post-secondary education & vocational training • Students • Name • BSSW/BSW Field Placement (OR) Work/Volunteer Exp. • Memorable client • First name • Age • Gender • Presenting problem • Assistance provided by you (i.e., intervention) • Outcome/disposition

4. Wednesday Evening Section

5. Wednesday Evening Section (con’t)

6. Friday Morning Section

7. Organization of Textbook

8. Review of Syllabus Assignments • Attendance ………. 15 pts. • 1 pt./class • Discussion ?’s …… 15 pts. • 1 pt./chpt. • Download from Carmen • Quizzes …………….. 40 pts. • 20 pts. each • Drawn from Discussion ?’s • Evaluation plan … 30 pts. • Rubric forthcoming Total ………………. 100 pts. Schedule • Wk. 1: Intro/overview • Wk. 2: Chpts. 1 & 2 • Wk. 3: Chpts. 3 & 4 • Wk. 4: Chpt. 5 • Wk. 5: Chpt. 6 • Wk. 6: Chpts. 7 & 8 (Quiz 1) • Wk. 7: Chpts. 9 & 10 • Wk. 8: Chpt. 11 • Wk. 9: Chpts. 12 & 13 • Wk. 10: Chpts. 14 & 15 (Quiz 2) • Wks. 11-13: Evaluation plan • Wk. 14: Off • Wk. 15: Wrap-up

9. Homework Due Aug. 29th & 31st • Read Chapters 1 & 2 • Answer Discussion Questions for Chpts. 1 & 2 downloaded from Carmen • Highlight & type answers to questions in MS Word • Bold correct answer for T/F and MC questions • Type answers for essay questions • Brief answers (3-5 sentences) sufficient for most essay questions • As much as possible, apply concepts to your work last year with your memorable client when answering essay questions • Bring printed copy of questions with answers to class to share your thoughts with others and turn in at end of class; pass/fail grading (0 pts. if late or incomplete)

10. Week 2 (Aug 29th & 31st) • Chapter 1 Introduction to Client-Centered Evaluation of Practice (C-CEP) • 8 steps/components • Case study (Phillip) • Graph (Fig. 1.1, p.8) • Chapter 2 Conceptualization: Naming What We See in the Client Situation • Concepts • Propositions • Theories: General vs. Local • Intervention Design, C-CEP & Logic Model Elements • Case study (Ben & Katherine)

11. CHAPTER 1:Introduction to Client-Centered Evaluation of Practice

12. Chapter 1 Introduction to Client-Centered Evaluation of Practice • C-CEP framework: 8 steps in evaluating practice • Case study example of Phillip, 10 y/o boy with diabetes who is feeling depressed 1. Client defines goals • “feel pretty good most of the time” [positive self-image] • feeling ladder from 1 (worst) to 5 (best); 3=ok 2. Select intermediary objectives and their specific targets • objective: weight loss • targets (empirical proxies) • exercise: riding bike around block 3 times • healthy diet: snack on carots & fruit

13. Chapter 1 Introduction to Client-Centered Evaluation of Practice • C-CEP framework (con’t) 3. ID evidence-based general practice from literature + evaluation-informed specific practice from work with client • review of literature to find promising or best practices • tailoring/application of general best practices to client’s situation • feedback from client over time on effectiveness of practice/intervention 4. Collect and plot data on graphs • Y-axis: goals, objectives, targets (e.g., 1-5 feeling about self) • X-axis: time (e.g., days) • pattern of targeted events (improvement, decline, no change) during various periods (baseline (A), intervention (B), maintenance (M), follow-up (F))

14. Chapter 1 Introduction to Client-Centered Evaluation of Practice • C-CEP framework (con’t) 5. Select benchmarks defining “success” • client-designated (e.g., “4=positive most of the time”) and/or • norms from standardized scales, agency/admin standards, etc. 6. Review and discuss with client at end of intervention phase (B) extent to which goals attained, helpfulness of intervention 7. Transition to maintenance phase (M) • Client assumes responsibility for ongoing intervention his/herself; drives solo (rather than w/ you in passenger seat) 8. Analyze empirical data (graphs) & compare with qualitative data (sharing during intervention sessions)

15. C-CEP Framework Summary • Identify the Client Who Defines the Goals in the Case • Select Intermediary Objectives and Their Specific Targets • Identify Evidence-Based General Practice From the Literature and Evaluation-Informed Specific Practice From Working With the Client • Collect Data in an Ongoing Basis and Plot Data on Graphs • Construct Clear Benchmarks to Know When the Results Are (or Are Not) Successful • Engage the Client in Determining If the Client’s Goals Have Been Attained During Intervention • Introduce a Maintenance Phase in Which the Client Is in Complete Control of the Intervention on His or Her Own • Analyze Patterns of Data to Coordinate With the Client's Statements

16. C-CEP Graph

17. Chapter 1 Introduction to Client-Centered Evaluation of Practice 1-1. Single-system designs are used only for individual clients, especially in institutional settings. A. True B. False 1-2. Single-system designs can be used in many different professional settings where clear targets can be defined and clear actions can be taken to attain these targets. A. True B. False 1-3. Evaluation is built into the codes of ethical practice of many helping professions. A. True B. False

18. Chapter 1 Introduction to Client-Centered Evaluation of Practice 1-5. Evidence-based general practice is: A. The same as evaluation-informed specific practice, when used with a given client. B. The same as empirical research, using control groups and experimental groups. C. The truth, and so practitioners ought to use it exactly as discussed in the literature. D. The best available information on some general topic, but it ordinarily cannot be used directly with your specific client.

19. Chapter 1 Introduction to Client-Centered Evaluation of Practice 1-6. When you use evaluation-informed specific practice, it means: A. That you have used have used an evidence-based model, without varying from the protocol or trying to alter the model to fit your specific client. B. That the information obtained in your single-system evaluation specifically fits this particular client, and you can make meaningful statements as to how well that client is doing. 1-7. It is easy to mix up goals and objectives in evaluation. Take a case example from your own practice experience (or make one up), and distinguish goals and objectives in the case.

20. Chapter 1 Introduction to Client-Centered Evaluation of Practice 1-8. Identify at least three basic ingredients for all single-system designs used in evaluation of practice from this list of terms: • a specified target (problem or desired goal) • operational definitions of targets, interventions, benchmarks • establishing a baseline • different phases (baseline, intervention, maintenance, follow-up) • take repeated measures in both baseline and intervention phases • use of an evaluation design, at least the basic design • analyses of data • decision making

21. Chapter 1 Introduction to Client-Centered Evaluation of Practice 1-9. We defined targets as empirical proxies. Explain what this means in your own language, but include in your discussion the idea of operational definitions. (That is, illustrate this phase, “empirical proxies,” by describing a client situation in which such proxies are involved.) 1-10. Why do we assert that you need to collect data on a repeated basis, even at those times when you are not even doing any intervention? 1-11. You are working with a group of young women (ages 15-20 years), all of whom are pregnant with their first child. They have determined, among themselves, to continue their education. Give one example of a benchmark in these situations. (Hint: is it measurable? How can we know whether or not each client has achieved this benchmark?)

22. BREAK

23. CHAPTER 2:Conceptualization: Naming What We See in the Client Situation

24. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Concept • Also known as a construct • An arbitrarily constructed general term derived from some class of events • Serves as a building block for larger conceptual terms (e.g., propositions, theories) and as a link to the information network where you can find evidence-based practice ideas (Britner & Bloom)

25. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Propositions • Sets of concepts related in such ways as to describe: • what we assume to exist, so we can get to the second proposition; • what we hypothesize to exist and which we can test in the empirical world; or • something we prefer or value, for which there is no empirical test.

26. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Theories • Systems of concepts and propositions that focus on, describe, explain, and predict a limited domain of the world. • General theoriesare abstract enough to apply to broad topics (e.g., behaviorism) • Local theories deal with specific situations (for a given client and their context)

27. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Case study: Ben & Katherine (Kat) • Young couple w/o children arguing about… • not having enough money (primary presenting problem) • having children • sex (secondary presenting problem) • religion • “Assessment and Problem Formation” phase of the Social Work Problem-Solving Process (Dr. Bean) • Problem definition = primary presenting problem(s) • A problem well-defined is a problem half-solved

28. The Social Work Problem-Solving Process

29. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Case study: Ben & Katherine (Kat) • Given primary concerns: financial problems & sex • C-CEP goals: financial solvency & sexual satisfaction • Concepts and propositions • Concepts relating to money • work; earnings; recurring bills to pay; other expenses; part-time vs. full-time employment • Propositions • Prolonged imbalanced of earnings compared to expenses leads to psychological stresses that may affect the continued well-being or survival of the group [marital relationship]. • When the stronger partner does not earn enough money to pay the bills, that partner will compensate his or her weakened situation by becoming more demanding in other domains (e.g., sex), which may be unwelcomed by the other partner.

30. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Case study: Ben & Katherine (Kat) • Theories • Local theory • Therefore, only when former (economic) difficulty is resolved will the latter (sexual conflict, etc.) be resolved. • General theory(ies) • Family Life Model (establishment)? • Family Life Cycle for families (new couple)? • Erickson’s developmental stages (industry vs. inferiority)? • Levinson’s developmental phases (early adulthood)?

31. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Case study: Ben & Katherine (Kat) • Action plan based on… • Presenting problems of financial troubles (primary) & sex (secondary) • Goals of financial solvency & sexual satisfaction • Local & general theories • General theory(ies) • Ben to find higher paying job to increase HH income • Fig. 2.1: Graph of job status (old vs. new) • Fig. 2.2: Graph of HH income ($ range per week) • Kat to extend educational career

32. Chapter 2 Conceptualization: Naming What We See in the Client Situation • Case study: Ben & Katherine (Kat) • C-CEP and Logic Model Elements • Intermediate Outcomes • Long-term Outcome • Condition • Objective 2 • Goal • Objective 1 • Problem • Financial insolvency • New job • HH income >300/wk • Decreased psychological stress • Operational measure? • Increased sexual satisfaction • Operational measure? • Increased well-being of marriage relationship • Operational measure? • Targets • Propositions & Theories • Assumptions • Inputs, Activities & Outputs • Intervention-related activities

34. The Social Work Problem-Solving Process

35. Thoughts On Interventions • Remember that good theory will have direct relevance to practice • There will be a translation process where you will using concepts and propositions of the theory to frame your practice model or intervention • Being aware of your ‘theory of change’ and communicating that to your client(s) is an essential component of the helping process • Objectives and targets, guided by local theory • Empirical evidence + theories (informed by review of literature & knowledge of client) are bases of plans of action [and revisions to action plans over time]

36. Chapter 2 Conceptualization: Naming What We See in the Client Situation 2-1. A concept is a general and abstract term, for which there may be many instances in the real world. A. True B. False 2-2. Two concepts combined with a linking term create a proposition. A. True B. False 2-3. Theories are either true or false. A. True B. False

37. Chapter 2 Conceptualization: Naming What We See in the Client Situation 2-4. What if, in the process of evaluating your client, some concept gains local empirical support? Does this make the concept “true”? Explain why or why not. 2-5. Explain in your own terms why propositions can be tested in the real world and be found to be true or false. How can we go from two or more abstract concepts linked together (i.e., a proposition) to generate some factual evidence with regard to the client’s actions? 2-6. Clearly distinguish between local and general theory, illustrated with one of your own cases, if possible. 2-8. How are conceptual definitions related to operational definitions, if at all?

38. “Caring Instructor” Intervention

39. Homework Due Sept 5th & 7th • Read Chapters 3 & 4 • Answer Discussion Questions for Chpts. 3 & 4; print & bring to next class • Provide memorable client brief description (i.e., first name, age, gender, presenting problem, assistance provided by you & outcome/disposition at last contact) • Develop C-CEP and Logic Model Elements (1-pg. diagram/figure) for your memorable case, using Ben & Kat case study as an example • Draw or print memorable client description and C-CEP/Logic Model elements diagram/figure and include with Discussion Question answers for Chpts. 3 & 4

40. Week 3 (Sept 5th & 7th) • Measurement issues (Chpt. 3) • Validity • Reliability • Error • Sustainability • Graphing (Chpt. 4) • Rules • Interpretation

41. CHAPTER 3:The Issues: Measurement Theory; Validity; Reliability; Error Messages; Sustainability

42. Validity • Definition • True, accurate measure of a concept • Example: digital scale is an accurate measure of weight • Types • Face: appears to be accurate measure (“faith”) • Content: comprehensive set of items/questions • Criterion: correlation with gold standard measure or behavior • Construct: measure supports proposition or theory • “Client”: achieved state = desired state • Assessed at end of intervention and maintenance phases

43. Reliability • Definition • Consistency • Example: digital scale also highly reliable measure of weight • Types • Interobserver: among two or more observers • Agreements / (Agreements + Disagreements) = reliability score • Test-retest: same measurement under same circumstances • Alternate forms: different questions measuring same concept • Internal consistency: high correlation among set of questions

44. Error & Sustainability • Measurement Error • Random: natural variability among people • Systematic: consistent bias due to setting, time of day/season, helping professional, etc. • Sustainability • Length of maintenance period “long enough” to be sure that gains realized during intervention period are retained post-intervention

45. Chapter 3 Discussion Questions 3-2. Validity means consistency in reporting what we see in the world A. True B. False 3-3. Reliability means how truthful a measure is in reflecting what is occurring in the world. A. True B. False 3-5. Which of the following is a true statement? A. If a measure is reliable, it will also give valid information. B. If a measure is valid, it will also give reliable information. C. Neither A or B is true.

46. Chapter 3 Discussion Questions 3-9. We have introduced the idea of client validity, meaning that whatever is true in a situation has to be true for the client in that situation, and only the client can tell us whether or not this is so…What are the limits or risks in depending on a client’s assessment of his or her own outcome at any given stage of practice? 3-11. Give an example from your client case, if possible, or make up a client case, in which you illustrate random and systematic errors. 3-12. Why is sustainability in client outcome important in client-centered evaluation of practice?

47. BREAK

48. CHAPTER 4:Graphing: The Basics; Exceptions to the Rules

49. The Rules • Draw a graph with horizontal and vertical axes • Label the graph • x-axis (time intervals); y-axis (target) • Phases, divided by vertical lines • Reconstructed baseline (A) -- if retrospective data used • Baseline A • Intervention B • Maintenance M • Follow-up – as needed, at less regular intervals • Add data points!

50. Exceptions • Follow the rules unless there is a compelling reason not to do so (e.g., missing data). • Justify your reasoning for an exception and document it. • As appropriate, make an annotation on the graph.