Legal Aid Services of Oklahoma, Inc. Gary W. Dart, Esq. and Denise Caudill, DrPH

1. Outcome Measurement:Assessing Clients’ Perspectives of the Impact of Legal Aid Services in Their Lives Legal Aid Services of Oklahoma, Inc. Gary W. Dart, Esq. and Denise Caudill, DrPH

2. Outcome Measurement Is … The regular, systematic tracking of the extent to which program participants experience the benefits or changes intended. United Way, 2000

3. What are Outcomes? • Benefits or changes for program participants during or after their involvement with a program United Way, 2000 Examples of Outcomes: • Change in knowledge (understand infant care needs) • Change in skills (increased reading level) • Change in attitude (greater self-esteem) • Change in circumstances (foster care to family) • Change in behavior (drop-outs attend school) • Change in status (unemployed to employed) Michael Quinn Patton, Utilization Focused Evaluation, 1997

4. Benefits Of OM % of UW agency executives agree or strongly agree that Outcome Measurement helps them: • Focus staff on shared goals (88%) • Communicate results to stakeholders (88%) • Clarify program purpose (86%) • Identify effective practices (86%) • Compete for resources (83%) • Enhance record keeping (80%) • Improve service delivery (76%)

5. You’ve got to be careful if you don’t know where you’re going ‘cause you might not get there. Yogi Berra

6. I. Get Ready • Assemble and orient an OM work group Director of Litigation, Managing Attorneys of largest two law offices in state, Resource Development Specialist; Managing Attorney of the Hotline, Information System Specialist, Community Ed and Pro Se Coordinator Support Staff 2. Decide which program(s) to start with Whose outcomes to measure -- all or some clients?

7. II. Choose the Outcomes You Want to Measure 1. Gather ideas for what the program’s outcomes are from a variety of sources 2. Select the outcomes that are important to measure 3. Construct a logic model

8. The Logic Model

9. Outcomes from Whose Perspective? Client? Both? Attorney?

10. Legal Aid Services Of Oklahoma - Logic Model Clients have increased security in achieving and protecting their basic needs, such as food, shelter, income, health care, personal safety, and family relationships. Clients obtain access to justice system, civil liberties, equal protection and intended benefit of law. Long-termOUTCOMES Intermediate OUTCOMES Clients act to obtain resolution. Clients feel empowered to make informed decisions (to act or not to act) Initial OUTCOMES How the System Works: Clients gain knowledge of legal system How the Clients Fit Into the Legal System: Clients gain knowledge of their legal rights and responsibilities Clients Learn What To Do or Not Do to Solve Their Legal Problems: Work with an advocate, help themselves through pro se, seek help thru non-legal means (social agencies) or not act at all. OUTPUTS # of Legal Counseling services provided # of Brief services provided # of Referrals to community partners # of Full Representation services provided # of Community Education & Involvement Activities Legal counseling, brief services, self-help materials, legal clinics, community education and involvement, referrals to community partners and full representation. ACTIVITIES INPUTS Financial resources from grants, contracts and donations to provide staff, operational expenses and equipment. Non-monetary resources such as; donated time from volunteers, donated facilities and partnerships from collaborative efforts. Senior citizens; low-income individuals and families; nonprofit groups and others in Oklahoma who apply and / or become clients of LASO

11. III. Specify Indicators 1. Specify one or more indicators for each outcome 2. Decide what factors could influence participant outcomes

17. Influencing Factors Respondents’ mental health: including learned helplessness, locus of control, self confidence, self esteem Respondents’ economic, educational and literacy levels • LASO level of service -- full or partial service: • Received or didn’t receive materials • Priority of case • Wanted a lawyer to represent them but didn’t get one Those things that prevent people from being able to do what needs to be done, e.g. transport, child care, money, rural / urban LASO phone system Respondents’ desire to please during interview and their understanding of questions Selection bias introduced by telephone communication, more stable clients instead of transient, moving

18. IV. Prepare to Collect Data on Your Indicators 1. Identify data sources for indicators 2. Design data collection methods 3. Pretest data collection instruments and procedures

19. How to Identify Cases and Clients for OM? LSC Case Definitions and Closure Categories Tools, Transactions and Representation

20. New Definitions of Service Categories • TOOLS: Giving a client advice, information, self-help (pro se) materials or forms that the client needs to understand possible options and/or to address their legal problems on their own. • TRANSACTIONS: Serving a client with non-advocacy legal work that resolves the client’s problem with the drafting of documents, such as; wills, deeds, contracts, DPOA, incorporation, application for tax benefits, advanced medical directives/DNR. • REPRESENTATION: Advocacy on behalf of a client with or without litigation, such as negotiation with a landlord, appearing in court or administrative hearing.

21. Data Collection Method: Telephone Interviews Tools 1 (Initial survey) All tools cases closed during previous week Transactions All transactions cases closed 5 weeks back Tools 2 (Intermediate survey) All tools cases closed 5 weeks back Representation All representation cases closed 5 weeks back

23. In the Oklahoma City field test, 78 calls were made and 24 interviews were completed, a response rate of 31%. In the Tulsa Hotline office, 26 interviews were attempted and completed. Field testing Process .49

24. Field testing Revisions • Simplify and improve wording of questions • Improve opening statement • Improve format of questionnaires, placement of questions and response devices • Reinforce procedures, i.e., callers should not leave messages on answering machines • Add more information to call sheet options • Process for providing feedback to attorneys if clients indicate need for more information • Identify need to attempt to reach people by phone in the evenings and/or week-ends

25. Field testing Lessons • Significant improvements in the questionnaire • LASO staff “buy-in” to the OM process. In one office, each attorney was asked to interview 10 cases. The staff were surprised by: • the challenge of reaching people during working hours • some of the more negative responses • Staff gained new perspectives in relating to clients. Since the field testing experience, their supervisor has seen positive changes in attitude toward clients among staff.

26. Field testing Results • In general, 55% to 65% of respondents in OKC answered most questions positively; Tulsa Hotline respondents ranged between 60% to 75% positive • Room for improvement in services primarily related to response time and difficulties with the phone system More Results from OKC Test

27. As a result of the help from Legal Aid, are you (and/or your family) better off today? • YES • “2000% better” • “Very much” • “ We have benefits” • “Got help paying family bills” • “Power of attorney” OKC: N = 8 • SOMEWHAT • “Problem still up in the air” • NOT SURE • “Didn’t get a response” • NO • “Didn’t do anything”

28. Did Legal Aid help you keep the problem from getting worse? OKC: N = 7 • YES • “Got me out of a bad situation, on with my life” • “Got benefits needed to live” • “She was there with me during the hearing” • SOMEWHAT • “It’s going to stay the same” • NO • “Didn’t call” • “Didn’t help or seem to know how”

29. Do you feel the help you received from Legal Aid gave you access to the justice system? OKC: N =8 • YES • “Got me into court with a lawyer” • “He knew the route to take” • “I couldn’t have done it myself” • NO • “I didn’t get to talk with anyone” • “I don’t think they understood” NOT SURE “Another lawyer done better than they do”

30. V. Try Out the Outcome Measurement System 1. Develop a trial strategy 2. Prepare the data collectors 3. Track and collect outcome data 4. Monitor the outcome measurement process

31. VI. Analyze and Report Findings 1. Enter the data and check for errors 2. Tabulate the data 3. Analyze data by key characteristics

32. VII. Improve Outcome Measurement System 1. Review trial-run experience, make necessary adjustments 2. Replicate in all offices statewide.

33. VIII. Use The Findings EXTERNAL • Retain & increase funding • Enhance public image • Promote program to partners & clients INTERNAL • Inform staff & board • Improve programs • Guide budgeting & resource allocation • Identify training needs BOTH • Document real differences program is making in people’s lives

34. Outcome Measurement Training • The Tulsa Area United Way provided Capacity Building funds for Outcome Measurement Training services to a select group of its member agencies in 2003. • Legal Aid Services of Oklahoma, Inc. (LASO) was one of 10 agencies to participate in the program. A consultant worked with the LASO team for a period of 110 hours during the year. • The United Way of Central Oklahoma has committed to provide funding to continue this process with the consultant for an additional 75 hours through 2004 • The United Way logic model approach described in the book, Measuring Program Outcomes: A Practical Approach, is the one being used by LASO.

35. Legal Aid Services of Oklahoma Inc. 2915 Classen Blvd Oklahoma City, OK 73106 T: 405-557-0200 F: 405-524-1257 gary.taylor@legalaidok.org gary.dart@legalaidok.org marilyn.staats@legalaidok.org The Outcome Zone™ Training, Consultation, Coaching to Strengthen Organizational Capacity for Success Denise Caudill, DrPH 2200 Dublin Road Oklahoma City, OK 73120 T: 405-755-6258 405-922-2814 F: 405-755-6218 DeniseCaudill@cox.net For more information, contact: