Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois

1. Some Practical Tips for Measuring Financial Success Dr. Angela LyonsUniversity of Illinois Evaluating Financial Education in Troubled Times Dr. Lance Palmer University of Georgia Dr. Angela Lyons University of Illinois – Urbana Champaign AFCPE Extension Pre-Conference November 2008

2. Motivation:Recent Financial Challenges • Difficulty paying debts • Low saving rates • Rising home foreclosures • Mounting bankruptcies • Higher unemployment • An increased awareness and impact of natural disasters Consumers are in financial trouble!!!

3. The Result…. • A number of financial education efforts have been developed to address consumers’ financial needs in these troubled times. • 3 key programming areas: • Bankruptcy • Homeownership issues • Natural Disasters

4. At the end of the day, are these financial education efforts making a difference?

5. On the front lines with the evaluation experts…. • “What do we mean by evaluation? • “What are we really trying to measure?” • “How do we define program success?” “How do we know if participants are improving?” “What financial outcomes and indicators should we be using?” “What constitutes a successful, or even acceptable, evaluation?”

6. “The Big Picture” 5 Biggest Evaluation Challenges in General 1. Identifying the “ideal” approach to evaluation. Evaluation methods and measures vary widely across programs and academic disciplines. Wide variation in financial outcomes across programs. Significant differences in financial needs across individuals and families. Some participants unable to implement certain financial behaviors. Shift towards more one-on-one individualized education.

7. 2. Defining “program success.” Setting realistic expectations for program participants. Choosing appropriate outcomes and indicators based on participants’ financial situation or other external constraints. Identifying participants’ individual financial desires and applying appropriate educational interventions and evaluation. Finding “the teachable moment” or motivated learner. Recognizing that “one size does not fit all.”

8. 3. Collecting impact data from program participants. Little incentive to complete evaluations (like “pulling teeth”). Reluctance to divulge personal information (surveys “too personal”; lack of trust). High drop out rates, low response rates, and difficult to track. Literacy levels (i.e., ESL, reading level). Collecting sensitive data and information. Tradeoff between participation and evaluation rigor.

9. 4. Designing and implementing effective program evaluations. Evaluation process is cumbersome. Lack of time, staff, and financial resources. The “PUSH” for increased rigor and “the rush to the finish line.” Rigor vs. Reality (e.g., measurement issues) The limitations of “one-shot” evaluations. (pre- and post-tests; intended vs. actual behavior change) The reality of conducting longitudinal studies with control groups. (follow-ups and tracking of program participants)

10. 5. Data analysis and measurement issues Measurement error and validity of indicators. Self-reports are subject to bias. • Social desirability • Norms and “rules of thumb” • Misperceptions and over-optimism • Memory distortion and recall bias Samples may not be representative. • Non-response bias • Program attrition • Self-selection • Low response rates (e.g., follow-ups)

11. Other measurement issues Environmental factors may affect outcomes. • Unexpected life events • Program incentives (e.g., rewards, special benefits, enrollment programs) • Individualized financial advice or “coaching” Psychological factors. • Inherent motivation • Ability • Attitudes • Stress

12. Bankruptcy Counseling and Education • Educational impacts are difficult to disentangle from the actual bankruptcy process. • Difficult for debtors to change certain behaviors when haven’t yet discharged debts and still under court supervision. • Conducting a follow-up is critical. • Dealing with a very heterogeneous population that is required to complete the counseling and education.

13. Bankruptcy Counseling and Education Cont’d • Content of the counseling and education is similar which can reduce the positive impact of the debtor education. • Selecting indicators that are independent of the debtors’ current financial situation and the bankruptcy discharge. • Constraints associated with curriculum content and timing of delivery.

14. Homeownership programs • Many different types of housing counseling and education programs: • Transitional housing programs • Homeownership education • Post purchase counseling • Default and delinquency counseling • Reverse mortgage counseling • Less emphasis on group education and more focus on one-on-one models of financial counseling or “financial coaching.” • Significant grant dollars are available to conduct these programs, but evaluations can be constrained by funders’ requirements.

15. Post-Natural Disaster Programs • Using traditional pre- and post-test surveys may be less effective than more qualitative methods such as focus groups, interviews, and case studies. • Informational and financial needs may vary widely across disaster victims. • Psychological factors are likely to significantly affect motivation, attitudes, and perceived abilities which will influence evaluation responses. • Disaster victims can be very transient – difficult to track and conduct follow-ups. • Very time and resource intensive.

16. Overcoming the ChallengesPlanning, planning, and more planning…. • Take stock of who you are – What is “your vision?” • Be strategic – What are you trying to accomplish? • Conduct a thorough needs assessment. • Spend time understanding your target audience(s). • Define program success for your audience. Be realistic! • Create an evaluation action plan • Identify resources and overcome constraints. • Take Action!

17. 4 Steps to Creating An Effective “Evaluation Action Plan” Step 1: Define objectives of the program. At the end of the day, what do you want to show? Who will be the target audience? What will be the primary delivery method? Who will use the evaluation and how? Step 2: Select appropriate methods for data collection. What is the most appropriate evaluation format? What types of questions will the evaluation seek to answer? What types of indicators will be used to show impact? Are these indicators appropriate for the target audience?

18. Step 3: Identify and overcome evaluation challenges. What are your biggest implementation challenges? What steps can you take to overcome the challenges? What financial and non-financial resources are available? Are there others who can help? What is the program timeline? Given constraints, what can you realistically do? Step 4: Analyze and disseminate findings. How will data be analyzed? What do you hope to learn from findings? What are the potential impacts? How will the results be used and disseminated?

19. Where do we go from here?Online resources at your fingertips

20. NEFE Financial Education Evaluation Toolkit®http://www2.nefe.org/eval/intro.html

21. NEFE Financial Education Evaluation Toolkit® • Database • Post evaluation only with option for follow-up • Pre and post evaluation with option for follow-up • Stages to Change Evaluation • Train-the-Trainer • Testing Knowledge • Building Skills • Taking Charge • Manual • How-to-guide for grass-roots level organizations • Examples (survey instruments, executive summary, reports) • Guidance on how to organize and present impact data

22. Manual http://www2.nefe.org/eval/manual.html Part I: Financial Education Overview Part II: Understanding Program Evaluation Part III: The Evaluation Planning Process Part IV: Using the Evaluation Database Part V: Reporting Program Impact

23. Appendix: Sample Evaluation Instruments

24. Database http://www2.nefe.org/eval/index.php

25. Knowledge Indicators

26. Behavior Indicators

27. Qualitative Info

28. Demographics

29. University of Wisconsin-Extension http://www.uwex.edu/ces/pdande/evaluation/index.html

30. Cornell University Extension http://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm

31. Penn State Extension http://www.extension.psu.edu/evaluation/

32. Contact Information Dr. Lance Palmer Assistant Professor Department of Housing and Consumer Economics University of Georgia Phone: (706) 542-4916 E-mail: lpalmer@uga.edu

33. Questions