Increasing Engagement in the Evaluation: Incorporating Empowerment Evaluation, Process Evaluation and Process Use in D

1. Increasing Engagement in the Evaluation: Incorporating Empowerment Evaluation, Process Evaluation and Process Use in Design Robin Rosell, LMSW-ACP,LMFT, Director, Tandem Collaboration, People’s Community Clinic Peg Gavin, LCSW, Director of Educational Services, LifeWorks Amy Pierce, MPH, Director, LifeWorks Teen Parent Services Dr. Carol M. Lewis, Associate Director, Center for Social Work Research, University of Texas at Austin School of Social Work

2. Learning Objectives Learn how empowerment evaluation, process evaluation and process use in evaluation design can make the evaluation a mutually beneficial exercise for researchers and agency staff. Learn creative ways to engage staff, collect data, maximize participation, and work toward program improvement. Learn how process evaluation can make use of feedback and short term findings for mid-course corrections and program improvements, helping ensure that goals are met and funder requirements are satisfied.

3. LifeWorks Adolescent Family Life Demonstration Project Five-year, federally funded community partnership in Austin, TX Intensive case management for pregnant and parenting teens provided by LifeWorks Teen Parent Services and Tandem Collaboration of People’s Community Clinic Evaluation led by Center for Social Work Research at UT Austin

4. TANDEM Teen Prenatal & Parenting Program Austin Child Guidance Center Mental Health Assessments & Services, Individual, Couples, Family & Group Therapy

5. Tandem Teen Prenatal & ParentingProgram

6. 1. Promote the health and well-being of participating young parents and their children. 2. Reduce the risk of subsequent unplanned pregnancies. Primary Goals

7. Tandem Program Measures for Success Utilization of health care services by young parents and their children Positive birth outcomes Use of contraceptive method following birth Rate and timing of subsequent pregnancies Utilization of mental health services Presence of positive parenting behaviors Participation in education or other activities promoting self-sufficiency Reduction in high risk behaviors (i.e. substance abuse) Reduction in risk of child abuse Involvement of young fathers and other family members in parenting children

8. Tandem Program Key Strengths

9. LifeWorks Teen Parent Services Services Provided Individualized case management School-based informational presentations School-based support groups Access to a comprehensive, coordinated service delivery system through LifeWorks Information, referral and linkage to community resources

10. LifeWorks Teen Parent Services Primary Goals Prevention of subsequent pregnancies Clients continue with education in order to complete high school and/or GED program.

11. LifeWorks Teen Parent Services Key Strengths

12. Program Evaluation Challenges Engagement of program staff, skepticism regarding evaluation, staff turnover Recruitment and retention of research participants Vulnerable, highly transient program populations Patchwork of funding streams and program partnerships, varied programmatic expectations, simultaneous evaluations Data collection, competition for time, uniform program data

13. Empowerment Evaluation… …the evaluation aims to strengthen the intervention by engaging staff in its implementation (Fetterman, 2005)

14. Effective process evaluation… …makes use of feedback and short term findings for mid-course corrections and program improvements, helping ensure that goals are met and funder requirements are satisfied.

15. Evaluation Capacity Building… “The impact of the evaluation [that] comes from application of evaluation thinking and engaging in evaluation processes (in contrast to impacts that come from using specific findings).” (Patton, 1997) The capacity to build and implement good evaluations.

16. Process use… …ensures that the evaluation is a meaningful exercise for all parties, and the time and expense invested in evaluation reap immediate benefits and transcend the final report (Patton, 2007).

17. Using the evaluation process can result in useful outcomes for both the intervention and the research…

18. Strategies to Balance Reality & Design Directly engage case managers in recruitment and data collection Monthly team meetings between program directors, staff and evaluators Midcourse corrections via interviews and surveys with case managers, clients and families

19. Resulting Process Use Outcomes a dramatic increase in collaboration among the project’s five partnering agencies the development of a unified Outcome Tracking Form that integrated client tracking data required by the evaluation and each agency the strengthening of the intervention’s mental health component

20. Limitations and challenges. . . Biases associated with case managers collecting data from their clients Obstacles to a true baseline related to time constraints, building rapport with clients and parental consent Ethical concerns among program staff regarding randomization and some research recommendations Delays associated with IRB requirements Staff turnover

21. 20/20 Hindsight Expand proposal process to include case managers, individual agencies Larger incentives for client participation from the start More structure and incentives for case managers to recruit and enroll their clients Realistic understanding of what is required to engage in an intensive research project

22. References Fetterman, D. “A Window into the Heart and Soul of Empowerment Evaluation: Looking Through the Lens of Empowerment Evaluation Principals.” In Ed. D. Fetterman and A. Wandersman (eds.), Empowerment Evaluation Principles in Practice. New York: The Gilford Press, 2005. Patton, M. Q. “Process Use as a Usefulism.” In J. B. Cousins (ed.), Process use in Theory, Research and Practice. New directions for evaluation, no. 116, 99-112. San Francisco, Calif.: Wiley, 2007. Patton, M. Q. Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, Calif: Sage, 1997. Weiss C. H. “Have We Learned Anything New About the Use of Evaluation?” American Journal of Evaluation, Mar 1998; 19: 21 - 33.

23. Contact Information Robin Rosell, Tandem Collaboration, People’s Community Clinic Robinr@austinpcc.org Peg Gavin, LifeWorks Educational Services Peg.gavin@lifeworksweb.org Amy Pierce, LifeWorks Teen Parent Services Amy.pierce@lifeowkrsweb.org Dr. Carol M. Lewis, UT Austin Center for Social Work Research Carolmarie@mail.utexas.edu