Getting Results – An Overview of the Excellence Through Evaluation Study

1. Getting Results – An Overview of the Excellence Through Evaluation Study Dr. Ronald Mincy Dr. Hillard Pouncy Serena Klempin, MSW 13th Annual International Fatherhood Conference June 10, 2011

2. Introduction to the Study Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

3. Study Purpose • We are conducting a feasibility study of a random assignment evaluation meant to tell the federal Office of Child Support Enforcement whether a demonstration partnership in Baltimore, Maryland between the state’s child support enforcement agency, its welfare agency and local community-based responsible fatherhood organizations could be reliably evaluated in a full-scale random assignment study.

4. Rounsaville, B.J., Carroll, K.M., & Onken, L.S. (2001). NIDA’s stage model of behavioral therapies research: Getting started and moving on from Stage I. Clinical Psychology: Science & Practice, 8, 133-142. Rounsaville, B.J., Carroll, K.M., & Onken, L.S. (2001). NIDA’s stage model of behavioral therapies research: Getting started and moving on from Stage I. Clinical Psychology: Science & Practice, 8, 133-142. Rounsaville, B.J., Carroll, K.M., & Onken, L.S. (2001). NIDA’s stage model of behavioral therapies research: Getting started and moving on from Stage I. Clinical Psychology: Science & Practice, 8, 133-142. Rounsaville, B.J., Carroll, K.M., & Onken, L.S. (2001). NIDA’s stage model of behavioral therapies research: Getting started and moving on from Stage I. Clinical Psychology: Science & Practice, 8, 133-142. Study Design • The Excellence Through Evaluation study is a stage 1b randomized controlled trial. • Stage 1b trials: • Fully develop the elements required to evaluate the impact of a behavioral interventionusing an experimental design. • Demonstrate the feasibility of the study design. • Determine whether or not the therapy possesses sufficient promise to warrant further investigation*. *Rounsaville, B.J., Carroll, K.M., & Onken, L.S. (2001). NIDA’s stage model of behavioral therapies research: Getting started and moving on from Stage I. Clinical Psychology: Science & Practice, 8, 133-142.

5. Study Design • Participants are randomly assigned by Maryland Child Support Enforcement Agency (CSEA) (through its vendor for child support enforcement in Baltimore, Policy Studies Inc., or PSI) either to a responsible fatherhood program operated by the Center for Urban Families (CFUF), or to one of 5 other community-based organizations to which PSI commonly makes referrals.

6. Evaluation Design Non-Compliant NCPs notified of Driver’s Revocation by phone – automatically enrolled in Help Us Help You (HUHY)* At HUHY Orientation Eligible NCPs screened and recruited (details below). yes Did Client Consent to Study Participation? Random Assignment no to treatment group to control group Control Sites CFUF COMPONENTS Intensive case management Child support navigation Curriculum sessions Strive Employment Program Peer support groups PSI adds supplemental data to client file *HUHY is Baltimore City Child Support Agency’s employment assistance program for non-custodial parents who cannot pay their child support.

7. Eligibility Criteria • In order to be eligible for the study, a NCP must: • Have voluntarily visited Baltimore Child Support Enforcement (PSI) to register for HUHY after having received a call informing them that their driver’s license may be revoked due to non-payment of child support. • Be a noncustodial father with a non-resident child between the ages of birth – 13 years who is subject to a Baltimore City child support order. • Be fluent in English. • Be currently able to work, and thus not requiring specialized services. • Be 18 or over. • In addition, potential participants may be ineligible if any of the following criteria are met: • Non-custodial parent is a female. • Non-custodial parent has an outstanding warrant. • Non-custodial parent enrolled in a career center or fatherhood program prior to attending orientation.

8. Treatment Site CFUF’s Baltimore Responsible Fatherhood Program (BRFP) Client Flow

9. Control Sites • There are 5 control sites, all located within Baltimore City. • All control sites offer employment placement services. • The control sites have differing levels of support for non-custodial fathers. • None of the control sites offer the intensity of child support case management that CFUF does.

10. Study Goals • Ultimately, the goal of the study is to provide information about the following outcomes: • Whether or not there are differential effects on employment and child support outcomes for CFUF. • The effectiveness of child support intermediation activities coordinated with CSEA.

11. Study Data Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

12. Demographic Data

13. Child Support Data *As indicated by automatic wage withholding

14. TANF Data

15. Attendance Data • Site assignment and program attendance are being monitored by PSI. • PSI routinely monitors program attendance for all of the clients whom they refer to employment programs during bi-weekly follow-up meetings.

16. Data Transmission Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

17. Intake Data

18. Child Support Data: Temporary System

19. Child Support and TANF Data: Permanent System

20. Attendance Data

21. Current Study Status Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

22. Recruitment Total

23. Data Received • Columbia has received all of the intake data for February – April. • Columbia has received 2 months worth of child support data that was manually collected by PSI. • Columbia will not receive any TANF data until the automated system becomes functional in late May / early June. • Columbia has received all of the program attendance data for February – April.

24. Data Analysis • Data is currently being entered and combined into a master spreadsheet at Columbia. • No analysis has been done yet.

25. Challenges Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

26. Recruitment • Clients were initially recruited one-on-one. Using this strategy, 1 out of every 2-3 eligible clients agreed to participate. • PSI altered their internal system for enrolling clients in HUHY, thereby increasing their client flow. • To handle the increased client flow, PSI experimented with a group study recruitment strategy. • The group recruitment strategy yielded 40 participants out of 42 eligible clients. • The group strategy is currently being reviewed by Columbia’s Institutional Review Board (IRB).

27. Retention • Participants are not returning to PSI to document their program attendance. • Participants stop attending programs once they find a job. • Participants are attending programs other than the one to which they were assigned.

28. Technical Delays • We underestimated the amount of time and effort that would be needed to create the automated system for generating child support and TANF data. • We grossly underestimated time required for IRB approval and partnership development.

29. Next Steps Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families

30. Next Steps • Finalize recruitment strategy. • Transition to automated data collection system. • Analyze study data. • Sort out contamination • Conduct power analysis to estimate effect size in order to determine the sample size that would be needed for an effectiveness evaluation. • Apply for additional grants to continue the current study and implement an effectiveness evaluation.

31. Questions? Excellence through Evaluation: Assessing, Addressing, and Achieving – An Enhanced Partnership to Strengthen Maryland’s Families