An Experimental Study of Child Welfare Worker Turnover

1. An Experimental Study of Child Welfare Worker Turnover Nancy S. Dickinson, University of Maryland ndickinson@ssw.umaryland.edu John S. Painter painter.eval@gmail.com National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

2. Child Welfare Staff Recruitment and Retention: An Evidence Based Training Model Study Objectives • Determine the feasibility of using an experimental design to study training outcomes • Understand the impact of worker perceptions on their intent to leave child welfare employment • Study the effectiveness of the intervention on worker retention National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

3. Influences on Recruitment, Selection and Retention • External Environment • Agency’s public image • Awareness of jobs • Worker Characteristics • Desire to help • Self-efficacy • Depersonalization • Education Recruitment • The Work • Role clarity • Role expectations • Workload Selection • Agency Climate • Shared mission • Affirmation & recognition • Shared authority • Growth & advancement • Org commitment • Supervision • Practice support • Emotional support • Team support Retention National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

4. Intervention Components National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

5. Research Questions Do workers in the intervention counties show statistically significant differences from those in the control counties on relevant survey scales? Does child welfare worker retention improve in the intervention counties compared with the control counties? National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

6. Procedures • Random assignment of county child welfare agencies to 17 intervention and 17 control groups • 33 project counties participated in data collection activities (1 agency withdrew after a year) National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

7. General Design Intervention: R O1 X O2 Comparison: R O1 O2 R = random assignment O = data collection (or observation) X = intervention or treatment National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

8. Instruments Online worker survey administered 5 times to all project child welfare workers between 6/1/05 and 6/1/08 Human Resources Database gathered employment information on all project workers between 12/1/04 and 9/1/08 National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

9. Worker Survey • 17 scales validated using reliability analysis and confirmatory factor analysis • Average response rate of 47% (45% - 48%) across 5 waves of delivery to an average sample of 831 workers (731-944) • Waves 1 & 2 were pre-intervention; waves 4 & 5 were post-intervention. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

10. Respondent Demographics National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

11. Demographics, Continued National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

12. Data Analyses • Multi-level regression analysis • Scales are compared pre-post intervention • Survival analysis • Days employed & status at end of study (exit vs. no exit) National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

13. Statistical Comparisons for Survey Scales • Four primary comparisons were made: • Intervention vs. control post-training • Individual level • County level • Pre vs. post training intervention group only • Individual level • County level National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

14. Overview of Survey Results National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

15. Impact of Intervention on Turnover: Data and Sample • HR Database used by all project counties • Internet accessible • Interactive database application • In 9/08, analysis file of 877 workers hired after January 1, 2004 • 485 workers from control counties • 392 from intervention counties National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

16. Worker Demographics for Original and Propensity Matched Samples National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

17. Analysis and Results • Cox regression survival analysis assessed the impact of the intervention on undesirable exits • Effect of the intervention is statistically significant (p<.05) • 27% of control group sample experienced an undesired exit • 17% exit in the intervention group National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

18. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

19. Type of Exit Post Intervention National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

20. Summary A rigorous research methodology can be used to test the effectiveness of a training intervention. Undesired exits by child welfare workers can be slowed significantly because of increased skills and behaviors of supervisors and managers. National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

21. Limitations • Absence of statewide employee database limits quality of data. • Some concern that project database was used inconsistently • Cannot track workers across counties to determine if worker left the profession or the agency National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

22. What Worked Well Recruiting counties thru site visits Random assignment Providing counties with data on turnover Longitudinal design Control group Lots of personal contact with counties HR dbase data proved key Web surveys were very efficient National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

23. Think Twice… Number of counties in study Number of times surveyed Web reports National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

24. Unexpected Challenges Data management! A beast… Some counties were inconsistent in use of HR dbase Inconsistent response to surveys left gaps in data Collecting baseline data before intervention was finalized National Human Services Training Evaluation Symposium Cornell University, June 15, 2011

25. Acknowledgement This study was supported by the U.S. Children’s Bureau (Grant No. 90CT0114) as part of the project Child Welfare Staff Recruitment and Retention: An Evidence-Based Training Model. THANKS National Human Services Training Evaluation Symposium Cornell University, June 15, 2011