Pro Bono Publico : Avoiding Child Welfare Data Abuse Panel on Meaningful Measurement in the Context of Litigation

1. Pro Bono Publico: Avoiding Child Welfare Data Abuse Panel on Meaningful Measurement in the Context of Litigation and Consent Decrees Daniel Webster, MSW, PhD Center for Social Services Research, University of California at Berkeley Isabel Blanco Deputy State Child Welfare Director, South Carolina Andy Barclay Fostering Court Improvement, Atlanta, GA Susan Smith, PhD Casey Family Programs Data Advocacy Group Presentation Originally Created by: Barbara Needell, MSW, PhD Melissa Correia, MSW Emily Putnam-Hornstein, MSW, PhD Bryn King, MSW AAPWA Preconference Symposium Demonstrating Sustained Well Being of Children and Youth in a Compliance Environment San Diego, CA September 9, 2012

2. The Current Placement System*(highly simplified) the foster care system a bunch of stuff happens CHILD IN CHILD OUT *adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference

3. Tracking Child Welfare Outcomes rate of allegations/ substantiated allegations home-based services vs. out of home care reentry to care permanency through reunification, adoption, or guardianship counterbalanced indicators of system performance use of least restrictive form of care length of stay positive attachments to family, friends, and neighbors stability of care Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill:  Jordan Institute for Families

4. There are three kinds of lies: Lies, Damned Lies and Statistics ^ Abused Statistics

5. PROS: greater performance accountability community awareness and involvement, encourages public-private partnerships ability to track improvement over time, identify areas where programmatic adjustments are needed Region/region and region/county collaboration transparency Public Data: Putting it All Out There

6. CONS: Potential for misuse, misinterpretation, and misrepresentation Available to those with agendas or looking to create a sensational headline Misunderstood data can lead to the wrong policy decisions “Torture numbers, and they’ll confess to anything” (Gregg Easterbrook) Public Data: Putting it All Out There

7. Six Ways to Abuse Data (without actually lying!): • Compare Apples and Oranges • Use ‘snapshots’ of Small Samples • Rely on Unrepresentative Findings • Logically ‘flip’ Statistics • Falsely Assume an Association to be Causal • Rely on Summary Statistics

8. 1) Compare Apples and Oranges Two doctors in San Diego, CA… Doctor #1Doctor #2 What if Doctor #1 is a podiatrist and Doctor #2 is a cardiologist? Lowest mortality rate? 20/1000 2/1000

9. Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 2) Data Snapshots… Crime in San Diego, CA… No change. Average = 73.5 Crime jumped by 49%!! Crime dropped by 16%

10. 2) Data Snapshots… A politician recently claimed that 92.3% of all the jobs lost since Obama took office were lost by women.* *http://www.forbes.com/sites/susanadams/2012/04/12/romney-claims-on-womens-job-loss-paint-a-misleading-picture/

11. 3) Unrepresentative Findings… Survey of people in San Diego, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders?

12. Headline in U-T San Diego: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes? 4) Logical “Flipping”…

13. A study of San Diego residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Finding an association between two factors does not mean that one causes the other… 5) False Causality… X Height Hair Length Gender

14. The average human has one breast and one testicle.* Disaggregation is required for this analysis to be really useful. * ~Des McHale www.quotegarden.com/statistics.htm 6) Reliance on Summary Statistics

15. Disaggregation • One of the most powerful ways to work with data… • Disaggregation involves dismantling or separating out groups within a population to better understand the dynamics • Useful for identifying critical issues that were previously undetected Aggregate Permanency Outcomes Race/Ethnicity County Age Placement Type

16. In San Diego County, entry rates are highest for infants. Entries are higher for African American children for almost all ages.

17. must have the will to weather the storm(s)… continued efforts to frame the data, educate interested media, policymakers, and others what do these findings mean? how can these data be used to gain insight into where improvements are needed? agencies must be proactive in discussing both the “good” and the “bad” (be first, but be right). be transparent if not playing offense…playing defense Response to Data Maltreatment?

18. Daniel Webster 510.290.6779 dwebster@berkeley.edu http://cssr.berkeley.edu/ucb_childwelfare/ Needell, B., Webster, D., Armijo, M., Lee, S., Dawson, W., Magruder, J., Exel, M., Cuccaro-Alamin, S., Putnam-Hornstein, E., Williams, D., Simon, V., Hamilton, D., Lou, C., Peng, C., Moore, M., King, B., Henry, C., & Nuttbrock, A. (2012). Child Welfare Services Reports for California. Retrieved 8/22/2012, from University of California at Berkeley Center for Social Services Research website. URL: <http://cssr.berkeley.edu/ucb_childwelfare> Questions?