Background

1. Strategies for Obtaining the Client-Level Data You Need: One Success Story Lisa Branton, M.S. Highly Accessible Technical Assistance Background Revised Client Consent Form Selected a Data System that was User-Friendly and Useful for End-Users First 5 Sacramento was initiated and funded in 1998 through Proposition 10 passed by the California voters, which imposed a 50 cent tax on tobacco products. The taxes are redistributed to each county Commission to fund agencies to provide services to families with children ages 0-5 years with the overall goal of improving “school readiness” for children. The services funded by First 5 address the comprehensive needs of the family in order to produce children who are ready to enter kindergarten. Some examples of First 5 services include: respite care for children of parents in crisis; parent education classes on parenting practices; one-on-one lactation consultations; cooking/nutrition classes for parents; exercise classes for children; and education for parents and children on transition into kindergarten. Walter R. McDonald and Associates, Inc. is leading the evaluation of First 5 Sacramento services, which began July 1, 2007. This evaluation involves reporting on the characteristics of clients receiving services, the number and types of services provided, and outcomes related to First 5 services received. Agencies contracted to provide First 5 services are asked to recruit clients to participate in the evaluation and subsequently enter the client’s data into the web-based data system. This evaluation also includes a longitudinal study in which we conduct an hour-long interview with a random sample of parents, the Parent Interview. The baseline Parent Interview was conducted with 459 parents. There will be two follow-up Parent Interviews, in the spring 2009 and 2010. This presentation explains the strategies used in this evaluation that contributed to our success in getting client-level data. The First 5 Sacramento population is especially diverse, particularly with migrants. In addition, some clients have a history of drug/alcohol abuse and/or Child Protective Services involvement. The previous evaluators did not obtain a good consent rate which was in part due to the high reading level and intimidating language of the consent form. The poor consent rate contributed to very little client data being entered into the previous data system. Due to these challenges, one of our first steps was to revise the consent form to be less intimidating, easier to read and condensed, while accurately communicating the evaluation procedures and risks to the client. The consent form did not prove to be overly discouraging to clients; the first year of this evaluation had a 96% consent rate. Another key reason for our success in obtaining client-level data for this evaluation is our efforts to ensure that technical assistance is readily accessible for all data system users, especially agency staff entering client data. It is critical that all users have technical assistance available to them in a fairly short time frame (i.e., 1-2 hours ideally, 1 day maximum without forewarning of the break in TA availability). We have been successful in this way because we have one full-time staff dedicated to providing the training and technical assistance for users. She makes a concerted effort to answer users’ questions as quickly as reasonably possible. Without responding to technical assistance needs in a timely manner, one will squelch the desire to enter data and any confidence to ask questions and ensure accurate data are entered. In addition, the Persimmony data system has a wonderful feature for providing technical assistance. This feature is referred to as “shadow” and enables us to remotely control the user’s Persimmony screen and determine the user error in data entry or walk the user through how to complete a task. This has proved to be an especially effective tool for providing technical assistance. Due to challenges with the previous data system, we oversaw a competitive process that included evaluating the various web-based data systems submitted to make a recommendation to the funding agent—the First 5 Sacramento Commission. The recommended and selected data system—Persimmony—has user-friendly data entry screens, accompanied by many pull-down menus to reduce data entry time and error. In addition, it employs one primary screen where users enter client data in order to reduce the likelihood that a user would “get lost” in the data system. This primary screen has three sections for different types of data—demographics, assessments/outcomes, and services. The data system also has a separate module used for submitting quarterly data to First 5 Commission staff for the purposes of contract or grant monitoring. First 5 staff access this module of the Persimmony data system to track the progress of funded agencies. Our most recent upgrades to the data system include a specific place for First 5 staff to record notes or requests to the agency, and the capability for staff to notify agencies of the added notes/requests. The data system also has a module for submitting aggregate level counts of clients by race/ethnicity, language, age of child, and children with special needs. Multiple forms can be set-up to have aggregate data submitted for different purposes, such as children with a substantiated case of abuse or neglect and another form for children with at least one investigation. An additional module provides a wide variety of reports for users. Two reports, in particular, are frequently utilized by our users: a summary service report and the quarterly report (for contract monitoring). Other reports are available for agencies to monitor the quality and aid in cleaning the data. Among others, there are also reports available to readily print mailing labels for the clients entered as well as extract the list of clients by month of birthday to aid in orchestrating a birthday card mailing. Streamlined Family Intake Form Welcomed and Responded to Agency Feedback The intake form intended to accompany the consent form was also redesigned. The primary goal was to make the Family Intake Form more streamlined, limiting the information collected to only the most necessary items. That is, we would need the contact information of families served by First 5 Sacramento (for contacting those selected for the Parent Interview) as well as data on parents and their children ages 0-5 years so that we could describe the population’s demographics, status of health/dental insurance, and special needs. Therefore we also revisited the intake question addressing race/ethnicity to ensure the categories were relevant to the Sacramento population as well as aligned with widely used demographic categories such as the Department of Finance of California. Because Sacramento has a large multi-racial population we also felt it was important to allow clients to record every race/ethnic group to which they identified. This allows us to report the most common multi-racial groups rather than reporting them as just “multi-racial”. • Another important element contributing to our success in obtaining client-level data has been welcoming feedback from agencies, whether in regards to the evaluation overall or the data system specifically. Early in the evaluation we received feedback from agencies that it would be especially helpful to have the consent forms printed in triplicate. While this was an unanticipated expense, the evaluation team agreed that this was a good suggestion that would increase the likelihood that all parties (client, evaluation team, and agency) would get their copy of the consent form as required by our IRB, as well as show our commitment not to overburden agencies. • We also requested the assistance and feedback of two, key, large agencies (service providers) in the pilot testing of the new data system. Their buy-in of the new data system would be critical to obtaining the buy-in of other agencies using the system. Both agencies entered test client-level (demographic, assessment/outcome, service) and aggregate data, their quarterly report data, and provided us with their feedback. • Possibly the most important feedback we welcome, encourage, and respond to is that related to the data system. In doing so, agencies truly begin to view the data system more as their own rather than as someone else’s database they are entering data into. This is quite helpful in obtaining the data needed for evaluation as when feelings of ownership of the data system increases, the desire and commitment to enter evaluation data increases. Users must feel that the data system is useful to them in order for agencies to use the system to its fullest. Some of the requests made by agencies and subsequently addressed include: • The ability to track the client’s service site and filter reports accordingly • The ability to link clients to family members already entered in the database • The ability to filter reports by various filters not already included in the reports (e.g., service provided) Provided an Incentive for Client Participation Having TA Staff who Fully Understand the Evaluation Ensured Buy-in of Agencies Participating Each family had the possibility of being sampled to participate in a longitudinal study—the Parent Interview—with parents interviewed once a year, for three years. Providing an incentive was important given that the consent form explains that clients have the chance of being selected to participate in the longitudinal study and might, therefore, need encouragement to agree to participate. At the time of agency staff presenting clients with the consent form and Family Intake Form, staff may also communicate that participants whom are selected to participate and complete the Parent Interview would receive a $20 Target gift card. Target gift cards were provided in-person to respondents at the completion of the face-to-face Parent Interview, as an incentive and thank you for their participation. We also mailed the Target gift cards to parents whom completed the interview over the telephone. When contacting parents to schedule the first follow-up Parent Interview (currently ongoing), parents have been quite enthusiastic and anxious to complete the Parent Interview right away, some even requesting to do the interview that day. The role of technical assistance is also best filled by someone who fully understands the evaluation and the data required to meet the needs of the evaluation. This is especially critical to ensuring that all necessary dataare obtained and are of the highest quality. For the purposes of this evaluation, the full-time staff member who provides technical assistance related to data entry was intimately involved in the research design and continues to be involved with all aspects of the evaluation on an ongoing basis. A comprehensive understanding of the evaluation and keen understanding of valid/invalid data enables the TA staff to effectively address data entry problems and explain the purpose and importance of particular data being entered for the evaluation. Explaining the bigger picture and the background has proven especially helpful to win-over users who are resistant, or to correct those who have difficulty understanding how to correctly enter the required data. One of the first major hurdles in any new evaluation/research study that requires agency staff to recruit clients is to obtain the buy-in of all key stakeholders. Early in the evaluation we met with the agencies that provide First 5 services to explain the proposed evaluation design. These meetings included a collaborative effort between the agencies and the evaluation team, in order to determine which data would be reasonable for the agencies to collect and still meet the needs of the evaluation. In addition, during these early meetings we communicated that we would have a reciprocal relationship with the agencies: the evaluation team was committed to, on a regular basis, sharing evaluation findings based upon data that agencies collect. OUR SUCCESS To date there are 38,504 clients entered into the Persimmony web-based data system, covering 21 months of services. Over the first year, 96% of clients agreed to sign consent to participate in the evaluation. The longitudinal study of this evaluation, the Parent Interview, obtained a 65% recruitment rate despite a highly transient population.

2. PersimmonyData System

3. Evaluation Forms

4. Consent Form