Overview of ENHANCE

1. Overview of ENHANCE Research Underway on the Validity of the Child Outcomes Summary Form (COSF) Lauren Barton Donna Spiker July 31, 2010 Measuring Child and Family Outcomes Conference Arlington, VA

2. Topics Need for ENHANCE Considerations - building a validity argument ENHANCE project and studies When and how will studies be conducted? Suggestions for project to better meet state needs

3. The Challenge

4. “…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool or school programs, and in the community.” (from Early Childhood Outcomes Center, http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf) Child Outcomes Goal of Early Intervention and Early Childhood Special Education

5. Demands Grew Data DataDataData

6. The Need High quality child outcomes data for accountability FAST

7. The Need for An approach or tool that: • Captures multiple sources of information and child’s functioning across settings • Doesn’t require programs to change assessments • Relates to age-expected child functioning • Measures progress over time • Is inexpensive • Is ready to be implemented now • Is valid and reliable

8. The Response The Child Outcomes Summary Form (COSF)

9. The COSF • Captures multiple sources of information and child’s functioning across settings • Doesn’t require programs to change assessments • Relates to age-expected child functioning • Measures progress over time • Inexpensive • Ready to be implemented now • Valid and reliable

10. More information is needed to improve guidance about the COSF process, to document its reliability and validity, and to inform appropriate use of the data we have Result COSF is in use in over 40 states

11. What is Your Experience? • Who is asking you for validity information on the COSF? • What do they want to know? • What would you like to be able to tell them? • Is your state taking actions to look at the validity of COSF data on your own?

12. Building a Validity Argument

13. Early Evidence Belief in potential for COSF to be valid based on: • Existing literature: team-based decision-making can be reliable and valid • Existing literature: teams are effective in identifying individual children’s functioning so that they can plan and deliver appropriate services • Early data from states: pilot sites, small n’s showing similarity in distributions, sensible patterns for subgroups • Anecdotal data from trainers: participants reach decisions fairly easily and consistently

14. Validity – What Are We Trying to Demonstrate? • Validity refers to the use of the information • Are you justified in reaching the conclusion you are reaching based on the data? Standards for Educational and Psychological Testing(1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

15. Validity Questions The question is NOT “Are the data valid?” • The question is: “Are the data for valid for the purpose of….?” • Are the data sufficiently trustworthy to lead to sound decisions? Examples: funding, TA, focused monitoring… • How much error is acceptable? There will be error…

16. Implications for COSF Data • Validity is NOT a characteristic of an assessment or measurement device. • It is a characteristic of the data produced by the tool and how these data are used. • Implications: State A’s COSF data could be valid; State B’s COSF data could not be.

17. Implications for Studying Validity of COSF Data in States • Need to examine the conditions under which COSF data produce more or less valid conclusions • Study across states with variation in how they implement the COSF • Examine if COSF data leads to appropriate conclusions when it is used • Use findings to generate guidance • States can use guidance to structure conditions that increase the likelihood of generating valid data with the COSF process.

18. Validation Process • Validation process: • Develop propositions (validity argument) - If data were valid for this use, then we would see…. • Collect evidence to examine those propositions in various locations with differences in implementation

19. Examples of propositions in the COSF Validity Argument 3. Children differ from one another with regard to level of functioning in the 3 outcome areas as reflected in COSF ratings. 7. Functioning (COSF ratings) in an outcome area at time 1 is related to functioning in that area at a later point in time. 9. COSF ratings will be related to the nature and severity of the child’s disability. 12. COSF ratings will not be related to: • the particular assessment tool used • extraneous child characteristics such as gender and ethnicity 14. Distributions of COSF ratings at entry will be similar across states serving similar populations and related to the percentage of children served in Part C or Part B Preschool.

20. ENHANCE: The Project and The Studies

21. ENHANCE Project launched by the Early Childhood Outcomes Center (ECO) and SRI International Funded by the U.S. Dept. of Education, Institute for Educational Sciences Series of studies designed to find out: the conditions under which the Child Outcomes Summary Form (COSF) Process produces meaningful and useful data for accountability and program improvement the positive and/or negative impact of the COSF process on programs and staff what revisions to the form and/or the process are needed

22. Five ENHANCE Studies • State Data Study • Comparison with Child Assessments • Team Decision-Making • Record Reviews • Provider Survey

23. State Data Study • Extant data from states using COSF • State level population data on children birth-5 with disabilities served under IDEA • Analyze characteristics of COSF data and relationships to other variables • Interested in volunteers for state data study

24. Sites for 4 Other Studies PART C 6 states with 3 local programs each PART B Preschool 6 states with 3 local programs each 18 Part C + 18 Part B Preschool = 36 Programs States: IL, ME, MN, NM, NC, SC, TX

25. Comparison with Child Assessments Study Goals Over time, compare: COSF ratings and scores on assessments (correlations) Conclusions derived (progress category determinations) Process COSF ratings and external BDI-2 & Vineland-II assessment near program entry again at program exit Sample 6 children at each site (216 total)

26. Goals Do teams reach a rating consistent with evidence presented? Examine understanding and application of outcomes and the rating criteria Process Videotape teams discussing and reaching consensus on a COSF rating Sample 10 children at each site (360 total) ½ entry COSF meetings ½ exit COSF meetings Team Decision-Making Study

27. Goals How well do the ratings reflect the broader evidence in the child’s file? Was understanding of the outcomes and rating criteria applied consistently? Quantity and quality of documentation used to determine the COSF rating Generate a protocol, procedures, and a set of analytic methods for future state use Process Onsite review of child records, including COSF ratings and documentation. Sample 10 children at each site (360 total) ½ entry records; ½ exit records Record Reviews

28. Goals What processes are being implemented to determine COSF ratings? What is knowledge of outcomes, COSF criteria, age-expected behavior? What is the impact of COSF on their practice? Provider Survey Process Survey responses (online or paper) Sample All providers in program who participate in COSF

29. Timing Child Assessments 2010 2011 2012… Nov. or Jan. March June Aug. Sept. Until children exit program Record Reviews Provider Survey Team Videos

30. Discussion Questions • What facets of the COSF process do you want studied? • What kinds of evidence-based guidance would be most useful for you? • Other questions, comments, or reactions?

31. Find out more ENHANCE Website http://ENHANCE.sri.com ECO Center Website http://www.the-ECO-center.org Contact ENHANCE staff E-mail: ENHANCE@sri.com

32. Questions? Comments? Reactions?