The How and Why of Functional Child Outcomes in an Outcomes Measurement System for Part C and Part B Preschool Programs

1. The How and Why of Functional Child Outcomes in an Outcomes Measurement System for Part C and Part B Preschool Programs Donna Spiker Kathy Hebbeler Steve Bagnato Conference on Research Innovations in Early Intervention San Diego, CaliforniaFebruary 2008

2. In this session, we will: • Provide a brief history of accountability systems for Part C and Part B preschool special education. • Describe the current Federal outcomes reporting requirements. • Discuss the 3 OSEP functional child outcomes; how and why we arrived at them. • Describe assessment dilemmas posed. • Present on Research Foundations on Assessment for Early Childhood Intervention. • Pose questions for dialogue about research and practices about assessment for early childhood intervention regarding functional outcomes in an accountability system.

3. A provocative quote….. “You can’t test four-year-old kids – it’s unreliable.” Comment by Ed Zigler in the Jan/Feb 2008 issue of Mother Jones magazine about the Head Start National Reporting System

4. Challenges in measuring child outcomes for children with disabilities • Outcomes are individualized; no consensus on general outcomes. • Extreme diversity in functional levels in the population of children served. • Variety of assessment tools used because no single tool appropriate for the entire population. • Children with severe disabilities will show poor outcomes – would policy-makers conclude the programs were ineffective?

5. And a follow-up quote….. “Head Start spends $7 billion a year. Taxpayers and Congress have the right to know whether it’s working. But there can be good accountability and lousy accountability.” Comment by Ed Zigler in the Jan/Feb 2008 issue of Mother Jones magazine about the Head Start National Reporting System

6. SO……the accountability question and dilemma How do we produce valid data for accountability on programs for young children with disabilities, while remaining true to what we know about best assessment practices?

7. Accountability for EI and preschool special education programs: Background

8. Critical events in accountability for programs for young children with disabilities • 1993 GPRA (Government Performance and Results Act) passed • 2002 PART finds there are no data on outcomes for Part C or 619 • 2003 OSEP begins to ask states for EI child outcome data (and funds the ECO Center) • 2005 OSEP releases federal reporting requirements • 2008 All states begin reporting first set of outcomes data to OSEP

9. PART Findings: “Results Not Demonstrated” (2002) Part C: “While the program has met its goal relating to the number of children served, it has not collected information on how well the program is doing to improve the educational and developmental outcomes of infants and toddlers served.” Part B preschool: “The Department has no performance information on preschool children with disabilities served by the program.”

10. GPRA to PART: Intervening years • Special Education (K-12) • National study found poor outcomes. • Push to include students with disabilities in statewide assessment systems. • Early Childhood • Debate about whether child outcomes should be measured at all. • Much discussion of the many problems in trying to measure outcomes for young children with disabilities.

11. Measuring child outcomes for Part C and Part B preschool (619) • The PART findings in 2002 put an end to the debate about whether or not to do it. • Unfortunately, almost no progress had been made in the intervening years as to HOW to do it.

12. OSEP’s response to PART findings • Required states to submit outcome data in their Annual Performance Report (APR). • Funded the Early Childhood Outcomes (ECO) Center in Sept 2003 to build consensus, make recommendations, and assist states in developing systems to measure outcomes.

13. What is the ECO Center? 5-year OSEP-funded project Collaboration among: • SRI International • Frank Porter Graham Child Development Institute • Juniper Gardens Children’s Project • RTI International • University of Connecticut

14. What was the goal of the ECO Center? • Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in national and state accountability systems: • Outcomes data that can be aggregated across all states and territories. • Outcome systems in states that meet the individual state’s need for data. • For more information, see: www.the-eco-center.org

15. Federal government is the driving force behind collection outcome data…….BUT… • Many state agencies and local programs understand the value of outcomes data for their own purposes: • To document program effectiveness • Increase in funding? • To improve programs: • Identify strengths and weaknesses. • Allocate support resources, such as technical assistance.

16. Were there any parameters from OSEP about state reporting? • Givens from OSEP: • States would not be required to use one instrument. • Data on % of IFSP/IEP goals attained would not be acceptable. • No additional resources would be provided for data collection.

17. What stakeholder input did ECO receive? • Reflect vision of what programs are trying to accomplish; consistent with IDEA • One set of outcomes 0-5 and across types of disabilities. • Use them to drive practice forward - consistent with best practices; no harm. • Avoid domain organization. • Implementation issues to consider: • Children participate in multiple accountability systems. • Providers are overburdened. • Statements should be easy to explain/understand. • Importance of family outcomes as well (similar, but separate process).

18. Overarching goal of programs • To help children become active and successful participants • Now and in the future. • In variety of settings: • in their homes with their families, • in childcare or school programs, and • in the community.

19. The three OSEP functional child outcomes • Children have positive social-emotional skills (including social relationships). • Children acquire and use knowledge and skills (including early language/communication [and early literacy]). • Children use appropriate behaviors to meet their needs.

20. Children have positive social relationships • Involves: • Relating with adults • Relating with other children • For older children, following rules related to groups or interacting with others • Includes areas like: • Attachment/separation/autonomy • Expressing emotions and feelings • Learning rules and expectations • Social interactions and play

21. Children acquire and use knowledge and skills • Involves: • Thinking and reasoning • Remembering • Problem solving • Using symbols and language • Understanding physical and social worlds • Includes: • Early concepts—symbols, pictures, numbers, classification, spatial relationships • Imitation • Object permanence • Expressive language and communication • Early literacy

22. Children take appropriate action to meet their needs • Involves: • Taking care of basic needs • Using tools (e.g., fork, toothbrush, crayon) • Getting from place to place • In older children, contributing to their own health and safety • Includes: • Integrating motor skills to complete tasks • Self-help skills (e.g., dressing, feeding, grooming, toileting, household responsibility) • Acting on the world to get what one wants

23. Outcomes reflect global functioning • Each outcome is a snapshot of: • The whole child • Status of the child’s current functioning • Functioning across settings and situations • Rather than: • Skill by skill • In one standardized way • Split by domains

24. OSEP child outcomes reporting categories Percentage of children who: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same-aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers

25. Implemented on a very fast timeline…… • Aug. 2005 - OSEP released child outcome reporting requirements (no additional funding provided) • First data due Feb. 2007 • Status data from children July 1, 2005- June 30, 2006 • Progress data entry to exit for children in program at least 6 months – due the following year and thereafter

26. Major state dilemmas • How to measure 3 functional outcomes with data from domain-organized assessments that often focus on more discrete skills? • What kind of assessment(s) to use? • Use of 1-to-1 standardized tools or observational approaches?

27. Problem: Domains assessed are interrelated • Children tend to develop in multiple areas simultaneously. • Language, cognition, motor skills march forward more or less together. • Even though development has been divided into domains for assessment and research, much of development is intertwined. • Performance on a given task often requires using skills that cross several domains. • These interconnections present challenges for obtaining a “pure” domain score. • These interconnections present challenges for obtaining valid scores when a child has delays in development in one or more domains.

28. What do we mean by functional outcomes? Important behaviors acquired and displayed by young children in everyday settings that cut across domains: • Integrated behaviors that are meaningful to the child in the context of everyday living • Integrated patterns of behaviors or skills that allow the child to achieve important everyday goals

29. Functional outcomes are NOT • A single behavior • The sum of a set of discrete behaviors or splinter skills such as…..

30. Functional outcomes • Not domains-based, not separating child development into discrete areas (communication, gross motor, etc.) • Refer to behaviors that integrate skills across domains • Can involve multiple domains • Emphasize how the child is able to carry out meaningful behaviors in a meaningful context

31. Isolated skill/behavior Knows how to imitate a gesture when prompted by others Uses finger in pointing motion Uses 2-word utterances Functional skill/behavior Watches what a peer says or does and incorporates it into his/her own play Points to indicate needs or wants Engages in back and forth verbal exchanges with caregivers using 2-word utterances Thinking functionally (within age-expected bounds)

32. How well do current assessments measure functioning for young children?

33. Problem: Nature of the young child makes accurate and reliable assessments difficult • Performance varies from • day to day, • place to place, • person to person. • Don’t perform well for strangers or on demand. • Growth is sporadic and uneven. • Not well suited to a standardized testing situation.

34. Problem: Difficulty of assessment increases with special populations • Concerns with the accuracy of assessments multiply with children: • from diverse cultures/new immigrants. • who are English language learners. • with disabilities, developmental delays, or other special needs.

35. Problem: Issues with assessing English language learners • Do they understand the directions? • Is the assessment tapping cognition or language? • Do norms exist for the population being tested? • Do cultural differences affect the child’s interactions with the assessor in ways that make the child’s performance an inadequate indication of his/her competence?

36. Problem: Issues with assessing children with disabilities or delays • Assessment assumes that child can see, hear, and understand spoken language, point, etc. • Few assessments include accommodations, nor were children with disabilities included in the norming sample. • Very little data on validity of accommodations with young children. • Are other behavioral or attentional factors influencing performance?

37. Recommended best practices for assessing young children • Instruments only used for intended purposes. • Assessors are well trained in early child development and assessment principles. • Assessors are knowledgeable about child’s culture, and assess children in their dominant language. • Assessments should include multiple sources of evidence gathered over time.

38. More on best practices from Steve Bagnato……

39. Questions for discussion • How do we reconcile: • Best practices in early childhood assessment? • The need to measure functional outcomes (and limitations of existing instruments available)? • The need to measure for purposes of accountability?

40. Applied issues for future research and practice…… • Design national measurement field-validation research for specific EI purposes. • Design measures with universal-design features. • Develop efficient and functional measures or mapping procedures for accountability. • Change from domain-based to functional whole-child focus. • Conduct research on authentic and alternative assessment procedures that balance internal and external validity.

41. Applied issues for future research and practice….. (cont) • Reduce burden on providers by enabling use of CBA measures already used by programs or efficient probes for accountability purposes mapped to OSEP/state standards. • Collect and link child outcome measures to measures of programmatic, family, & contextual dimensions. • Consider advantages of applying an overarching interdisciplinary framework for disability outcome research, such as WHO-ICF-CY.

42. New Pew Charitable Trusts report about early childhood accountability Full report is available at: http://www.pewtrusts.org/uploadedFiles/wwwpewtrustsorg/Reports/Pre-k_education/task_force_report1.pdf

43. Additional references about EC assessment • Where We Stand?—Curriculum, Assessment, & Program Evaluation (NAEYC, 2002) • DEC Recommended Practices (Sandall et al, 2005; Neisworth & Bagnato, 2005) • DEC—Promoting Positive Outcomes—Recommendations for Curriculum, Assessment, and Program Evaluation (DEC, 2007) • Child Outcomes: Measuring Success in Head Start (Children & Families, NHSA, 2003)

44. Contact information • Donna Spiker – donna.spiker@sri.com • Kathy Hebbeler – kathleen.hebbeler@sri.com Additional resources on assessment and accountability systems for programs serving young children: see www.the-eco-center.org