Outcomes and Evidence Statements

1. Outcomes and Evidence Statements Kathy Hebbeler SRI International Prepared for the Michigan Results Group Lansing, MI July 2005

2. Objectives for this presentation • Review the ECO outcomes • Introduce some concepts related to evidence statements to assist Michigan in formulating its evidence statements • Share ECO’s recommendations for evidence statements • Describe what OSEP will be requiring • Introduce the measurement approach ECO is developing

3. Early Childhood Outcomes (ECO) Center: Mission Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in national and state accountability systems

4. ECO Center 5-year project funded by OSEP in October 2003 Collaboration among: • SRI International • Frank Porter Graham Child Development Institute (U of North Carolina, Chapel Hill) • Juniper Gardens Children’s Project (U of Kansas) • National Association of State Directors of Special Education • University of Connecticut

5. Public Policy Context • Age of accountability • Accountability increasingly means looking at results – not just process • Applies across all private and public human service and education programs

6. President’s Commission on Excellence in Special Education (2002) Major Recommendation 1: Focus on results – not process “IDEA will only fulfill its intended purpose if it raises expectations for students and becomes result-oriented—not driven by process, litigation, regulation and confrontation. In short, the system must be judged by the opportunities it provides and the outcomes achieved for each child.”

7. Federal push for outcome data: Intro to Acronyms • GPRA =Government Performance and Results Act • PART = Program Assessment Rating Tool • OMB = Office of Management and Budget • IDEA = Individuals with Disabilities Education Act

8. GPRA (Government Performance and Results Act) • IDEA goals and indicators established • Indicators and data collection further along for school age population than for EC • For early childhood, data have been collected on: • Number of children served (Part C) • Settings (both Part C and 619) Note: Part C = Programs for 0-3 year olds; 619 = Programs for 3-5

9. PART (Program Assessment Rating Tool) • Tool used to review federal programs • Four critical Assessment Areas including “results and accountability” • Programs given ratings from ineffective to effective • Purpose – “to enhance budget analysis”

10. PART Evaluation: Findings and Recommendations • Part C and 619 Findings: No long-term child outcome goals or data • The PART required the Department to: 1. Establish long-term outcome-oriented objectives 2. Develop a strategy to collect performance data • Office of Special Education Programs (OSEP) asked states to provide outcome data in their Annual Performance Reports (APRs)

11. Reality Check (aka Challenges) • Michigan will be required to submit data to OSEP • OSEP has recently released the indicators on which it will be requiring data and the timeline is impossible • ECO is simultaneously working on many of the same issues the states are. We have a measurement approach in development but it is far from complete. • We have no choice but to live with the uncertainties and move forward as best we can

12. Steps in Development of an Outcomes System • Identify overall purpose for the system (including priorities) • Identify outcome areas for children and families • Develop outcome statements • Formulate the evidence statements • Identify measurement approaches (e.g., select instruments) ***Steps 6 to 11 after that

13. Need to acknowledge the range of emotional responses to outcomes systems • Hate it, have to do it so I will • Hate it, have to do it but I won’t do a very good job of it because it is a waste of my time • Neutral, just another fad that will pass • Not sure, might be good but not likely and probably will be useless to harmful • Worried that this will take time away from children and families • Hopeful that this will live up to the potential • Excited about the prospect of having data on child outcomes

14. Identify outcomes areas • What are the important outcomes areas? • Child • Family • Everyone wants to know “How are children doing?” but need to decide “with regard to what?”

15. Definitions Outcome—a statement of a measurable condition(s) desired for the population of children or their families • Children take appropriate action to meet their needs • Families know how to advocate for the services they need

16. Identify outcome areas • Outcome statements are the system’s overall vision for children and for families • Not the same as outcomes on an IFSP or IEP which are very specific outcomes for an individual child or family • Outcomes in an accountability system are global statements of what we are trying to do for children and families

17. Identify outcome areas • How should we think about child outcomes? • “…for more than three decades, researchers and service providers have struggled with both the identification of significant child outcomes and their valid and reliable measurement.” National Research Council and Institute of Medicine, 2000

18. Identify outcome areas • How should we think about child outcomes? • Domains (language, cognitive, adaptive, etc.) • Functional skills (feeding, dressing, mobility, etc.) • Underlying functional capacities (Self-regulation, engagement, knowledge acquisition skills, etc.) • Pre-academic skills (Early reading, math, science, etc.)

19. Pitfall Alert! • What we want for children and families (desired outcomes) reflects our values • What we want for young children may not be the same as what can be easily measured • When what is valued and what can be measured easily or well do not line up, EITHER: • Outcomes will be determined by what can be measured easily or well, OR • Some of the outcomes will be more easily measured or more adequately measured than others

20. ECO Family Outcomes • Families know their rights and advocate effectively for their children. • 2. Families understand their children's abilities and special needs • 3. Families help their children develop and learn. • 4. Families have social supports. • 5. Families are able to gain access to desired services, programs, and activities in their community.

21. Need for an overarching goal for children • The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community

22. ECO Child Outcomes Children have positive social relationships Children acquire and use knowledge and skills Children take appropriate action to meet their needs

23. Issues from stakeholder discussions • Functional outcomes • Best practice – kind of outcomes recommended for IFSPs and IEPs • Consistent with transdisciplinary service delivery • Not reflected well in a pure domains framework • Not captured well in current assessment tools

24. Children have positive social relationships Children acquire and use knowledge and skills Relation-ships with adults Follows group rules Symbol use, abstract thinking Applies knowledge Relation-ships with peers Knowledge of physical world & culture Children take appropriate action to meet their needs Practicing Attending Playing Listening Exploring Play Self-care, health and safety Masters the environ-ment Being curious Touching Persisting Engaging Elaboration of the ECO Outcomes To be active and successful participants now and in the future in a variety of settings To be active and successful participants now and in the future in a variety of settings

25. Overarching Goal: Children will be active and successful participants now and in the future in a variety of settings. Children have positive social relationships Children acquire and use knowledge and skills Children take appropriate action to meet their needs ECO Outcomes: Domains: Communication Cognition Social-Emotional Content Areas: Language Arts Math Music Self Help Motor Approaches to Learning Science Social Studies Art Note: Each of these can be broken down further into sub-areas Processes: Memory Self-regulation Recognizing and interpreting sensory input Listening Attending Etc. Body Functions: Movement -flexibility -strength -postural response Vision Seeing Speech production Etc. Alternative Ways of Thinking about Child Outcomes Example of how movement can be sub-divided

26. What is happening with these outcome areas and statements? • ECO’s work is independent of OSEP • OSEP has been involved in ECO’s work from the beginning • ECO submitted these outcomes to OSEP as its recommendations • ECO submitted recommended indicators based on these outcomes to OSEP for the APR/SPP instructions that were available for public comment

27. Evidence Statements (or what do you want to know about those outcomes)

28. The Importance of Evidence Statements • What constitutes evidence of good outcomes? • If you were asked to testify before your state legislature to demonstrate the effectiveness of EI, what kind of evidence do you want to have? (Purpose = accountability) • What kind of evidence about outcomes is needed by states and local programs to improve services for children and families? (Purpose = program improvement)

29. Definition of Evidence Statement Evidence Statement - a statement that incorporates a statistic and provides evidence as to whether not an outcome has been achieved • % of children who have positive social relationships • % of children who show progress toward acquiring knowledge and skills Note: States will be asked to submit a specified kind of evidence to OSEP. Each state will need to be able to produce this kind of evidence but might want other kinds of evidence as well.

30. Features of a Good Evidence Statement • Credible: Based on valid data • Meaningful: The evidence can be interpreted • Powerful: The evidence is convincing to those who will be receiving it (purpose = accountability) • Useful: The evidence is helpful to those who will use it (purpose = program improvement)

31. MeasurementEvidence Statements • Evidence statements are the ultimate product in an outcomes-based accountability system • The nature of the evidence statement that can be produced will depend on what is measured and how often

32. Possible Categories of Evidence Statements for Outcome Data • Status (achievement at one point in time) Example: “% of children who [achieved this..] • Progress (change relative to earlier status) Example: “% of children who made progress in..]

33. Two Ways to Think about Progress with Young Children • Continuous Progress = acquisition of new skills and behaviors (almost all children show this) Or • Age-anchored Progress = change in developmental trajectory (i.e., narrowing the gap between a child with a delay and typical development; “less delayed” after intervention)

34. Examples of Two Kinds of Progress • Continuous– increase in vocabulary • Age-anchored– change from being 1.5 standard deviations below norm to .5 standard deviation

36. Advantages and Disadvantages of Different Evidence Statements

37. Status (where children are) • Reports the percentage of children who achieved or could do X • Would likely involve some kind of comparison to expectations for children of a given age Examples: • 53% of early intervention graduates were rated as “typical and proficient” with regard to having positive social relationships

38. Status Evidence Statement Pro • Requires one measurement point • Could be strong evidence • Nearly all K-12 evidence statements are of this form (grad. rates, “proficient” reading) Con • What is the benchmark that makes sense for all children with disabilities? • How does one interpret the data (is this finding good news or bad news?)

39. Continuous Progress (change compared to earlier status) • Reports % of children who improved • Progress on a curriculum-based assessment Examples: • % of preschool children who made progress toward having positive social relationships • % of preschool children who showed an increase of 3 of more objectives related to positive social relationships scale after one year

40. Continuous Progress Pro • Straightforward • Could quantify extent of child growth • Incorporates all levels of gain, even the very small gains made by children with severe impairments Con • Requires at least 2 points of data • Nearly all children will improve over time • Weak evidence. What constitutes “good news”?

41. Age-Anchored Progress (closing the gap toward typical development) • Captures progress relative to same age peers Examples: • 23% of children moved from low performing to age appropriate in positive social relationships by kindergarten entry • With regard to acquiring and using knowledge and skills, 75% of children made progress sufficient to maintain their functioning at an age appropriate level

42. Age-Anchored Progress (closing the gap toward typical development) Pro • Consistent with the intent of intervening • Could produce strong evidence Con • Requires at least 2 time points of data • Closing the gap is not an expectation for all children with disabilities. • How is the evidence to be interpreted? • If some children don’t close the gap, is this interpreted as lack of success for the program?

43. Deciding on desired evidence • Not an either-or • Different levels will want different kinds of evidence • Many different kinds of evidence can be produced from the same set of data • If the desired evidence statements are identified as part of the planning

44. What OSEP will be requiring from states for Part C and Preschool Special Education

45. Reporting Requirement: Part C Percent of infants and toddlers with IFSPs who demonstrate improved: • Positive social-emotional skills (including social relationships) • Acquisition and use of knowledge and skills (including early language/communication) • Use of appropriate behaviors to meet their needs

46. The details • % of infants and toddlers who reach or maintain functioning at a level comparable to same-aged peers • % of infants and toddlers who improve functioning • % of infants and toddlers who did not improve functioning. If children meet the criteria for a, report them in a. Do not include children reported in a in b or c.

48. 3 OSEP Evidence Statements OSEP a • Maintained functioning comparable to age peers • Achieved functioning comparable to age peers OSEP b • Moved nearer functioning comparable to age peers • Made progress; no change in trajectory OSEP c • Did not make progress

49. Two kinds of outcomes questions for each child with a disability • How is this child doing relative to the individualized outcomes that were established through the IFSP or IEP process? • How is this child doing relative to same-aged peers?

50. Part C Indicators Family Outcomes • Percent of families participating in Part C who report that early intervention services have helped the family • Know their rights; • Effectively communicate their children’s needs; and • Help their children develop and learn.