outcomes and evidence statements n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Outcomes and Evidence Statements PowerPoint Presentation
Download Presentation
Outcomes and Evidence Statements

Loading in 2 Seconds...

play fullscreen
1 / 68

Outcomes and Evidence Statements - PowerPoint PPT Presentation


  • 88 Views
  • Uploaded on

Outcomes and Evidence Statements. Kathy Hebbeler SRI International. Prepared for the Michigan Results Group Lansing, MI July 2005. Objectives for this presentation. Review the ECO outcomes

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Outcomes and Evidence Statements' - amanda


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outcomes and evidence statements

Outcomes and Evidence Statements

Kathy Hebbeler

SRI International

Prepared for the Michigan Results Group

Lansing, MI

July 2005

objectives for this presentation
Objectives for this presentation
  • Review the ECO outcomes
  • Introduce some concepts related to evidence statements to assist Michigan in formulating its evidence statements
  • Share ECO’s recommendations for evidence statements
  • Describe what OSEP will be requiring
  • Introduce the measurement approach ECO is developing
early childhood outcomes eco center mission
Early Childhood Outcomes (ECO) Center: Mission

Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in national and state accountability systems

eco center
ECO Center

5-year project funded by OSEP in October 2003

Collaboration among:

  • SRI International
  • Frank Porter Graham Child Development Institute (U of North Carolina, Chapel Hill)
  • Juniper Gardens Children’s Project (U of Kansas)
  • National Association of State Directors of Special Education
  • University of Connecticut
public policy context
Public Policy Context
  • Age of accountability
  • Accountability increasingly means looking at results – not just process
  • Applies across all private and public human service and education programs
president s commission on excellence in special education 2002
President’s Commission on Excellence in Special Education (2002)

Major Recommendation 1: Focus on results – not process

“IDEA will only fulfill its intended purpose if it raises expectations for students and becomes result-oriented—not driven by process, litigation, regulation and confrontation. In short, the system must be judged by the opportunities it provides and the outcomes achieved for each child.”

federal push for outcome data intro to acronyms
Federal push for outcome data: Intro to Acronyms
  • GPRA =Government Performance and Results Act
  • PART = Program Assessment Rating Tool
  • OMB = Office of Management and Budget
  • IDEA = Individuals with Disabilities Education Act
gpra government performance and results act
GPRA (Government Performance and Results Act)
  • IDEA goals and indicators established
  • Indicators and data collection further along for school age population than for EC
  • For early childhood, data have been collected on:
    • Number of children served (Part C)
    • Settings (both Part C and 619)

Note: Part C = Programs for 0-3 year olds; 619 = Programs for 3-5

part program assessment rating tool
PART (Program Assessment Rating Tool)
  • Tool used to review federal programs
  • Four critical Assessment Areas including

“results and accountability”

  • Programs given ratings from ineffective to effective
  • Purpose – “to enhance budget analysis”
part evaluation findings and recommendations
PART Evaluation: Findings and Recommendations
  • Part C and 619 Findings: No long-term child outcome goals or data
  • The PART required the Department to:

1. Establish long-term outcome-oriented objectives

2. Develop a strategy to collect performance data

  • Office of Special Education Programs (OSEP) asked states to provide outcome data in their Annual Performance Reports (APRs)
reality check aka challenges
Reality Check (aka Challenges)
  • Michigan will be required to submit data to OSEP
  • OSEP has recently released the indicators on which it will be requiring data and the timeline is impossible
  • ECO is simultaneously working on many of the same issues the states are. We have a measurement approach in development but it is far from complete.
  • We have no choice but to live with the uncertainties and move forward as best we can
steps in development of an outcomes system
Steps in Development of an Outcomes System
  • Identify overall purpose for the system (including priorities)
  • Identify outcome areas for children and families
  • Develop outcome statements
  • Formulate the evidence statements
  • Identify measurement approaches (e.g., select instruments)

***Steps 6 to 11 after that

need to acknowledge the range of emotional responses to outcomes systems
Need to acknowledge the range of emotional responses to outcomes systems
  • Hate it, have to do it so I will
  • Hate it, have to do it but I won’t do a very good job of it because it is a waste of my time
  • Neutral, just another fad that will pass
  • Not sure, might be good but not likely and probably will be useless to harmful
  • Worried that this will take time away from children and families
  • Hopeful that this will live up to the potential
  • Excited about the prospect of having data on child outcomes
identify outcomes areas
Identify outcomes areas
  • What are the important outcomes areas?
    • Child
    • Family
  • Everyone wants to know “How are children doing?” but need to decide “with regard to what?”
definitions
Definitions

Outcome—a statement of a measurable condition(s) desired for the population of children or their families

  • Children take appropriate action to meet their needs
  • Families know how to advocate for the services they need
identify outcome areas
Identify outcome areas
  • Outcome statements are the system’s overall vision for children and for families
  • Not the same as outcomes on an IFSP or IEP which are very specific outcomes for an individual child or family
  • Outcomes in an accountability system are global statements of what we are trying to do for children and families
identify outcome areas1
Identify outcome areas
  • How should we think about child outcomes?
    • “…for more than three decades, researchers and service providers have struggled with both the identification of significant child outcomes and their valid and reliable measurement.”

National Research Council and Institute of Medicine, 2000

identify outcome areas2
Identify outcome areas
  • How should we think about child outcomes?
    • Domains (language, cognitive, adaptive, etc.)
    • Functional skills (feeding, dressing, mobility, etc.)
    • Underlying functional capacities (Self-regulation, engagement, knowledge acquisition skills, etc.)
    • Pre-academic skills (Early reading, math, science, etc.)
pitfall alert
Pitfall Alert!
  • What we want for children and families (desired outcomes) reflects our values
  • What we want for young children may not be the same as what can be easily measured
  • When what is valued and what can be measured easily or well do not line up, EITHER:
    • Outcomes will be determined by what can be measured easily or well, OR
    • Some of the outcomes will be more easily measured or more adequately measured than others
eco family outcomes
ECO Family Outcomes
  • Families know their rights and advocate effectively for their children.
  • 2. Families understand their children's abilities and special needs
  • 3. Families help their children develop and learn.
  • 4. Families have social supports.
  • 5. Families are able to gain access to desired services, programs, and activities in their community.
need for an overarching goal for children
Need for an overarching goal for children
  • The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community
eco child outcomes
ECO Child Outcomes

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

issues from stakeholder discussions
Issues from stakeholder discussions
  • Functional outcomes
    • Best practice – kind of outcomes recommended for IFSPs and IEPs
    • Consistent with transdisciplinary service delivery
    • Not reflected well in a pure domains framework
    • Not captured well in current assessment tools
slide24

Children have positive social relationships

Children acquire and use knowledge and skills

Relation-ships with adults

Follows group rules

Symbol use, abstract thinking

Applies

knowledge

Relation-ships with peers

Knowledge of physical world & culture

Children take appropriate action to meet their needs

Practicing

Attending

Playing

Listening

Exploring

Play

Self-care, health and safety

Masters the environ-ment

Being curious

Touching

Persisting

Engaging

Elaboration of the ECO Outcomes

To be active and successful participants

now and in the future

in a variety of settings

To be active and successful participants

now and in the future

in a variety of settings

slide25

Overarching

Goal:

Children will be active and successful participants now and in the future in a variety of settings.

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

ECO Outcomes:

Domains:

Communication

Cognition

Social-Emotional

Content Areas:

Language Arts

Math

Music

Self Help

Motor

Approaches to Learning

Science

Social Studies

Art

Note: Each of these can be broken down further into sub-areas

Processes:

Memory

Self-regulation

Recognizing and interpreting sensory input

Listening

Attending

Etc.

Body Functions:

Movement

-flexibility

-strength

-postural response

Vision

Seeing

Speech production

Etc.

Alternative Ways of Thinking about Child Outcomes

Example of how movement can be sub-divided

what is happening with these outcome areas and statements
What is happening with these outcome areas and statements?
  • ECO’s work is independent of OSEP
  • OSEP has been involved in ECO’s work from the beginning
  • ECO submitted these outcomes to OSEP as its recommendations
  • ECO submitted recommended indicators based on these outcomes to OSEP for the APR/SPP instructions that were available for public comment
slide27
Evidence Statements

(or what do you want to know about those outcomes)

the importance of evidence statements
The Importance of Evidence Statements
  • What constitutes evidence of good outcomes?
  • If you were asked to testify before your state legislature to demonstrate the effectiveness of EI, what kind of evidence do you want to have? (Purpose = accountability)
  • What kind of evidence about outcomes is needed by states and local programs to improve services for children and families? (Purpose = program improvement)
definition of evidence statement
Definition of Evidence Statement

Evidence Statement - a statement that incorporates a statistic and provides evidence as to whether not an outcome has been achieved

  • % of children who have positive social relationships
  • % of children who show progress toward acquiring knowledge and skills

Note: States will be asked to submit a specified kind of evidence to OSEP. Each state will need to be able to produce this kind of evidence but might want other kinds of evidence as well.

features of a good evidence statement
Features of a Good Evidence Statement
  • Credible: Based on valid data
  • Meaningful: The evidence can be interpreted
  • Powerful: The evidence is convincing to those who will be receiving it (purpose = accountability)
  • Useful: The evidence is helpful to those who will use it (purpose = program improvement)
measurement evidence statements
MeasurementEvidence Statements
  • Evidence statements are the ultimate product in an outcomes-based accountability system
  • The nature of the evidence statement that can be produced will depend on what is measured and how often
possible categories of evidence statements for outcome data
Possible Categories of Evidence Statements for Outcome Data
  • Status (achievement at one point in time)

Example: “% of children who [achieved this..]

  • Progress (change relative to earlier status)

Example: “% of children who made progress in..]

two ways to think about progress with young children
Two Ways to Think about Progress with Young Children
  • Continuous Progress = acquisition of new skills and behaviors (almost all children show this)

Or

  • Age-anchored Progress = change in developmental trajectory (i.e., narrowing the gap between a child with a delay and typical development; “less delayed” after intervention)
examples of two kinds of progress
Examples of Two Kinds of Progress
  • Continuous– increase in vocabulary
  • Age-anchored– change from being 1.5 standard deviations below norm to .5 standard deviation
status where children are
Status (where children are)
  • Reports the percentage of children who achieved or could do X
  • Would likely involve some kind of comparison to expectations for children of a given age

Examples:

    • 53% of early intervention graduates were rated as “typical and proficient” with regard to having positive social relationships
status evidence statement
Status Evidence Statement

Pro

  • Requires one measurement point
  • Could be strong evidence
  • Nearly all K-12 evidence statements are of this form (grad. rates, “proficient” reading)

Con

  • What is the benchmark that makes sense for all children with disabilities?
  • How does one interpret the data (is this finding good news or bad news?)
continuous progress change compared to earlier status
Continuous Progress (change compared to earlier status)
  • Reports % of children who improved
  • Progress on a curriculum-based assessment

Examples:

    • % of preschool children who made progress toward having positive social relationships
    • % of preschool children who showed an increase of 3 of more objectives related to positive social relationships scale after one year
continuous progress
Continuous Progress

Pro

  • Straightforward
  • Could quantify extent of child growth
  • Incorporates all levels of gain, even the very small gains made by children with severe impairments

Con

  • Requires at least 2 points of data
  • Nearly all children will improve over time
  • Weak evidence. What constitutes “good news”?
age anchored progress closing the gap toward typical development
Age-Anchored Progress (closing the gap toward typical development)
  • Captures progress relative to same age peers

Examples:

    • 23% of children moved from low performing to age appropriate in positive social relationships by kindergarten entry
    • With regard to acquiring and using knowledge and skills, 75% of children made progress sufficient to maintain their functioning at an age appropriate level
age anchored progress closing the gap toward typical development1
Age-Anchored Progress (closing the gap toward typical development)

Pro

  • Consistent with the intent of intervening
  • Could produce strong evidence

Con

  • Requires at least 2 time points of data
  • Closing the gap is not an expectation for all children with disabilities.
  • How is the evidence to be interpreted?
    • If some children don’t close the gap, is this interpreted as lack of success for the program?
deciding on desired evidence
Deciding on desired evidence
  • Not an either-or
  • Different levels will want different kinds of evidence
  • Many different kinds of evidence can be produced from the same set of data
    • If the desired evidence statements are identified as part of the planning
reporting requirement part c
Reporting Requirement: Part C

Percent of infants and toddlers with IFSPs who demonstrate improved:

  • Positive social-emotional skills (including social relationships)
  • Acquisition and use of knowledge and skills (including early language/communication)
  • Use of appropriate behaviors to meet their needs
the details
The details
  • % of infants and toddlers who reach or maintain functioning at a level comparable to same-aged peers
  • % of infants and toddlers who improve functioning
  • % of infants and toddlers who did not improve functioning.

If children meet the criteria for a, report them in a. Do not include children reported in a in b or c.

3 osep evidence statements
3 OSEP Evidence Statements

OSEP a

  • Maintained functioning comparable to age peers
  • Achieved functioning comparable to age peers

OSEP b

  • Moved nearer functioning comparable to age peers
  • Made progress; no change in trajectory

OSEP c

  • Did not make progress
two kinds of outcomes questions for each child with a disability
Two kinds of outcomes questions for each child with a disability
  • How is this child doing relative to the individualized outcomes that were established through the IFSP or IEP process?
  • How is this child doing relative to same-aged peers?
part c indicators family outcomes
Part C Indicators Family Outcomes
  • Percent of families participating in Part C who report that early intervention services have helped the family
    • Know their rights;
    • Effectively communicate their children’s needs; and
    • Help their children develop and learn.
measurement guidelines
Measurement guidelines
  • State selected data source, sampling permitted
  • % of families who report that early intervention services have helped their family in each of the three areas
spp apr measurement implications
SPP/APRMeasurement implications
  • Requires data collection from family members/caregivers
  • Data at one time period
  • Asks families to recognize that early intervention services contributed to the change or status listed.
  • Does not require family data to be linked with child data
key questions
Key Questions
  • How can Michigan provide the indicators being requested by OSEP?
  • What kinds of evidence statements does Michigan want to be able to produce for its own use?
  • What do the answers mean for measurement?
rolling up assessment data to the outcomes
Rolling up assessment data to the outcomes
  • Given:
    • Different children will have different kinds of assessment data.
  • Also:
    • The same child may have several different kinds of assessment data.
  • Challenge: How to roll up multiple pieces of information (e.g., data from different assessment tools) to a single score?
slide56

Overarching

Goal:

Children will be active and successful participants now and in the future in a variety of settings.

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

ECO Outcomes:

Domains:

Communication

Cognition

Social-Emotional

Content Areas:

Language Arts

Math

Music

Self Help

Motor

Approaches to Learning

Science

Social Studies

Art

Note: Each of these can be broken down further into sub-areas

Processes:

Memory

Self-regulation

Recognizing and interpreting sensory input

Listening

Attending

Etc.

Body Functions:

Movement

-flexibility

-strength

-postural response

Vision

Seeing

Speech production

Etc.

Example of how movement can be sub-divided

different assessments on different children
Different assessments on different children

Assessment 1

Roll up Process

Assessment 2

Single

Score

Assessment 3

Assessment 4

slide58

Different pieces of information on the same child

Assessment 1

Roll up Process

Assessment 2

Single

Score

Parent Report

Informed Clinical Opinion

rolling up from the different assessments
Rolling up from the different assessments
  • ECO is working on a scale that will provide a way to give a child a numerical rating on each outcome
    • It is NOT an assessment tool.
    • It assumes teachers and providers are already using an assessment tool to collect information about how children are doing.
    • The scale is a way to take what those closest to the child know and put a number on it.
what does the outcomes scale look like
What does the outcomes scale look like?
  • 5 pt and a 7 pt version
    • The precision question….
  • The high point (5 or 7) indicates outcome achieved at an age-expected level
  • The lowest point (1) indicates the farthest distance from age-expectations
what does the outcomes scale look like1
What does the outcomes scale look like?
  • Challenge is in how to anchor and provide the descriptors for the middle points on the scale.
  • Progress is moving up the scale between time 1 and time 2.
measuring the family outcomes
Measuring the Family Outcomes
  • ECO working on a tool
  • ECO working on a document with guidance for states that reviews key decisions and includes several options
staying in touch
Staying in touch
  • Web site: the-eco-center.org
    • Follow developments related to work of the Center
    • Obtain ECO resource documents
    • Obtain other related resources
  • Email: staff@the-eco-center.org
  • ECO sends to existing listservs
  • Can join the ECO mailing list
vision
Vision
  • To build a system where all the pieces will fit together smoothly
  • Meaningful outcomes data collected regularly that can used in programs by programs AND reported to the state (and feds)
slide66

Local data

State needs