slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Hawai’i January, 2008 PowerPoint Presentation
Download Presentation
Hawai’i January, 2008

Loading in 2 Seconds...

play fullscreen
1 / 95

Hawai’i January, 2008 - PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on

Progress toward Measuring Goals in Early Intervention: What’s New from What Counts. Kathy Hebbeler ECO at SRI International. Hawai’i January, 2008. Objectives. Review why data are being collected Describe national trends Identify and address challenges to good data

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Hawai’i January, 2008' - nevina


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Progress toward Measuring Goals in Early Intervention: What’s New from What Counts

Kathy Hebbeler

ECO at SRI International

Hawai’i

January, 2008

objectives
Objectives
  • Review why data are being collected
  • Describe national trends
  • Identify and address challenges to good data
  • Discuss some preliminary data from Hawai’i

Early Childhood Outcomes Center

slide3
Why are we doing this?

Early Childhood Outcomes Center

slide4
Keeping our eye on the prize:

High quality services for children and families that will lead to good outcomes.

Early Childhood Outcomes Center

high quality data on outcomes
High Quality Data on Outcomes
  • Data are a piece of a system that helps to achieve overarching goals for children and families
  • Data yield
    • Findings that can be interpreted as having a particular meaning that should lead to specific actions to improve the system.

Early Childhood Outcomes Center

system for producing good child and family outcomes
System for Producing Good Child and Family Outcomes

Adequate funding

Good outcomes for children and families

High quality services and supports for children 0-5 and their families

Good Federal policies and programs

Good State policies and programs

Good Local policies and programs

Strong Leadership

  • Prof’l Development
  • Preservice
  • Inservice
the vision using data as a tool for program improvement
The Vision: Using Data as a Tool for Program Improvement
  • Hawai’i will have quality data available on an ongoing basis about multiple components of the system
    • Goals for children and families
    • Services provided
    • Personnel (types, qualifications, etc.)
    • Etc.

Early Childhood Outcomes Center

driving force for data on child goals comes from the federal level
Driving Force for Data on Child Goals Comes from the Federal Level
  • Government Performance and Results Act (GPRA)
  • Program Assessment Rating Tool (PART)
  • Individuals with Disabilities Education Act (IDEA)

Early Childhood Outcomes Center

government performance and results act gpra passed in 1993
Government Performance and Results Act (GPRA) passed in 1993
  • Requires goals and indicators be established for IDEA
  • Indicators and data collection further along for school age population than for EC
  • Previously, for early childhood data had been collected on:
    • Number of children served (Part C)
    • Settings (both Part C and 619)

Early Childhood Outcomes Center

osep part evaluation results 2002
OSEP: PART evaluation results (2002)
  • 130 programs examined in 2002; 50% programs had no performance data
  • Programs looking at inputs, not results
  • Part C and Section 619
    • No long-term child outcome goals or data
    • Need to develop a strategy to collect annual performance data in a timely manner

Early Childhood Outcomes Center

federal funding for early intervention
Federal Funding for Early Intervention

Early Childhood Outcomes Center

individuals with disabilities education act
Individuals with Disabilities Education Act

SEC. 616. <<NOTE: 20 USC 1416.>> MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.-…..

…..

``(2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- ``(A) improving educational results and functional outcomes for all children with disabilities;

Early Childhood Outcomes Center

slide16
Where are we now:

Federal reporting requirements

Early Childhood Outcomes Center

osep reporting requirements the goals
OSEP Reporting Requirements: the Goals
  • Positive social emotional skills (including positive social relationships)
  • Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])
  • Use of appropriate behaviors to meet their needs

Early Childhood Outcomes Center

osep reporting categories
OSEP Reporting Categories

Percentage of children who:

a. Did not improve functioning

b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

3 outcomes x 5 “measures” = 15 numbers

Early Childhood Outcomes Center

reporting schedule
Reported February 2007

Entry information: Age expected? Yes, No

One time requirement

Reported for children entering between July 1, 2005, and June 30, 2006

Due February 2008

Data in reporting categories at exit for all children who have been in the program for at least 6 months

Must be reported for the year beginning July 1, 2006

Repeat with next year’s data in 2009, etc.

Reporting Schedule

Early Childhood Outcomes Center

slide20
Also
  • States are required to
    • Make public data reported to OSEP
    • Analyze state data by program (i.e., compute a through e for each program)
    • Make public the data by program

Early Childhood Outcomes Center

point of clarification
Point of clarification

“Why are we comparing children with delays and disabilities to typically developing children?”

Early Childhood Outcomes Center

point of clarification1
Point of clarification
  • Process is NOT about comparing groups of children – it IS about asking how close children are to being able to do what is expected at their age
  • Early learning guidelines
  • Kindergarten and access to the general curriculum

Early Childhood Outcomes Center

slide28
Where are we now:

State decisions and activities

Early Childhood Outcomes Center

slide29

WHY?

Purpose

To meet provider/teacher, local and/or state need for outcome information

and

to respond to federal reporting requirements

To respond to federal reporting requirements

Early Childhood Outcomes Center

state approaches
State approaches
  • Most states have embraced outcomes measurement and are collecting outcomes data for their own purposes.
  • Many states are building bigger systems than needed to produce the federal data.
  • Go to www.the-eco-center.org for more information about what other states are doing

Early Childhood Outcomes Center

how are states collecting child outcomes goal data
How are states collecting child outcomes/goal data?

Possible state approaches to collection of child data

  • Child Outcomes Summary Form (COSF) [= Early Intervention Child Goals Summary Form in HI]
  • Publisher’s online assessment system
  • Single assessment statewide
  • Other approaches

Early Childhood Outcomes Center

state approaches to measurement for part c child outcomes
State approaches to measurement for Part C child outcomes
  • 40 states using the ECO Child Outcomes Summary Form (COSF)
  • 8 states using 1 assessment tool statewide
  • 3 states using on-line assessment systems with the capacity to report OSEP data reports
  • 5 states using other unique approaches

Early Childhood Outcomes Center

variations across states in cgsf implementation
Variations across states in CGSF implementation
  • Some states started early (HI); some did not start until mid to late 2007
  • Some states completing at IFSP; others at a separate meeting
  • Some states including parents in the discussion; some are not

Early Childhood Outcomes Center

where states are now
Where states are now
  • First data on 5 categories due to OSEP February 3
  • Many states do not have data on many children yet
  • Many states focusing on improving the process of collecting the data

Early Childhood Outcomes Center

slide36
What do we know so far:

Positive impacts of the goals rating process

Early Childhood Outcomes Center

positive impacts reported by states
Positive impacts reported by states
  • Increases focus on functional outcomes on IFSPs
  • Easier to write functional outcomes on IFSP
  • Facilitates communication with parents

Early Childhood Outcomes Center

benefits of discussing the 3 goals
Benefits of discussing the 3 goals
  • “Requires us to talk & think in terms of functional behaviors, not test items
  • Incorporates the parents as active and knowledgeable participants
  • Looks at all settings and situations
  • Bridges the gap between assessment tools and real life.”

From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center

benefits
Benefits
  • “Is more meaningful to families
  • Prepares the family for setting IFSP outcomes – thinking about the skills they want their child to have to function in their daily family life
  • Guides us towards discipline-free contextualized goals.”

From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center

benefits of including families
Benefits of INCLUDING families
  • “Determining child progress requires we use the family’s expertise and knowledge of their child across setting and situations
  • Our discussion becomes more inclusive with the family as an equal source of information for assessment purposes.”

From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center

benefits of including families1
Benefits of INCLUDING families
  • “One of the biggest shifts in practice, for many systems, was the move to compare their children in Part C to their same age peers.
  • Looking to children in the frame of same age peers allows us to have authentic, honest discussions with families about their child’s strengths and needs.”

From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center

benefits of including families2
Benefits of INCLUDING families
  • “We need to be comfortable with reporting strengths AND areas of delay, while being family friendly.”

From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center

slide43
What do we know so far:

Challenges to getting good information

Early Childhood Outcomes Center

need for good data
Need for good data
  • Encompasses all three levels: federal, state, local
  • Depends on how well local programs are implementing procedures

Early Childhood Outcomes Center

what we are learning nationally
What we are learning nationally

The process of training for child outcomes data collection has uncovered other areas of significant need related to professional development.

Early Childhood Outcomes Center

essential knowledge for completing the child goals summary form
Essential Knowledge for Completing the Child Goals Summary Form

Between them, team members must:

  • Know about the child’s functioning across settings and situations
  • Understand age-expected child development
  • Understand the content of the three child outcomes
  • Know how to use the rating scale
  • Understand age expectations for child functioning within the child’s culture

Early Childhood Outcomes Center

important point
Important point
  • It is not necessary that all team members be knowledgeable in all 5 areas
  • Especially, no expectation that parents understand the rating scale or typical child development
  • But the professionals have to!

Early Childhood Outcomes Center

providers need to know more about
Providers need to know more about:
  • Assessment
    • How to gather assessment data to reflect functioning across settings and situations, especially how to gather child functioning information from families
    • Understanding the results of the assessment
    • Sharing assessment results sensitively and honestly with families

Early Childhood Outcomes Center

providers need to know more about1
Providers need to know more about:
  • Functional outcomes
    • What are they?
    • How do they differ from outcomes organized around domains?
    • What do they mean for how professionals from different disciplines operate as a team?
  • Typical child development
    • What are the functional expectations for children at different ages with regard to each of the 3 goal statements?

Early Childhood Outcomes Center

is this process too subjective to produce good data
Is this process too subjective to produce good data?
  • Best practices in assessment requires looking at multiples sources of information
  • Assessment as a tool vs. assessment as a process
  • Research on judgment-based assessment indicates it is as good or better than traditional assessment

Early Childhood Outcomes Center

what is informed opinion
What is informed opinion?

Clinical judgment (informed opinion) – knowledgeable perceptions of caregivers and professionals about the elusive and subtle capabilities of children in different settings

Early Childhood Outcomes Center

clinical judgment provides good data when
Clinical judgment provides good data when….
  • Operational definition of child characteristics to be judged
  • Structured format for quantifying characteristics
  • Information from multiple setting and individuals
  • Training in methods that structure and quantify characteristics
  • Decision making based on consensus

From Bagnato, Smith-Jones, Matesa & McKeating-Esterle, 2006

Early Childhood Outcomes Center

ratings clarification
Ratings clarification
  • Highest category (Completely, 7) = Child functions in an age appropriate manner across settings and situations
  • Next highest (6) – Child functions in an age appropriate manner but there is a significant concern about some aspect of the child’s functioning

Early Childhood Outcomes Center

ratings clarification1
Ratings clarification
  • Somewhat (5) = Child shows a MIX of age appropriate and not age appropriate behaviors across settings and situations
  • Between emerging and somewhat (4) – Child shows some age appropriate behavior but rarely

Early Childhood Outcomes Center

ratings clarification2
Ratings clarification
  • Emerging (3) – No age appropriate behavior yet. Shows immediate foundational skills in some to all settings and situations
  • Between not yet and emerging (2) – No age appropriate behavior yet. Rarely uses immediate foundational skills (but does show some).

Early Childhood Outcomes Center

ratings clarification3
Ratings clarification
  • Not yet (1) – No age appropriate behavior yet. No immediate foundational skills yet.

Early Childhood Outcomes Center

should the rating be whatever parents want
Should the rating be whatever parents want?
  • No, the rating is a team consensus
  • Need to think about what and how parents are being involved in this process
  • Have the assessment results been thoroughly explained?
  • Like so much in EI, the rating requires a partnership

Early Childhood Outcomes Center

slide59

ECO Discussion Prompts: Child has positive social relationships

(see ECO Tools)

  • Thinking about relating to adults, relating to other children, and (for those older than 18 months) following rules related to groups or interacting with others.
    • Δ How does the child relate to his/her parent(s)?
    • Δ How does the child relate to other relatives or extended family and close family friends (e.g., grandparents, aunts, extended kin, etc.)? Do these interactions with people differ depending on the setting the child is in with these people?
    • Δ How does the child interact with familiar caregivers (e.g., child care providers, babysitters)?
    • Δ How does the child relate to strangers? At first? After a while? In different settings and using different approaches?
    • Δ How does the child interact with/respond to people in community settings (e.g., park, library, church, grocery store, with neighbors on walks, at the bus stop, in restaurants, at playgroups or outings, etc.)?
    • ….

Early Childhood Outcomes Center

obtaining good data
Obtaining good data
  • Threats to good data
    • Local providers do not understand the procedures
    • Local providers do not follow the procedures
    • And others…..
  • Process requires good training procedures
    • Initial
    • Ongoing

Early Childhood Outcomes Center

many steps to ensuring quality data
Many steps to ensuring quality data

Early Childhood Outcomes Center

slide62
Initial Data from Hawai’i

Early Childhood Outcomes Center

slide63
These data are very, very preliminary.

Early Childhood Outcomes Center

eis average ratings at initial ifsp
EIS Average Ratings at Initial IFSP

Early Childhood Outcomes Center

hs average ratings at initial ifsp
HS Average Ratings at Initial IFSP

Early Childhood Outcomes Center

questions to ask
Questions to ask
  • Do the data make sense?
    • Am I surprised? Do I believe the data? Believe some of the data? All of the data?
  • If the data are reasonable (or when they become reasonable), what might they tell us?

Early Childhood Outcomes Center

validity
Validity
  • Validity refers to the use of the information
  • Does evidence and theory support the interpretation of the data for the proposed use?

Or

  • Are you justified in reaching the conclusion you are reaching based on the data?

Standards for Educational and Psychological Testing (1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

Early Childhood Outcomes Center

how will might these data be used
How will/might these data be used?
  • Federal level
    • Overall funding decisions (accountability)
    • Resource allocation (e.g., what kind of TA to fund?)
    • Decisions about effectiveness of program in individual states
  • State level
    • Program effectiveness??
    • Program improvement??
  • Local level
    • Program improvement??

Early Childhood Outcomes Center

slide84
What has ECO learned after 4+ years?

Early Childhood Outcomes Center

eco message strong commitment
ECO Message: Strong commitment
  • States are committed to building good systems to collect data on how children are progressing
    • Variations in how data are being collected
    • Variations in how states plan to use the information
  • Common thread: Widespread recogniton of the importance of the data

Early Childhood Outcomes Center

eco message need to build state capacity
ECO Message: Need to build state capacity
  • Implement oversight procedures around data quality
  • Examine data for validity
  • Analyze and interpret data for program improvement
  • Develop messages for policy-makers, public, media, families from the data

Early Childhood Outcomes Center

eco message need to build provider capacity
ECO Message: Need to build provider capacity
  • Assessment
  • Functional outcomes
  • Typical child development

Early Childhood Outcomes Center

eco message need for better early childhood assessment tools
ECO Message: Need for better early childhood assessment tools
  • Designed around the 3 functional outcomes for all children
  • Designed to capture child functioning in a variety of setting and situations
  • Designed to be used in accountability and program evaluation
  • Current tools are antiquated:
    • Need to incorporate latest research, recommended practices, psychometrics

Early Childhood Outcomes Center

eco message need for more resources to ensure quality data
ECO Message: Need for more resources to ensure quality data
  • National resources to support and coordinate across states:
    • Training needs
    • Analysis and use of data
  • Support for states to continue to develop and validate their systems
  • Investment in research to examine how outcomes data collection being carried out (impact on quality, local practice, etc.)
  • Investment in new assessment tools

Early Childhood Outcomes Center

slide90
Change can be difficult….

Early Childhood Outcomes Center

slide91
For more information….

Early Childhood Outcomes Center

slide93
http://hawaii.gov/health/family-child-health/eis/whatcounts.htmlhttp://hawaii.gov/health/family-child-health/eis/whatcounts.html

Early Childhood Outcomes Center

slide95
www.the-eco-center.org

Early Childhood Outcomes Center