Measuring Child and Family Outcomes

Measuring Child and Family Outcomes Session One Linking Assessment to Functional Outcomes and Accountability Anne Brager, MS, RN Program Supervisor Frederick County Infants and Toddlers Program Adapted from ECO Center

Good Outcomes Begin with Good Assessment! What are Outcomes? • Outcomes are the benefits that children and families experience as a result of early intervention • The three child outcomes, as measured for accountability purposes are different than IFSP outcomes in that: • OSEP’s three child outcomes for accountability apply to all children and reflect global functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met) • IFSP outcomes are specific to an individual child, based on his or her individual needs

What is Assessment? “Early childhood assessment is a flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical and mental health services needs of young children and their families.” (Bagnato and Neisworth, 1991) Quoted in DEC Recommended Practices, 2005

Assessment is… “Assessment is a generic term that refers to the process of gathering information for decision-making.” Mary McLean, 2004

Purposes of Assessment In the early intervention system assessment serves several purposes: • Identification of children with special needs (including screening, diagnosis, and determination of eligibility for services) • Planning instruction or intervention • Monitoring of the child’s level of knowledge or progress • Program evaluation and accountability

Accountability??? • Accountability involves embedding ongoing data collection into program practices because the need to examine the effectiveness of the services being provided is ongoing • Given the number of factors that influence the provision of services (e.g., staff changes, reimbursement formulas), it is not reasonable to assume that a specific program that was effective at one point in time will always be effective

Assessment for Accountability • Policymakers and funders are raising legitimate questions about which programs are effective as they struggle with how to allocate resources • The Government Performance and Results Act (GPRA) means Part C Programs need to provide evidence that program participants demonstrate positive outcomes • An important issue for assessment is that the Office for Special Education Programs (OSEP) decided not to require states to use the same assessment tool. Each state is free to chose their own

Assessment as a Process It is important to distinguish between assessment as a process and assessment as a single instrument or test! • Assessment as a process involves multiple ways of collecting information about a child’s performance, including, for example, observation, documentation of the child’s work, checklists, rating scales, portfolios and norm-referenced or curriculum based tests

Variation in What Tools are Used So far these are the tools we know that states are using for accountability: • Of 28 states who listed specific assessment instruments: • HELP-15 states • BDI/BDI-2-13 states • AEPS-11 states • Creative Curriculum-6 states • ELAP-6 states • Not yet determined-23 states

Variation in HowAssessments Have Been Used In the past, there has been much variation in assessment procedures in early intervention programs across the country: • In some programs, the only formal assessment has been for eligibility • Other states completed assessment annually • Some states have always completed assessment at exit

Views on Assessment There have been major changes in last 15 years in how assessment of young children is viewed by the early childhood community: • Old position: Do not test little kids • New position: Ongoing assessment is part of a high quality early childhood program

New Thoughts on Assessment “The best way to understand the development of children is to observe their behavior in natural settings while they are interacting with familiar adults over prolonged periods of time” --Uri Bronfenbrenner

What Changed our Thinking The purpose of assessment was redefined in the last decade: • It’s no longer about sorting, labeling, using to deny access to services • It’s now about getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills • It’s now about monitoring progress • New and different tools became available for general early childhood: Curriculum-based assessments were developed, e.g., Creative Curriculum, Work Sampling, etc.

What Changed Our Thinking • Assessment had always been seen as a process with multiple purposes • Distinctions have been made between good and bad uses of assessment with young children • Good assessment practices are now widely recognized as a key component of high-quality early childhood programs • Good uses are now promoted! • For more information: National Association for Education of Young Children (NAEYC) web site (Position statement on Curriculum, Assessment and Evaluation)

NAEYC Position Statement Policymakers, early childhood professionals and other stakeholders in young children’s lives have a shared responsibility to: • Make ethical, appropriate, valid and reliable assessment a central part of all early childhood programs. • To assess young children’s strengths, progress, and needs, use assessment methods that are • developmentally appropriate • culturally and linguistically responsive • tied to children’s daily activities • supported by professional development • inclusive of families

NAEYC Position Statement continued… Connected to specific, beneficial purposes: • Making sound decisions about teaching and learning • Identifying significant concerns that may require focused intervention for individual children • Helping programs improve their educational and developmental interventions

DEC Recommended Practices for Assessment The Recommended Practices emerged from focus groups and are supported by literature from the Division of Early Childhood (DEC): • Parents and professionals who participated in these focus groups identified repeatedly two concerns • Principle stakeholders, parents and family members must play a vital and indispensible role in assessment from beginning to end • Assessment methods and materials must accommodated children’s developmental and disability specific characteristics

DEC Recommended Practices for Assessment The Recommended Practices are organized around five statements: • Professionals and families collaborate in planning and implementing assessment • Assessment is individualized and appropriate for the child and family • Assessment provides useful information for intervention • Professionals share information in respectful and useful ways • Professionals meet legal and procedural requirements and Recommended Practice guidelines

Assessment for Accountability The assessment process we use to determine eligibility and program planning can be used for accountability! The three child outcomes: • Children have positive social-emotional skills (including social relationships) • Children acquire and use knowledge and skills (including early language/communication [and early literacy]) • Children use appropriate behaviors to meet their needs • Assessment tools can inform us about children’s functioning in each of the three outcome areas • Challenge: There is no assessment tool that assesses the three outcomes directly

Additional Challenges • Young children are unreliable test takers • Because of their limited ability to understand and follow the directions of the test • They may not be able to control their behavior • They have short attention spans • And are easily distracted • What young children demonstrate in a test situation may not be an accurate representation of what they are capable of doing on a daily basis

Types of Tests • Standardized vs curriculum based • By definition, standardized tests must be administered the same way to all test takers. • The use of a consistent approach during test administration enhances reliability between administrators and confidence that children are responding to the same tasks. • Standardized tests usually provide normative data to compare the child’s functioning with that of other children of the same age.

Standardized Tools Few standardized assessment tools provide detailed guidelines for appropriate accommodations during testing to effectively measure capabilities among the broad range of children who have special needs.” McLean, 2005

Curriculum-based Assessments • Bagnato and Neisworth (1991) define curriculum-based assessment as a “form of criterion-referenced assessment wherein curricular objectives act as the criteria for the identification of instructional targets and for the assessment of status and progress” • Some of the newer assessment tools emphasize critical skills to facilitate the development of fuctional behavior

Currently Available Assessment Tools • Most assessment tools are organized around developmental domains and are based on sequences of skills that build on each other • This organization may pose difficulties for accurate assessments of functioning in young children with disabilities because delayed skills in one domain can influence assessment of or functioning in other domains • Each assessment tool sees children through its own lens • Each lens is slightly different • Lens are not right or wrong

Variations Among States • States are continuing to make key decisions related to their accountability systems, so any information about state choices must be seen as time-limited and interpreted cautiously • In many state, some assessment-related decisions are being made by local programs, so there is variation within the states • The information available thus far suggests that much of the data on child outcomes will be collected with curriculum-based tools

Variations Among States • Most states are building their accountability systems around assessments that were already being administered for other purposes (such as eligibility determination or program planning) • Few have adopted a new statewide measure solely for collecting data for accountability purposes

Maryland’s Top Five • Following a survey distributed to all local Program Directors in the state, Maryland state Department of Education (MSDE) identified the 5 most commonly used assessment instruments in Maryland. • Results from at least one of these assessment instruments (or other instrument if more appropriate), in conjunction with information obtained from other possible sources, including record review, interview, observation, and screening, will be used in providing the information required to complete the Child Outcomes Summary Form (COSF). • Ideally, the same tool would be used at entry and exit. However, use the tool that is most appropriate for the child.

Maryland’s Top Five • The five most common assessment measures are: • Early Learning Accomplishment Profile (E-LAP) • Early Intervention Developmental Profile (EIDP) • Hawaii Early Learning Profile (HELP) • Battelle Developmental Inventory II • Ages and Stages

Key Questions for Accountability: • How much and what information will the tool provide about the child’s current level of functioning and the attainment of the 3 child outcomes? • Would providers intentionally or unintentionally give children higher scores than warranted, knowing that conclusions about program effectiveness would be based on the information?

Assessment Tool Lens • Each assessment tool carries its own organizing framework • Most are organized around domains but children integrate skills when performing functional activities • No right or wrong lens: • How can what the assessment tool provides be used to develop functional outcomes for young children? • How can what the assessment tool provides be used as information about the achievement of the three outcomes?

Critical Assumptions Related to the 3 Child Outcomes • Achievement of the outcomes is age based, children of different ages will demonstrate achievement in different ways • There are many pathways to competence for children with atypical development (using sign language, wheelchair). This seems obvious but can get lost when an assessment tool uses a different lens.

Remember This • Flexibility is required in using assessment tool results to make decisions about a child’s functioning on the outcomes • Team needs to decide what information from an assessment tool is relevant to understanding child’s functioning

More DEC Recommended Practices for Assessment Assessment provides useful information for intervention! • Assessment results are immediately useful for planning program goals and objectives • Good assessment practices assess not only immediate mastery of skill, but also whether child can demonstrate skill consistently across settings and with other people • Good assessment practices also assess the presence and extent of atypical child behavior that may be a barrier to intervention and progress

Using the Assessment Results for Outcomes What is an Outcome: • Outcomes are the benefits that children and families experience as a result of early intervention • The three child outcomes, as measured for accountability purposes are different than IFSP outcomes in that: • The three child outcomes reflect global functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met) • IFSP outcomes are specific to an individual child, based on his or her individual needs

Let’s Look at Using Assessment Information to Develop Functional IFSP Outcomes What Makes an Outcome Functional? • Functional Outcomes: • Refer to things that are meaningful to the child in the context of everyday living • Refer to an integrated series of behaviors of skills that allows for the child to achieve the important everyday goals

Meaningful Outcomes Meaningful outcomes promote a child’s functioning in three key foundations of early development - social interaction, mastery over environment, and engagement in learning - in ways specific to each family and child (McWilliam, 2002; National Research Council, 2000).

Writing IFSP Family Centered, Measureable, Functional Outcomes • Family-centered outcomes are written to address: • The desires and goals a family has for them and their child • Created with the family after assessing what activities are meaningful to individual family members. • Includes not only activities of daily living but also the family's ability to participate in cultural and social experiences they hold valuable

Functional Outcomes • “Function” refers to those activities identified by the family that support the development of the child’s physical, social and psychological well being, i.e. the ability to feed oneself, to hold a toy, to communicate, or to play • Early intervention teams do not “treat” the primary diagnosis, i.e. cerebral palsy, autism, but instead treat the functional delay or disability

Determining Appropriate IFSP Functional Outcomes A few questions to consider: • What are the activities the family would like to do that are difficult? • Has the family given up on any family outings, experiences or chores? • What kinds of things would the family like for the child to do that would make life easier for them or more fun for the child? Read more at MD’s IFSP Tutorial: Assisting Families to Select Functional Outcomes

Measureable IFSP Outcomes • Outcomes should be written so that all members of the team will understand when the outcome has been met. • Outcomes should address one of the following areas: • They should enhance the family’s ability to care for or to engage in activity with their child • They should enhance the child’s ability to participate in functional activities (feeding, dressing, moving in his environment, communicating, playing, etc.) • They should expand on activity settings in which the child already participates Read more at MD’s IFSP Tutorial: Measurable Criteria: How we Know we have Achieved Outcomes

Effective Strategies on an IFSP Identify How Functional Outcomes Will be Achieved Each strategy should address: • Individualization- Do strategies build on child and family interests? • Context- Do strategies build on familiar places, people, and routines? • Mastery- Do strategies ensure generalization of a child’s actions and interactions in multiple settings and tasks? • Collaboration- Do strategies specify who will do what? Read more at MD’s IFSP Tutorial: Effective Strategies Identify How Functional Outcomes will be Achieved

Making a Difference! The results of improved assessment and individual IFSP outcomes development and services is two-fold: • First and foremost, it ultimately leads to enhanced services for children and families • Which, as a result, can’t help but positively impact Maryland’s outcome measurement results!

More to Come… Next week: • Explore Maryland’s approach to gathering child outcome data to measure OSEP’s Child Outcomes • The role of the Child Outcomes Summary Form • Explore other state’s approaches

Measuring Child and Family Outcomes