The family outcomes survey revisions data uses
This presentation is the property of its rightful owner.
Sponsored Links
1 / 55

The Family Outcomes Survey: Revisions, Data, Uses PowerPoint PPT Presentation


  • 60 Views
  • Uploaded on
  • Presentation posted in: General

The Family Outcomes Survey: Revisions, Data, Uses. Measuring Child and Family Outcomes National TA Meeting Baltimore, MD August 26, 2007. Don Bailey, RTI International Robin Nelson, Texas Part C Program Nyle Robinson, Illinois Part C Program Chelsea Guillen, Illinois Part C Program

Download Presentation

The Family Outcomes Survey: Revisions, Data, Uses

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The family outcomes survey revisions data uses

The Family Outcomes Survey:Revisions, Data, Uses

Measuring Child and Family Outcomes National TA Meeting

Baltimore, MD

August 26, 2007

Don Bailey, RTI International

Robin Nelson, Texas Part C Program

Nyle Robinson, Illinois Part C Program

Chelsea Guillen, Illinois Part C Program

Melissa Raspa, RTI International


Goals for today

Remind everyone of the goals and format of the Family Outcomes Survey

Describe the revisions made to the survey this year and the rationale

Summarize selected data from an initial pilot study using the scale in Illinois and Texas

Describe plans to validate the scale

Discuss ways the scale could be used to help states answer questions of interest

Goals for today


Eco family outcomes

Understand their child’s strengths, abilities, and special needs

Know their rights and advocate effectively for their children

Help their children develop and learn

Have support systems

Access desired services, programs, activities in their community

ECO Family Outcomes


Eco scale family outcomes survey

ECO Scale:Family Outcomes Survey

  • Three items for each of the five proposed family outcomes

  • Each item rated on a scale from 1-7

  • Descriptive statements for ratings of 1, 3, 5, 7

  • Blanks left for “in between” ratings of 2, 4, 6

  • Three additional items for states to use in responding to APR requirements (two versions, one for Part C, one for Part B)


The family outcomes survey revisions data uses

  • UNDERSTANDING YOUR CHILD'S STRENGTHS, ABILITIES, AND SPECIAL NEEDS

  • Your child is growing and learning. How much does your family understand about your child’s development?

2. Some children have special health needs, a disability, or are delayed in their development. These are often referred to as “special needs.”

How familiar is your family with your child's special needs?

3. Professionals who work with you and your child want to know if the things they do are working. How often is your family able to tell

if your child is making progress?


The family outcomes survey revisions data uses

THE HELPFULNESS OF EARLY INTERVENTION

The next questions ask how well early intervention has helped your family. When answering, think about the early intervention services you have received.

16.To what extent has early intervention helped your family know and understand your rights?

17.To what extent has early intervention helped your family effectively communicate your child’s needs?

18.To what extent has early intervention helped your family be able to help your child develop and learn?


What have we done in the past year

What have we done in the past year?

  • States have begun to use the instrument

  • Conducted a pilot study in Illinois and Texas (summer, 2006)

  • Reviewed and revised the scale (fall, 2006)

  • Submitted application to IES to validate the scale (summer, 2007)

  • Completed cognitive testing of scale with Spanish speaking families (summer, 2007)

  • Currently conducting a second, larger pilot study in Illinois and Texas (summer, 2007)


Questions addressed in pilot study

Questions Addressed in Pilot Study

  • How should the survey be distributed and what response rates might be expected?

  • What distribution of responses is seen for each item?

  • Are respondents more likely to pick the odd numbered items?

  • How did responses on the APR items compare with the outcome items?

  • Did patterns of responses differ across the states?

  • Did ratings of outcomes vary as a function of language of the survey (English vs. Spanish?)

  • What is the internal reliability of the instrument?


Methodological similarities

Methodological Similarities

  • Identical survey

  • Similar accompanying materials

  • Distribution method almost identical

  • Identical return mail methods

  • Timeframe was almost the same


Differences in approach

Differences in Approach

  • Texas used sampling

  • Illinois surveyed all programs statewide

  • Illinois found that IFSPs near 3 were often extended, new meetings not held

    • So, Illinois did not distribute surveys within 3 months of age 3 during the pilot

  • Texas did more follow-up, utilized replacement, Illinois depended on volume


Sampling distribution return rates

Sampling, Distribution, Return Rates

  • Illinois

    • Did not sample

    • Distributed 1,976 surveys at 6-month and/or annual reviews

  • Texas

    • Stratified programs by region and size

    • Randomly selected 3 programs from 7 regions

    • Randomly selected 1262 families

  • Return rates

    • Texas – 44%, representative

    • Illinois – 29%, not as representative

  • Data presented today represent 890 surveys (~100 Spanish)


Additional construct looking towards the future

Additional Construct: Looking Towards the Future

  • Both states added three additional items to the survey to assess:

    • Current life situation

    • Transition preparation

    • Future life situation


Understanding your child s strengths abilities special needs

Understanding Your Child’s Strengths, Abilities & Special Needs


Understanding your child s strengths abilities special needs1

Understanding Your Child’s Strengths, Abilities & Special Needs


Understanding your child s strengths abilities special needs2

Understanding Your Child’s Strengths, Abilities & Special Needs


Knowing your rights advocating for your child

Knowing Your Rights & AdvocatingFor Your Child


Knowing your rights advocating for your child1

Knowing Your Rights & AdvocatingFor Your Child


Knowing your rights advocating for your child2

Knowing Your Rights & AdvocatingFor Your Child


Helping your child develop learn

Helping Your Child Develop & Learn


Helping your child develop learn1

Helping Your Child Develop & Learn


Helping your child develop learn2

Helping Your Child Develop & Learn


Having support systems

Having Support Systems


Having support systems1

Having Support Systems


Having support systems2

Having Support Systems


Accessing your community

Accessing Your Community


Accessing your community1

Accessing Your Community


Accessing your community2

Accessing Your Community


Findings on range of responses

Findings on Range of Responses

  • The items generated a range of responses

  • We are not getting a normal distribution at the item level, but what kind of distribution would you expect?

  • Ratings vary across items, suggesting that parents respond differentially and specifically to each item

  • 77% of responses used the odd-numbered ratings (the ones with descriptors)


Knowing and understanding rights

Knowing and Understanding Rights


Knowing and understanding rights1

Knowing and Understanding Rights


Helping your child develop and learn

Helping your Child Develop and Learn


Helping your child develop and learn1

Helping your Child Develop and Learn


Summary comments and analyses of apr and outcome items

Summary Comments and Analyses of APR and Outcome Items

  • The two sets of items do not generate equivalent response patterns

  • APR items more likely to have ratings of 6 and 7

  • Correlations are moderate

    • .58 knowing rights

    • .44 helping child develop and learn

    • .26 communicate needs


Other findings

Other Findings

  • IL & TX had almost identical mean scores across items, although patterns across some items differed

  • Same highest (EI help child dev & learn) and lowest (participation in activities) rated items

  • High internal consistency for total scale and OSEP items (Cronbach’s alpha > .90)

  • Factor analysis yields 1-2 factors

  • Means for Spanish-speakers considerably lower than English-speakers on most items


Specific question comparisons

Specific Question Comparisons

  • IL slightly higher on “access to medical care”

    • Better, more inclusive Medicaid system

  • TX slightly higher on “knowledge of rights”

    • Both states have strong emphasis; does service model make a difference?

  • IL lower on “transition preparation”

    • Already aware of the problem, IL has put considerable emphasis on improving transition


Example effects of primary language english vs spanish

Example: Effects of Primary Language (English vs. Spanish)


Example effects of primary language english vs spanish1

Example: Effects of Primary Language (English vs. Spanish)


Value of comparisons

Value of Comparisons

  • Does service model make a difference?

    • Texas SC has more contact with family

  • Did Illinois exclusion of kids nearest exit have an impact?

    • Families were receiving services longer

  • Effect of demographics?

    • Culture, acculturation


Demographic comparisons

Illinois

55% White

24% Hispanic

17% Black

3% Asian

13% primary language Spanish

60% Medicaid

Texas

38% White

47% Hispanic

12% Black

2% Asian

19% primary language Spanish

61% Medicaid

Demographic Comparisons


Scale revision

Scale Revision

  • Systematic review by survey methodologists

  • Cognitive testing with 12 parents


Results from scale analysis

Results from Scale Analysis

  • Problems identified with unclear meanings and inconsistencies across items

  • MUCH back and forth between methodologists and IL/TX folks about specific wording, resulting in significant revision (rewording only, same format and # items

  • Revised instrument now available on ECO web site


Consistent set of descriptors used at each level

Consistent Set of Descriptors Used at Each Level

  • 7Great deal, almost always, very,

    almost all, excellent

  • 5Usually, good amount, generally,

    many, good

  • 3Some, sometimes, somewhat, fair

  • 1Just beginning, seldom, few, poor


Changes in apr item modifiers

Old Version

1 - EI has not helped…

3 - EI has done a few things…

5 - EI has provided good help…

7 – EI has provided excellent help…

Current Version

1 – EI has done a poor job…

3 – EI has done a fair job

5 – EI has done a good job

7 - EI has done an excellent job

Changes in APR Item Modifiers


Ies application validating the family outcomes survey

IES Application: Validating the Family Outcomes Survey

  • Specific Aim 1: Determine the test-retest reliability of the FOS by examining stability of scores over a short (two-week) period of time.

  • Specific Aim 2: Establish the criterion validity of scores on the FOS by examining its relationship with six well-established criterion measures of family well being, addressing family empowerment, social support, parenting self-efficacy, quality of life, hope, and parenting stress.

  • Specific Aim 3: Determine sensitivity to change in scores on the FOS by comparing changes in family outcomes over 18-months to changes in scores on the six criterion measures.

  • Specific Aim 4: Document the convergent validity of scores on the FOS by examining the relationship between changes in family outcomes over 18 months and the extent to which early intervention programs used recommended practices in providing services to families.


Using the family outcomes survey what are the possibilities

Using the Family Outcomes Survey:What are the possibilities?

  • Using the scale to describe and classify outcomes attained by families

  • Using the scale to identify child, family, or program variables associated with outcome attainment

  • Using the scale to improve programs at the state or local level

  • Using the scale to work with individual families


Data uses

Data Uses

  • How can states use the data from the Family Outcomes Survey to help identify both the “success stories” of early intervention and the families who may need more immediate help?

  • How do states track the progress of families in early intervention?

  • How can states examine the relationship between the quality and delivery of services and family outcomes?

  • Are family outcomes related to child outcomes?


Data uses1

Data Uses

  • Three areas of analysis

    • Measurement-related analyses

      • Internal consistency

      • Factor structure

    • Descriptive analyses

      • Basic descriptive statistics

      • Grouping families

    • Predictive analyses

      • Describing variation between families

      • Predicting variation between families


Measurement related analyses

Measurement-Related Analyses

  • Internal consistency

    • Total scale

    • Five subscales

  • Factor Structure

    • One factor

    • Multiple factors


Descriptive analyses

Descriptive Analyses

  • Basic descriptive statistics

    • Mean, min, max, standard deviation

  • Threshold scores

    • 5 or higher = outcome attained

    • 3 or lower = “score of concern”

  • Score variation

    • Most variable respondent

    • Least variable respondent


Descriptive analyses1

Descriptive Analyses

  • Grouping families with similar response patterns

    • Empirically derived

      • Success stories (scores of 5 or higher)

      • Borderline (scores consistently 3 to 5)

      • Immediate concern (scores 3 or below)

    • Statistically derived

      • Clusters of families with similar response patterns


Predictive analyses

Predictive Analyses

  • Two types of analyses: describing variation and predicting variation

  • Three types of “predictor variables”

    • Child and family variables (e.g., health status, eligibility, ethnicity, income, mother’s education)

    • Service quality variables (e.g., family-centered practices)

    • Service delivery or models (e.g., service coordination model, amount of services, location of services)


Predictive analyses1

Predictive Analyses

  • Describing variation between families

    • Do outcomes vary by child and family characteristics?

    • Does the quality of early intervention services relate to family outcomes?

    • Are service delivery models associated with family outcomes?


Predictive analyses2

Predictive Analyses

  • Predicting variation between families

    • Do families of children who just entered early intervention have lower family outcomes than families whose children have been receiving services for a longer period of time?

    • Is a dedicated and not independent service coordination model associated with better family outcomes?


Family outcomes vs child outcomes

Family Outcomes vs. Child Outcomes


Discussion

Discussion


  • Login