Understanding value added
This presentation is the property of its rightful owner.
Sponsored Links
1 / 35

Understanding Value-Added PowerPoint PPT Presentation


  • 131 Views
  • Uploaded on
  • Presentation posted in: General

Understanding Value-Added. Lesson 2: How Value-Added Works. What is the Value-Added Metric?. Academic Growth = Student Learning. 2009. 2010. Value-Added is the District’s measure of elementary school growth. Value-Added is a nationally recognized way of measuring growth.

Download Presentation

Understanding Value-Added

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Understanding value added

Understanding Value-Added

Lesson 2: How Value-Added Works


What is the value added metric

What is the Value-Added Metric?

Academic Growth = Student Learning

2009

2010

  • Value-Added is the District’s measure of elementary school growth.

  • Value-Added is a nationally recognized way of measuring growth.

  • Emphasizes continual student improvement

  • Provides information to understand what drives continual improvement


Measuring growth not attainment

Measuring Growth, Not Attainment

200

200

210

210

220

220

230

230

240

240

250

250

260

260

270

270

280

280

290

290

300

300

In this school, the percent meeting state standards is 25% in both Year 1 and Year 2.

Attainment is unchanged – but are students learning?

Analyzing growth provides this information

(Year 2)

ISAT Scale Score

(Year 1)


Accounting for student populations

Accounting for Student Populations

  • Student academic growth varies by grade, prior performance, and demographics.

  • The goal of the Value-Added metric is to measure the school’s impacton student learning independent of student demographic factors.

  • Value-Added accounts for the following student factors:

    • Controlling for the factors above gives proper credit for growth to low attainment schools and schools that serve unique populations.


How it works

How it Works

  • Value-Added is not a comparison to similar schools.

    • We do not look for a comparison group of schools that match each other on all 9 student factors…such a group might not exist.

  • Rather, Value-Added compares growth of students in each school to growth of students across the District, controlling for the list of student factors.

  • To do this, we utilize a regression methodology, developed in collaboration between CPS and academic experts from the University of Wisconsin.


  • Regression lines

    Regression Lines

    Regression shows how growth relates to another variable—in this case prior performance on the ISAT.

    This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.

    It is downward-sloping because at higher levels of prior performance, average growth is smaller.

    25

    Repeat the process for each grade level.

    3rd to 4th Grade ISAT Test Scores Only

    All ISAT Math Scores for the District

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    375

    125

    225

    275

    325

    2009 ISAT Score


    Regression lines1

    Regression Lines

    Regression shows how growth relates to another variable—in this case prior performance on the ISAT.

    This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.

    It is downward-sloping because at higher levels of prior performance, average growth is smaller.

    25

    Repeat the process for each grade level.

    3rd to 4th Grade ISAT Test Scores Only

    All ISAT Math Scores for the District

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    375

    125

    225

    275

    325

    2009 ISAT Score


    Regression lines2

    Regression Lines

    Regression shows how growth relates to another variable—in this case prior performance on the ISAT.

    This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.

    It is downward-sloping because at higher levels of prior performance, average growth is smaller.

    25

    Repeat the process for each grade level.

    3rd to 4th Grade ISAT Test Scores Only

    All ISAT Math Scores for the District

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    375

    125

    225

    275

    325

    2009 ISAT Score


    Regression lines3

    Regression Lines

    Regression shows how growth relates to another variable—in this case prior performance on the ISAT.

    This line shows the average gain on ISAT math between 2009 and 2010 for 4th graders.

    It is downward-sloping because at higher levels of prior performance, average growth is smaller.

    25

    Repeat the process for each grade level.

    3rd to 4th Grade ISAT Test Scores Only

    All ISAT Math Scores for the District

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    375

    125

    225

    275

    325

    2009 ISAT Score


    Controlling for one variable

    Controlling for One Variable

    Gain of 4th to 5th Grade students at a single school controlling for prior performance compared to the District average

    This student grew faster than other 5th grade students with the same prior ISAT score.

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    This student grew slower.

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    The orange line shows the average gain of non-ELL students between 4th and 5th grade.

    The blue line shows the average gain for ELL students between 4th and 5th grade.

    This line shows the average gain for all students from 4th to 5th grade.

    Now we identify which students are English Language Learners

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables1

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    The orange line shows the average gain of non-ELL students between 4th and 5th grade.

    The blue line shows the average gain for ELL students between 4th and 5th grade.

    This line shows the average gain for all students from 4th to 5th grade.

    Now we identify which students are English Language Learners

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables2

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    The orange line shows the average gain of non-ELL students between 4th and 5th grade.

    The blue line shows the average gain for ELL students between 4th and 5th grade.

    This line shows the average gain for all students from 4th to 5th grade.

    Now we identify which students are English Language Learners

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables3

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    The orange line shows the average gain of non-ELL students between 4th and 5th grade.

    The blue line shows the average gain for ELL students between 4th and 5th grade.

    This line shows the average gain for all students from 4th to 5th grade.

    Now we identify which students are English Language Learners

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables4

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    The orange line shows the average gain of non-ELL students between 4th and 5th grade.

    The blue line shows the average gain for ELL students between 4th and 5th grade.

    This line shows the average gain for all students from 4th to 5th grade.

    Now we identify which students are English Language Learners

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for multiple variables5

    Controlling for Multiple Variables

    Regression allows us to control for multiple factors at one time – in this case prior performance and ELL status.

    Although this student grew slower than other 5th graders with the same pretest score, she grew faster than other ELL students with the same pretest score.

    25

    20

    15

    Scale Score Gain, 2009 to 2010

    10

    5

    0

    175

    150

    200

    225

    250

    2009 ISAT Score


    Controlling for many variables at once

    Controlling for Many Variables at Once

    Now, we can control for other factors besides prior performance for Student A.

    Based on Student A’s demographics, adjustments are made

    Compared to similar students district-wide, Student A has above average gain

    Scale Score Gain, 2009 to 2010

    2009 ISAT Score


    Controlling for many variables at once1

    Controlling for Many Variables at Once

    Now, we can control for other factors besides prior performance for Student A.

    Based on Student A’s demographics, adjustments are made

    Compared to similar students district-wide, Student A has above average gain

    Scale Score Gain, 2009 to 2010

    2009 ISAT Score


    Controlling for many variables at once2

    Controlling for Many Variables at Once

    Now, we can control for other factors besides prior performance for Student A.

    Based on Student A’s demographics, adjustments are made

    Compared to similar students district-wide, Student A has above average gain

    Scale Score Gain, 2009 to 2010

    2009 ISAT Score


    Summary of regression

    Summary of Regression

    • By measuring the impact of each student factor, the regression model isolates the impact of the school on student growth.

    • In other words, some growth is explained by external factors. We can measure the average impact of these external factors on growth at the District level and subtract that impact from the school’s absolute growth.

    • The growth that is left over after removing the impact of these factors is attributed to the school. This is the value added by the school.


    Oak tree analogy

    Oak Tree Analogy

    • For an illustrative example of regression, view the “Oak Tree Analogy” presentation at:

      http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/

    • The Oak Tree presentation illustrates the Value-Added model by using an analogy of two gardeners tending to oak trees.


    Some things to know

    Some Things to Know

    • Tested Students

      • All students making normal grade progression who took ISAT in both the previous year and current year are included in analysis.

    • Mobile Students

      • Mobile students count towards the Value-Added score in each school they attended, but are weighted in the analysis by the amount of time they were in the school during the year.

    • English Language Learners

      • ELL students in Program Years 0 through 5 are excluded from the analysis.

      • This includes students who were in PY0-5 during the pretest year, even if they have since exited the ELL program or moved to PY6.

    • Students with Disabilities

      • IEP status is differentiated by type of IEP.

      • For example, the impact of a severe and profound disability is considered separately from the impact of a speech and language disability.


    Value added scores

    Value-Added Scores

    Value-Added measures the difference between the growth of students at a school and the growth of similar students across the District.


    Standardization of scores

    Standardization of Scores

    200

    210

    220

    240

    230

    • Growth on the ISAT is measured in ISAT scale score points

    • However, one ISAT scale score point of growth is more difficult to obtain in some grade levels than others.

    • As a result, standardization is used to ensure that all Value-Added scores are on the same scale.

    Student A “grew” by 35 ISAT scale score points


    Standardization of scores1

    Standardization of Scores

    • Standardization is a common statistical process. In this case, it is used to convert ISAT scale score points to a standard scale.

    • The unit of measure is the “standard deviation” which is a measure of distance from the mean.

      • i.e., how much does School A’s score deviate from the mean?

  • This places all scores on the same scale, allowing for more precise comparisons between scores at different grade levels.


  • The standard scale

    The Standard Scale

    • Features of the Standard Scale

    • The scale ranges from approximately -6 to 6.

    • Zero (0) is the District average.

    • About 68% of scores fall between -1 and 1.

    • About 95% of scores fall between -2 and 2.

    • About 99% of scores fall between -3 and 3.

    • Only about 1% of scores are less than -3 or more than 3.

    34%

    34%

    2.5%

    2.5%

    13.5%

    13.5%


    Reading the value added reports

    Reading the Value-Added Reports

    Percentile: This is the percent of scores that fall below this score. Percentiles range from 0th to 99th

    Value-Added Score

    Performance Category: This is based on the percentile.

    Confidence Interval: This is explained in the next set of slides.

    Number of Students in the calculation: This is weighted by the amount of time students were in the school between the pretest and posttest.


    Confidence intervals

    Confidence Intervals

    • The Value-Added model controls for factors that CPS can measure, but there are some factors that cannot be measured, such as:

      • Motivation to learn

      • Family circumstances

      • Health

  • In addition, the Value-Added model is a statistical estimation of the school’s impact on student learning and therefore contains a certain amount of random error.

  • For these reasons, the Value-Added model includes confidence intervals.


  • Real world example political polling

    Real World Example: Political Polling

    • A Political Polling company surveys a representative random sample of 1,000 community households about for whom they are going to vote on Election Day. The question they pose is:

    • If the election were held today, for whom would you cast your ballot?

    • The percentages of responses breakdown as follows:

      • Candidate Jones would receive 54% of the vote

      • Candidate Smith would receive 46% of the vote

      • There is a +/- 3% margin of error


    Confidence intervals in political polling

    Confidence Intervals in Political Polling

    With the margin of error of +/- 3%, the range of the percentage of people who plan on voting for each candidates is as follows:

    Candidate Jones would receive between 51% and 57% of the vote.

    43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54%55% 56% 57%

    Candidate Smith would receive between 43% and 49% of the vote.

    The confidence intervals do not overlap. Therefore the race is NOT “too close to call.” We can predict with a high degree of confidence that Candidate Jones will win the race.


    Confidence intervals in value added

    Confidence Intervals in Value-Added

    • A confidence interval is a range of scores around the Value-Added estimate.

    • We are 95% confident that the true Value-Added score falls within the confidence interval range.

    • The confidence interval is “n” dependent, meaning larger samples yield smaller confidence intervals.

      • This is because in larger samples, a score that is different from the average is less likely to be due to random error alone.

    1.0

    • The Value-Added estimate is 1.0.

    • The confidence interval is ± 0.3.

    • The confidence interval range is from 0.7 to 1.3.

    Example:

    0.7

    1.3


    Statistical significance

    Statistical Significance

    • If the confidence interval does not include zero, we say that the score is statistically significant, meaning we are 95% confident that the score is different from zero.

    • A color is associated with each score based on the statistical significance:


    How confidence intervals are reported

    How Confidence Intervals are Reported

    This is how Value-Added scores are displayed in the reports.

    This school has a Value-Added score of -0.5 in reading

    (the score is ½ of a standard deviation below the mean)

    • The confidence interval ranges from -1.9 to 0.8

    • Because the confidence interval includes zero, we say that this school is not statistically different from zero at the 95% confidence level.

    • For that reason, the bubble is yellow.


    Using value added information

    Using Value-Added Information

    • Performance Management

      • As an assessment of school performance

      • To identify areas needing additional support or professional development

      • To identify best practice strategies for improving student growth

    • School Accountability (i.e., Performance Policy)

    • Additional Compensation Plans (i.e. Chicago TAP)

    In all of these applications, Value-Added is used as just one additional piece of information, along with other data.


    For more information

    For More Information

    • More lessons and other resources for understanding Value-Added are available at:

      http://research.cps.k12.il.us/cps/accountweb/Research/ValueAdded/

      • Lesson 2 (Part 2): Oak Tree Analogy

      • Lesson 3: Technical Specifications of the Value-Added Regression Model


  • Login