Effect Size Lens on
This presentation is the property of its rightful owner.
Sponsored Links
1 / 56

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008 PowerPoint PPT Presentation


  • 42 Views
  • Uploaded on
  • Presentation posted in: General

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008. Presentation by Johnny Wilson, Charles Lanier, and Mike Gallagher, all from the Accountability Services Dept., NCDPI. Purposes – Overall Session.

Download Presentation

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Effect size lens on local sat and state math test results nc accountability conference february 2008

Effect Size Lens on Local SAT and State Math Test Results NCAccountability ConferenceFebruary 2008

  • Presentation by Johnny Wilson, Charles Lanier, and Mike Gallagher, all from the Accountability Services Dept., NCDPI


Purposes overall session

Purposes – Overall Session

  • Provide background and access paths to important data. (important = helpful)

  • Show how LEAs and schools can access and compare test results across EOG, EOC and SAT.

  • Entice folks to exploit the usefulness of such comparisons.


Who s on first

Who’s on First?

  • Johnny Wilson, Reporting Section, SAT Data Report. [email protected]

  • Charles Lanier, Test Development Section, State Testing Data in the Green Book. [email protected]

  • Mike Gallagher, Test Development, Effect Size Look at SAT and State Test Results. [email protected]


Purposes part 1 sat

Purposes – Part 1: SAT

  • Get acquainted with the major trends in SAT results for North Carolina.

  • Look at school level variability of SAT results from year-to-year.

  • Show the access path for information at the school and LEA levels.


Sat math score trends

SAT Math Score Trends

Between 1997 and 2007, the Math gap between NC and the Nation narrowed from 23 pts. to 6 pts.

NC’s Math scores fell in 2007 after more than a decade of gains.


Sat math score trends1

SAT Math Score Trends


Sat verbal score trends

SAT Verbal Score Trends

By comparison, NC’s Verbal scores have been up and down over the same period.

Between 1997 and 2007, the Math gap between NC and the Nation has narrowed from 15 pts. to 7 pts.


Sat verbal score trends1

SAT Verbal Score Trends


Sat total score trends for nc and the nation

SAT Total Score Trends for NC and the Nation

  • After 10 consecutive years of gains,

  • NC’s mean total SAT scores fell in

  • 2006 and 2007.

  • Gap between NC’s mean total score

  • and the nation’s score has narrowed

  • from 53 in 1990 to 13 points in

  • 2007.


Total score trends for nc and the nation

Total Score Trends for NC and the Nation


Nc s schools with declining sat scores

NC’s Schools with Declining SAT Scores

At the school level, math scores also declined in 2006 & 2007, compared with the previous year.

Notably, fewer schools increased math scores in 2007 than in 2006.


Percent of nc s schools with declining sat scores

Percent of NC’s Schools with Declining SAT Scores


Changes in sat math scores

Changes in SAT Math Scores

Most changes in SAT scores from year to year are not unusual.

Low-Volume Schools tend to have larger changes.

The larger the test-taking population, the smaller changes in SAT scores tend to be.


Changes in sat math scores by test taking volume us

Changes in SAT Math Scores by Test-taking Volume (US)


Changes in sat math scores by test taking volume nc

Changes in SAT Math Scores by Test-taking Volume (NC)


Changes in sat math scores all schools us

Changes in SAT Math Scores All Schools (US)


Changes in sat math scores all schools nc

Changes in SAT Math Scores All Schools (NC)


Factors that might influence score fluctuations

Factors that Might Influence Score Fluctuations

Fatigue Factor

SAT fee increase in 2005 resulted in decline in repeat test-takers.

Difficulty of the test


Nc sat math scores by race

NC’s Asian students had the highest SAT score among racial/ethnic groups, followed by White students.

Only Asian and White students scored above the state average.

NC SAT Math Scores by Race


Nc sat math scores by race1

NC SAT Math Scores by Race

Mean SAT Score


Sat mathematics scores by race gender

SAT Mathematics Scores By Race/Gender

Mean SAT Score


For more information

For More Information . . .

SAT Reports from 1995 to 2007 include information and data

for:

the Nation

the State

117 Public School Systems

Charter Schools

North Carolina School of the Arts

North Carolina School of Science and Mathematics

http://www.ncpublicschools.org/accountability/reporting/sat/


Sample sat report page

Sample SAT Report Page

  • Some questions to explore:

  • Are districts closer to the state average in Math than in Critical Reading?

  • How about variability among schools within districts?

  • How about participation rates?


Question of the day

Question of the Day …

  • SAT puts an “official” question of the day on its web site.

  • Excellent way to get a better understanding of what students will be facing. Why not have 8th graders try them?

  • www.collegeboard.com


Purposes part 2 green book

Purposes – Part 2: Green Book

  • Find out more about getting data from the Green Book, …

  • both in its standard form …

  • and its interactive, electronic form.


Electronic greenbook a web tool for data

Electronic GreenBook:A Web-tool for Data


Electronic greenbook a web tool for data1

Electronic GreenBook:A Web-tool for Data

Your


Electronic greenbook

Electronic GreenBook


Electronic greenbook http report ncsu edu ncpublicschools

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

  • Equally accurate

  • Equally reliable

  • More timely

  • More accessible


Electronic greenbook http report ncsu edu ncpublicschools1

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools2

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools3

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools4

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools5

year

students

subject

grade

geography

display

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools6

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

1. participation

2. performance


Electronic greenbook http report ncsu edu ncpublicschools7

year

geography

grade & subject

students

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools8

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

So far …

  • Questions?

  • Answers?

  • 5% / 95% Rule


Electronic greenbook http report ncsu edu ncpublicschools9

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools10

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools11

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools12

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools13

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools14

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools15

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic greenbook http report ncsu edu ncpublicschools16

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Classic green book

Classic Green Book

  • SAMPLE PAGES FOLLOWING …..

  • Finding average scale scores for LEAs.

  • Finding standard deviations for EOC and EOG scale scores.

  • ….. Which leads in to … PART 3


Purposes part 3 the lens

Purposes – Part 3: THE LENS

  • Discuss effect size, a useful tool for a fuller understanding of LEA or school performance across grade and time.

  • Compute some math effect sizes and highlight where the information is obtained.

  • Show the relationship between LEA math performance on SAT, EOC, and EOG.


The importance of where

The Importance of “Where.”

  • ‘A school (or LEA) is data driven wise if everyone knows where they are and where they are going.”

  • … Donna Snodgrass, et al.


Where is this lea

Where is this LEA?

  • Math ABC Results for 2006-2007

  • (Percent at or above grade level)

  • EOG Grade 5 ….. 59.8 (state 66.7)

  • EOG Grade 8 ….. 48.9 (state 65.1)

  • EOC Geometry …45.0 (state 63.0)

  • Pass rates and gaps in pass rates leave out a big part of the picture.


The average is above average

The Average is above Average

  • The percent above a cut score counts a score the same whether it is just above (below) the cut or far above (below) the cut. BUT the average includes all the score information.

  • Year-to-year changes in percent above are affected by where on the distribution the cut is located. BUT the average follows the center of scores.


Then where is this lea

Then …Where is this LEA?

  • Math Score Averages for 2007:

  • EOG Grade 5 …. 353.0 (State 354.7)

  • EOG Grade 8 …. 357.1 (State 360.5)

  • EOC Geom. … 147.1 (State 151.3)

  • SAT … 493 (State 509)

  • SAT % Tested … 72.8 (State 71.0)

  • That’s still difficult to answer with different tests on different scales.


Effect size takes care of differing scales

Effect Size takes care of Differing Scales

  • Example: Calculate the effect size for math SAT:

  • Subtract the overall average from the local average: 493-509 = -16

  • Divide by the Standard Deviation:

  • -16 / 110 = -0.1 (rounded to 10ths)


So where is this lea

So … Where is this LEA?

  • Math Effect Sizes for 2007:

  • EOG Grade 5 ….. -0.2

  • EOG Grade 8 ….. -0.4

  • EOC Geometry … -0.4

  • Average SAT . . . . -0.1

  • SAT is very close to the state. Grade 5 is low but still close. Geometry and Grade 8 are much lower.


Questions to pose

Questions to pose ..

  • In your LEA (or school) which grades, courses, SAT results are stronger?

  • In your LEA how do schools compare on average EOG, EOC, SAT?

  • How does your LEA (school) compare with similar LEAs (schools)?

  • What is the effect of what you’ve been trying to improve?


Nc math s great progress

NC math’s great progress

  • NAEP results from 1992 to 2005

  • Compared with the U.S. average and with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to 2005.

  • For the NAEP data in the following slides, go to:

  • http://nces.ed.gov/nationsreportcard/mathematics/


Naep scale score averages a decade of great progress

NAEP Scale Score Averages – A Decade of Great Progress

Grade 8

U.S. Average

North Carolina

Grade 4

U.S. Average

North Carolina


  • Login