Effect Size Lens on
This presentation is the property of its rightful owner.
Sponsored Links
1 / 56

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008 PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008. Presentation by Johnny Wilson, Charles Lanier, and Mike Gallagher, all from the Accountability Services Dept., NCDPI. Purposes – Overall Session.

Download Presentation

Effect Size Lens on Local SAT and State Math Test Results NC Accountability Conference February 2008

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Effect Size Lens on Local SAT and State Math Test Results NCAccountability ConferenceFebruary 2008

  • Presentation by Johnny Wilson, Charles Lanier, and Mike Gallagher, all from the Accountability Services Dept., NCDPI


Purposes – Overall Session

  • Provide background and access paths to important data. (important = helpful)

  • Show how LEAs and schools can access and compare test results across EOG, EOC and SAT.

  • Entice folks to exploit the usefulness of such comparisons.


Who’s on First?

  • Johnny Wilson, Reporting Section, SAT Data Report. [email protected]

  • Charles Lanier, Test Development Section, State Testing Data in the Green Book. [email protected]

  • Mike Gallagher, Test Development, Effect Size Look at SAT and State Test Results. [email protected]


Purposes – Part 1: SAT

  • Get acquainted with the major trends in SAT results for North Carolina.

  • Look at school level variability of SAT results from year-to-year.

  • Show the access path for information at the school and LEA levels.


SAT Math Score Trends

Between 1997 and 2007, the Math gap between NC and the Nation narrowed from 23 pts. to 6 pts.

NC’s Math scores fell in 2007 after more than a decade of gains.


SAT Math Score Trends


SAT Verbal Score Trends

By comparison, NC’s Verbal scores have been up and down over the same period.

Between 1997 and 2007, the Math gap between NC and the Nation has narrowed from 15 pts. to 7 pts.


SAT Verbal Score Trends


SAT Total Score Trends for NC and the Nation

  • After 10 consecutive years of gains,

  • NC’s mean total SAT scores fell in

  • 2006 and 2007.

  • Gap between NC’s mean total score

  • and the nation’s score has narrowed

  • from 53 in 1990 to 13 points in

  • 2007.


Total Score Trends for NC and the Nation


NC’s Schools with Declining SAT Scores

At the school level, math scores also declined in 2006 & 2007, compared with the previous year.

Notably, fewer schools increased math scores in 2007 than in 2006.


Percent of NC’s Schools with Declining SAT Scores


Changes in SAT Math Scores

Most changes in SAT scores from year to year are not unusual.

Low-Volume Schools tend to have larger changes.

The larger the test-taking population, the smaller changes in SAT scores tend to be.


Changes in SAT Math Scores by Test-taking Volume (US)


Changes in SAT Math Scores by Test-taking Volume (NC)


Changes in SAT Math Scores All Schools (US)


Changes in SAT Math Scores All Schools (NC)


Factors that Might Influence Score Fluctuations

Fatigue Factor

SAT fee increase in 2005 resulted in decline in repeat test-takers.

Difficulty of the test


NC’s Asian students had the highest SAT score among racial/ethnic groups, followed by White students.

Only Asian and White students scored above the state average.

NC SAT Math Scores by Race


NC SAT Math Scores by Race

Mean SAT Score


SAT Mathematics Scores By Race/Gender

Mean SAT Score


For More Information . . .

SAT Reports from 1995 to 2007 include information and data

for:

the Nation

the State

117 Public School Systems

Charter Schools

North Carolina School of the Arts

North Carolina School of Science and Mathematics

http://www.ncpublicschools.org/accountability/reporting/sat/


Sample SAT Report Page

  • Some questions to explore:

  • Are districts closer to the state average in Math than in Critical Reading?

  • How about variability among schools within districts?

  • How about participation rates?


Question of the Day …

  • SAT puts an “official” question of the day on its web site.

  • Excellent way to get a better understanding of what students will be facing. Why not have 8th graders try them?

  • www.collegeboard.com


Purposes – Part 2: Green Book

  • Find out more about getting data from the Green Book, …

  • both in its standard form …

  • and its interactive, electronic form.


Electronic GreenBook:A Web-tool for Data


Electronic GreenBook:A Web-tool for Data

Your


Electronic GreenBook


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

  • Equally accurate

  • Equally reliable

  • More timely

  • More accessible


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


year

students

subject

grade

geography

display

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

1. participation

2. performance


year

geography

grade & subject

students

Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools

So far …

  • Questions?

  • Answers?

  • 5% / 95% Rule


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Electronic GreenBookhttp://report.ncsu.edu/ncpublicschools


Classic Green Book

  • SAMPLE PAGES FOLLOWING …..

  • Finding average scale scores for LEAs.

  • Finding standard deviations for EOC and EOG scale scores.

  • ….. Which leads in to … PART 3


Purposes – Part 3: THE LENS

  • Discuss effect size, a useful tool for a fuller understanding of LEA or school performance across grade and time.

  • Compute some math effect sizes and highlight where the information is obtained.

  • Show the relationship between LEA math performance on SAT, EOC, and EOG.


The Importance of “Where.”

  • ‘A school (or LEA) is data driven wise if everyone knows where they are and where they are going.”

  • … Donna Snodgrass, et al.


Where is this LEA?

  • Math ABC Results for 2006-2007

  • (Percent at or above grade level)

  • EOG Grade 5 ….. 59.8 (state 66.7)

  • EOG Grade 8 ….. 48.9 (state 65.1)

  • EOC Geometry …45.0 (state 63.0)

  • Pass rates and gaps in pass rates leave out a big part of the picture.


The Average is above Average

  • The percent above a cut score counts a score the same whether it is just above (below) the cut or far above (below) the cut. BUT the average includes all the score information.

  • Year-to-year changes in percent above are affected by where on the distribution the cut is located. BUT the average follows the center of scores.


Then …Where is this LEA?

  • Math Score Averages for 2007:

  • EOG Grade 5 …. 353.0 (State 354.7)

  • EOG Grade 8 …. 357.1 (State 360.5)

  • EOC Geom. … 147.1 (State 151.3)

  • SAT … 493 (State 509)

  • SAT % Tested … 72.8 (State 71.0)

  • That’s still difficult to answer with different tests on different scales.


Effect Size takes care of Differing Scales

  • Example: Calculate the effect size for math SAT:

  • Subtract the overall average from the local average: 493-509 = -16

  • Divide by the Standard Deviation:

  • -16 / 110 = -0.1 (rounded to 10ths)


So … Where is this LEA?

  • Math Effect Sizes for 2007:

  • EOG Grade 5 ….. -0.2

  • EOG Grade 8 ….. -0.4

  • EOC Geometry … -0.4

  • Average SAT . . . . -0.1

  • SAT is very close to the state. Grade 5 is low but still close. Geometry and Grade 8 are much lower.


Questions to pose ..

  • In your LEA (or school) which grades, courses, SAT results are stronger?

  • In your LEA how do schools compare on average EOG, EOC, SAT?

  • How does your LEA (school) compare with similar LEAs (schools)?

  • What is the effect of what you’ve been trying to improve?


NC math’s great progress

  • NAEP results from 1992 to 2005

  • Compared with the U.S. average and with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to 2005.

  • For the NAEP data in the following slides, go to:

  • http://nces.ed.gov/nationsreportcard/mathematics/


NAEP Scale Score Averages – A Decade of Great Progress

Grade 8

U.S. Average

North Carolina

Grade 4

U.S. Average

North Carolina


  • Login