Weighing the Evidence:
1 / 18

Weighing the Evidence: - PowerPoint PPT Presentation

  • Uploaded on

Weighing the Evidence: How Do Accountability Programs Impact Student Achievement?. Virginia L. Hudson & Gary W. Ritter Office for Education Policy University of Arkansas Christopher J. Lucas, Ronna C. Turner, & Karen Minkel

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Weighing the Evidence: ' - sheng

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Weighing the Evidence:

How Do Accountability Programs

Impact Student Achievement?

Virginia L. Hudson & Gary W. Ritter

Office for Education Policy

University of Arkansas

Christopher J. Lucas, Ronna C. Turner,

& Karen Minkel

University of Arkansas

Tracy L. Williams

Hamline University

American Education Finance Association (AEFA), Louisville, KY March 2005


  • Context: State accountability plans

    • No Child Left Behind

    • More studies completed in recent years

  • Research Question: What is the impact of accountability programs (focused on schools or students) on student performance?

  • Method: Conducted comprehensive, systematic review of literature

Method systematic review
Method: Systematic Review

  • A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review.

  • Sources:

    • Campbell Collaboration:

    • http://www.campbellcollaboration.org

    • Evidence for Policy and Practice Information and Coordination Centre (EPPI-Centre)

    • http://eppi.ioe.ac.uk

Search strategy
Search Strategy

  • Electronic databases

    • Ebsco, Psycinfo, ERIC, Digital Dissertations, ProQuest

  • Hand-searching journals

    • Education Policy Analysis Archives (EPAA)

    • Educational Evaluation and Policy Analysis (EEPA)

    • Education Next

    • Journal of Policy Analysis and Management (JPAM)

    • Review of Educational Research (RER)

Search strategy1
Search Strategy

  • Education Weekarchives

    • Seeking newsworthy but unpublished studies

  • Reviewing lists of references from relevant articles

    • Ongoing

Inclusion criteria
Inclusion Criteria

  • Conducted and published after 1980

  • Focus on impact of statewide or district-wide K-12 accountability plans in U.S.: setting standards, designing related assessments, and imposing consequences

  • Quantifiable results (no case studies/qualitative inquiries, theoretical analyses, etc.)

  • At least one outcome measure from among:

    • reading/language, math, or science exams; graduation rates; promotion/retention rates

  • No studies focusing solely on population subgroup

  • Recently added MCTs

Data extraction synthesis
Data Extraction & Synthesis

  • Outcome measures

  • Results (did accountability affect achievement?)

  • Methodological weaknesses & criticisms of study designs

  • Source (peer-reviewed journal or not)

  • Inclusion decision (met criteria or not)

  • Study citation

  • Type of accountability & level of government imposing accountability

  • Study sample and unit of analysis (state or school)

  • Time frame

  • Analytical techniques (simple pre-post? comparison group? multiple regression?)

Down to 19 or 14
Down to 19 (or 14)

  • After a thorough review of the 25, reduced studies that were related to accountability, but …

    • Tested whether state test score changes were aligned with changes on norm-referenced exams

    • Studies of district accountability systems within states with accountability systems

  • Crude Quality Measure (5 more)

    • Treatment N = 1

    • Pre-Post growth with no controls

Accountability types
Accountability Types

1990s System-wide Accountability (6)

  • 3 focus on all states (NAEP), 2 on FL, 1 on Chicago

    1990s HS Testing (3)

  • Amrein/Berliner identifed 18 HS states; 3 studies reviewed these same states

    First Wave MCT Reforms (5)

  • Used NELS, identified students in states with minimum competency exams in 1980s (pupil = unit of analysis)

Results of 14 hq studies

Outcomes (some overlap)

NAEP = 6

Other Exams = 4

Graduation or Dropout Rate = 5

Analytic Methods

Pre-Post with CG = 4

Multivariate Models = 10

Preliminary Results = Simple Vote Counting

Outcomes either +, neutral, or -

Results of 14 HQ Studies


  • Overall Outcome

    • No evidence that the existing accountability programs are “harming” students in terms of overall student achievement

    • 10 of 14 studies have positive results

  • Characteristics

    • NAEP (and other test scores) tend to show positive results

    • Grad and Dropout Rate more likely to show negative result

  • Limitations … next steps

    • MCT and accountability systems vary

    • Small set of research studies (for now)

    • No meta analysis (appropriate?)