1 / 36

Washington’s New Accountability Index

Washington’s New Accountability Index. WERA Spring Conference March 2009 Pete Bylsma, EdD, MPA Consultant to the State Board of Education http://www.sbe.wa.gov/spa.htm bylsmapj@comcast.net. Overview of Presentation. Rationale for creating a new system Guiding principles

connor-howard
Download Presentation

Washington’s New Accountability Index

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Washington’s New Accountability Index WERA Spring Conference March 2009 Pete Bylsma, EdD, MPA Consultant to the State Board of Education http://www.sbe.wa.gov/spa.htm bylsmapj@comcast.net

  2. Overview of Presentation • Rationale for creating a new system • Guiding principles • Process used to develop the proposed system • Proposed indicators and outcomes • Ratings and tier assignments • School and district results in 2007 and 2008 • Handling special cases • Recognition categories and criteria • AYP vs. new system • Identifying “Priority” schools and districts • Remaining Issues

  3. Why a state accountability system? • Legislative mandate for the SBE to “adopt objective, systematic criteria” for successful schools and districts and for those in need of assistance • Accountability is part a comprehensive state education reform system, which includes standards & assessments and assumes adequate funding • Federal accountability system (AYP) is viewed as unfair, too complicated, and punitive • State provides most of the education funding • Supports legislative efforts to revise funding system

  4. Guiding Principles • Be fair, reasonable, and consistent • Be transparent and simple to understand • Be valid and accurate • Use existing data • Rely on multiple measures • Include state assessment results from all grades and subjects • Apply to as many schools & districts as possible • Provide multiple ways to show success and earn recognition • Rely mainly on criterion-referenced measures • Use familiar concepts • Use concepts of AYP when appropriate • Be flexible enough to accommodate future changes

  5. Process to Date • Initial proposal reviewed by an advisory panel • Convened diverse set of advisors to provide input on technical details for a revised proposal • Applied proposed rules to data to ensure validity, made adjustments to increase precision • Ongoing feedback received from OSPI and others • Presentations and feedback at SBE meetings and ESDs • More than 60 districts have received data for their schools • Further refinements taking place

  6. Proposed Outcomes/Indicators Five Outcomes Results from 4 assessments (reading, writing, math, science) aggregated together from all grades Extended graduation rate for all students Four Indicators • Achievementby non-low income students (% meeting standard/ext. grad rate) • Achievement by low income students (eligible for FRL) • Achievement vs. Peers (Learning Index and ext. grad rate controlling for ELL, low-income, special ed., gifted, mobility) • Improvement (change in Learning Index from previous year) Creates a 5x4 matrix with 20 outcomes

  7. Outcome/Indicator Matrix * Simple average of all rated cells

  8. Outcome/Indicator Benchmarks & Ratings

  9. Outcome/Indicator Benchmarks & Ratings

  10. Achievement vs. “Peers” • Recognizes context affects outcomes (“statistical neighbors”) • Makes “apples to apples” comparisons using multiple regressionto control for 5 student variables—percent ELL, low-income (FRL), special education, mobile, gifted • Include expenditure level (adj.) in district regression • Regressions weighted by number of students assessed • Separate regressions for each type of school (e.g., elementary, middle, high, multiple grades) • Non-regular schools not included in regression (not rated) • Viewed as significant new contribution • Others are starting to do this type of analysis • Complexity presents a communications challenge

  11. Illustration of Ach. vs. Peers A 7 B 4 1

  12. Tier Names and Ranges Assigned to a “tier” based on index score

  13. School and District Results (2007) 228 schools were in the Struggling tier with 74,000 students (1 in every 14 students); 98 were alternative schools or served other special populations.Over 2-year period, 7.5% of all schools averaged < 2.50 (Struggling tier) 17 districts were in the struggling tier in 2007 (average 1,000 students)

  14. Key Changes Made • Slow down implementation schedule • Avoid double-counting students by creating separate ratings for low income and non-low income students • Move to a 7-point scale, add another tier • Exclude results for ELL students in first 3 years or until reaching Level 3 on WLPT (whichever comes 1st) • Think of other ways to hold alternative school accountable • Exclude improvement indicator for those with very high performance • Simplify recognition system

  15. Example Results(Middle School, 2008)

  16. District Accountability • Uses the same indicators, outcomes, benchmarks, ratings, recognition system as schools • Combines all the grades together(no separate results for each grade band) • N is 10 for entire district (very few left out) • Control for funding level for “peer” analysis

  17. Showing Results for All Schools(2008)

  18. Showing Results for All Schools(2008)

  19. Showing Index for All Schools (2008)

  20. Special Case #1 – ELL Exemption • Propose exempting ELL results in the first 3 years of enrollment or until advanced proficiency achieved on the WLPT (whichever comes first) • Results currently count in AYP in 2nd year of enrollment • Research found it usually takes at least 3 years to achieve “academic” proficiency in English • OSPI requested this policy but was denied; WA could still use this policy when calculating the index • ELLs would still take the test in their 2nd year, WLPT results would be made public to increase accountability • Would not affect many students (most ELLs in tested grades have been in US for 3 years or have achieved intermediate proficiency); would have a small positive impact on index where there are many ELLs present

  21. OSPI Proposal to US Education Dept. • Title III application submitted in January • Redefines “proficiency” for ELL cells in AYP based on WLPT • - Uses sliding scale: Lower scale score counts as proficient on the WASL for each grade, subject, and WLPT Level. For example: • Grade 3 student in Level 2 of WLPT is considered proficient withWASL scale score of 359; if in Level 3, considered proficient with 388 • Cut scores are lower for math • No scores proposed yet for writing or science • Concept similar to WASL-Basic for special education(lower scale score considered proficient), rejected by feds • Has almost no effect on AYP results, but produces useful data about student progress toward meeting standard

  22. Special Case #2 – Alternative Schools • Designation at district discretion, some miscoding occurs (a special program may not be given a separate school code) • Most are small: average 145 students, range from 1 to 2200+290 schools (14%) with 42,000 students (4%) • About half serve only secondary (usually 9-12), many serve K-12 • Many types exist • Correctional facilities (jails, prisons, detention centers) • Contracts for vulnerable groups (treatment centers, group homes) • Schools for specific types of students (ELLs, gifted, spec. ed.) • Growing number serve students learning via the Internet • Parent Partnerships • Some don’t use a normal school approach or a normal building (e.g., college campus, night school)

  23. “Alternative” Schools Index Results • Given this diversity, no “peer” indicator is computed for non-regular schools • Half do not have any index results due to N<10, many only have a few cells rated • Average index is much lower (2.94 vs 3.95 for “regular”) but some perform very well (magnet schools, gifted) • 33% in Struggling tier, 9% in Very Good or Exemplary tiers • Usually made AYP (below the radar due to N< 30 cont. enrolled) • Two Options Being Considered • 1: Use regular process, use in-depth analysis to determine if school is using best practices, showing progress, and their role/status/resource level within the district • 2: Allow schools serving high-risk/special populations to use additional measures to determine their tier (e.g., credits earned, attendance, gains on pre-post tests)

  24. Special Case #3Improvement by High Performers • Propose excluding improvement indicator when reaching very high achievement levels • Improvement is difficult when achievement is very high • Cannot receive the highest rating (7) after the Learning Index reaches 3.85/4.00 and when the graduation rate reaches 94% • Would go into effect when achievement is in the top range two years in a row, making it impossible to receive the maximum rating in the second year • No school or district has yet to met these criteria for the assessments; 11% met these criteria for graduation rate • School/district could decide to include the results

  25. Recognition System • Guiding principles • Multiple ways to demonstrate success, earn recognition • Criterion-based system Theory of Change: People are motivated more by successthan guilt or blame; need clear, challenging, attainable goals; small wins build momentum for continued improvement • Use same accountability matrix, receive recognition when meeting specific benchmarks • Based on 2-year average

  26. Recognition Recommendations 1. Provide recognition to schools & districts for each of the 20 cells when the 2-year average is at least 5.50 and when the index is averages at least 5.00 (21 cells) 2. Require some minimum conditions to occur • Must have ratings of 5-7 in both years for 20 “inner” cells • Recognition for non-low income cells in reading and writing requiresa minimum 2-year average of the low income group of 4.00 • Must have at least 4 cells rated each year for recognition in Index cell 3. Coordinate recognition system with OSPI 4. Give recognition each fall via public announcement (joint SBE/OSPI press release), post on web 5. Consider giving special recognition in some cases

  27. Suggested Cells and Criteria Recognize results in 20 cells + Index (21 total) * Minimum 2-year average rating to earn recognition

  28. EFFECT • Distribution of Recognition in21 Cells • Math, science, andlow income cells recognized least often • Districts receive less recognition than schools

  29. Complement Existing Recognition Federal recognition (competitive) • Blue Ribbon Schools • Academic Achievement Award • Distinguished Schools Award • Title I Improvement Award OSPI recognition (status TBD) • Schools of Distinction (top 5%) • Improvement Award (schools and districts)

  30. Current Recognition Note: Improvement is measured in different ways.

  31. Role of AYP • Widespread belief that AYP is overly complex, less inclusive, too narrow, not transparent, and not valid for identifying success or most in need • Recommend using disaggregated results to help determine which schools and districts need the most help (“Priority” designation) • Will submit proposal to US Ed. Dept. to use the index to determine AYP and “steps” of improvement

  32. Advantages of Proposed System • More valid – Uses performance of all students in more subjects, more differentiated than a “Yes/No” system, does not count students multiple times, looks at improvement and how the outcomes compare with “peers” • More inclusive/comprehensive – Uses smaller minimum N (10 students across entire school/district), includes results of all students (regardless of how long they have been enrolled), includes both writing and science (helps prevent a narrowing of the curriculum), uses Learning Index to measures performance across the range of assessment results (reduces the focus on bubble kids) • Less volatile over time – Assessment results are combined across all grades (not individual grades) so N is larger and students do not change much from year to year • More transparent – Does not include a margin of error, benchmarks are the same over time and among different subjects, fewer subgroups and rules, schools and districts evaluated using same criteria, same N for all groups and for both schools and districts • Encourages high expectations/standards – “Compensatory” model gives no incentive to lower standards so all can be counted as “proficient” • Requires deeper analysis before final decisions are made – Not solely a quantitative approach

  33. Identifying Priority Schools • Those in “Struggling” tier undergo deeper analysis to determine which need more help • Many issues to examine: • Contextual issues • In-depth analysis of disaggregated WASL/WAAS • Federal AYP results • Other data • Identification process similar to that used now by OSPI

  34. Issues to be Resolved • Determine how index is used to made AYP determinations. Ideas include the following: • Minimum N=10, no margin of error used, no results by grade • Only results from index (average of cells) are used over time • Possible ways of not making AYP (warning year) • In Struggling tier • In Acceptable tier, no increase in improvement average in 2 straight years • In Good/Very Good tier, index declines > .70 in 2 straight years • District with 20% of students in schools not making AYP • Entering a “Step” • Not making AYP two years in a row • Always requires “deeper data dive” before decision is made about moving (may stay in warning year or current step, even if not making AYP) • Sanctions would differ, could be fewer steps for schools • Move back a step if making AYP

  35. Issues to be Resolved • How to include other outcome measures andshow “reciprocal” accountability • Student and staff characteristics • District revenues and expenditures • Community and state support • More student outcomes • How SBE and OSPI systems work together • Will develop joint proposal for US Education Dept. • Show results on Report Card • Determine how index results relate to recognition/state assistance • What happens to those in a Step now

  36. Issues to be Resolved • Timing of implementation • Phase I this fall for Recognition (?) • Announcing index results • Provides public more accurate picture of school and district performance (AYP results provide a false picture) • Gives educators with useful data for improvement efforts • Introduces concepts before used for federal accountability • Provides SPI with additional data for assistance decisions • Phase II depends on federal response and resources to support those in Priority tier

More Related