1 / 21

To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010

Getting the Most Value for Your Assessment Dollar – Designing Adapting and Maintaining Quality Assessment Programs During Tough Economic Times. To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010. Joining a Consortium.

Download Presentation

To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Getting the Most Value for Your Assessment Dollar – Designing Adapting and Maintaining Quality Assessment Programs During Tough Economic Times To Consortia, or not to Consortia CCSSO National Conference on Student Assessment June 23, 2010

  2. Joining a Consortium Implementing a new, innovative assessment program in a consortium as a way to save costs.Or, maintaining a current program without having to make drastic cuts • Is it doable? • Can a consortia of states implement a new assessment at a significantly reduced cost than a single state acting alone? • How large does the consortia need to be? • Where are the cost savings opportunities? www.assessmentgroup.org

  3. Joining a State Assessment Consortium • Joining a state assessment consortium can have its advantages but. . . • Requires a lot of planning, coordination & desire • Several successful examples: • NECAP • WIDA • Achieve Algebra 2 • PARCC & SBAC (responses due today)

  4. Stanford/Nellie Mae Study Purpose of the study was to see if it is possible to create an affordable “high quality” assessment • Step one – Model a current typical assessment in ELA and Math – Cost $19-$20 • Step two – Model a high quality assessment for the same state – Cost $55-$56 a student • Step three – Implement several cost savings strategies Assessment Solutions Group www.assessmentgroup.org

  5. Cost Reduction Strategies • Participation in a consortium • Looked at 10, 20 and 30 state sizes • Cost reduction - $15 per student • Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items • Together, these innovations account for cost savings of about $3 to $4 per student • Likely to account for more as efficiencies are developed in programming and using technology for these purposes • Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring

  6. Cost Reduction Strategies • Two different models for teacher-moderated scoring: • Professional development model - no additional teacher compensation beyond that supported by the state or district for normal professional development days (NY Regents) • Stipend model - assume a $125/day stipend for teachers to score the performance items. • Note: teachers were assumed to score all performance items in a distributed scoring model • These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid) • Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21 Assessment Solutions Group www.assessmentgroup.org

  7. Consortia Size How big do you have to be? • Stanford/Nellie Mae study found that 80% of the cost benefits of joining a consortium are realized at the 10 state size. • Rough estimate is that a 5 state consortium could achieve 75%+ of the cost savings of a 10 state consortia • Perhaps $3 - $6 per student • $2.7 M/year for the average sized state (600K students)

  8. Where are the Cost Savings? • Big cost savings opportunity in development • Largely a fixed cost function • Increase in forms cost partially offsets the savings • Other fixed cost functions such as IT, Quality Assurance and Psychometrics provide savings • Even functions that are largely variable in nature also have a fixed cost component • Some functions like program management allow for economies of scale www.assessmentgroup.org

  9. Assessment Costs by Consortium Size

  10. Where are the Cost Savings? • Consortia size can make assessment technology more affordable • Online test delivery (CBT and CAT) • Artificial intelligence scoring of CRs • More states/students more bargaining power • A common assessment with common standards and operational methods s/b more efficient • Need to weigh this against potential additional collaboration costs and risks

  11. PARCC & SBAC Support • We recently assisted both consortia in preparing their cost estimates for the NIA responses • Both consortia had innovative ideas for new assessments and a wide variety of design and operational decisions to make • Each idea/design choice came with unique cost implications www.assessmentgroup.org

  12. PARCC & SBAC Support • Initially, each consortia’s design was deemed too expensive in both the operational and ongoing periods. Each needed adjustments: • The number of choices and variables can be daunting as there are many variables and moving parts • Ultimately, each consortia created innovative assessment systems with the designs they wanted

  13. Assessment Design Decision Tree • Delivery Method • Paper based • Computer (linear or CAT) • Mixed (both CBT and PPT) • Assessment Types • Summative, through course summative • Interim/benchmark, End of Course, Formative • Domains, special populations Indicates a major cost element for either PARCC or SBAC

  14. Decisions and Cost Variables (cont.) • Development • Types of items (SR, CR, Computer enhanced, PE, PT) • Mix of item types • Number of forms, CAT algorithm (750-1000 items per grade), number of attempts • Release rates (by item type) • Breach form (develop?, print?) • Grades/domains tested • Item bank development

  15. Decisions and Cost Variables • Paper based testing/cutover to CBT • How long to cut over (operating in both modes is very expensive)? • Different production strategies • Minimize print page “signatures” • Use of color (B/W, grey scale, 4 color) • Breach form (print?) • Security measures (# of forms, labels, seals, student ID) www.assessmentgroup.org

  16. Decisions and Cost Variables • Logistics • Transportation mode (ground, air) • Carrier selection • Ship from/to locations (consolidated shipping) • Meetings and Travel (online vs. live) • Scoring • Computer vs. Human (incl. scanning and editing)

  17. Design Decisions & Costs • Scoring (cont.) • Human Method (teacher or 3rd party) • Holistic vs. analytic scoring • Requires a lot of work to develop innovative items that can be scored in a timely manner • Alternatively, a test design where these items are scored during a classroom period may make sense (PEs) • AI scoring for open ended items • Math vs. ELA • Items requiring inference can’t easily be scored using AI • System training fees (fixed cost); per score costs

  18. Design Decisions & Costs • Open-Ended Scoring (cont.) • Double scoring/Read behind rates (by grade) • Distributed vs. on-site • Reporting • Paper vs. online reporting • Number and complexity of reports www.assessmentgroup.org

  19. Conclusion • Even a small consortium of states can achieve significant reductions in assessment cost • Such a strategy can be useful in developing a new, high quality assessment or maintaining a current one during times of budgetary stress • Participating in a consortium also allows for the implementation of innovative technologies that can improve assessment quality and reduce costs • Teacher scoring of open-ended items is critical for implementing a high quality assessment • There are a myriad of design and operational decisions that have significant cost impacts

  20. Conclusion “You can’t always get what you want; but if you try sometime you just might find you get what you need.” - Mick Jagger www.assessmentgroup.org

  21. Questions? • Barry Topol btopol@assessmentgroup.org • John Olson jolson@assessmentgroup.org • Ed Roeber eroeber@assessmentgroup.org www.assessmentgroup.org

More Related