John F. Olson Assessment Solutions Group CCSSO National Conference on Student Assessment

1. Getting the Most Value for Your Assessment Dollar: Cost Analysis for the New Kentucky Assessment System John F. Olson Assessment Solutions Group CCSSO National Conference on Student Assessment Detroit, MI June 23, 2010

2. Overview The ASG Cost Model was used to develop the cost of the new Kentucky assessment system components. ASG’s report provided the KDE with a realistic estimate of the cost of the key components of its new 2012 assessment system that can be used for budgeting purposes. Modeling the costs of the new system using the actual design of the new assessment program will provide an accurate estimate of the costs KDE should expect to see when it bids the program in 2010 Assessment Solutions Group www.assessmentgroup.org Confidential

3. Overview (cont.) • ASG model outputs also allowed the KDE to gain a deeper understanding of the costs of the various components of each assessment • This provides KDE the ability to make trade-offs in assessment program design prior to competitive bidding in order to develop the highest quality and cost-effective assessment for the budgeted dollars. • The ASG Cost Model is a variable input, metric-based calculation output model. • Specific assessment program variables are input to the model and applied against cost factors to derive assessment cost • Several hundred variables are contained in the model. • The cost for the entire assessment is built from the ground up, element by element, function by function. • Costs are provided for total assessment, by year, by content area, by grade, and by functional activity. Assessment Solutions Group www.assessmentgroup.org Confidential

4. Key Program Elements & Assumptions • The new 2012 Kentucky assessment system consists of the following elements: • Combined CRT/NRT assessments in • Math grades 3-8, 11 • Reading grades 3-8, 10 • Science grades 4, 7, 11 • Social Studies grades 5, 8, 11 • A writing test grades 4, 6, 8, 10, 12 • A writing editing & mechanics exam grades 4, 8, 12 • Two forms of each assessment will be developed for year 1 (2012) with a new form developed each subsequent year. Assessment Solutions Group www.assessmentgroup.org Confidential

5. Key Program Elements & Assumptions • Assessments will be given to students during the last two weeks of the school year, as specified in the new state law (May – early June). • Test results will be mailed to parents and posted online in August. • For this study ASG calculated: • Incremental year zero (2011) costs required to develop and field test the items to be used in the operational, year one (2012) administration • Ongoing total assessment costs for the years 2012, 2013, and 2014 Assessment Solutions Group www.assessmentgroup.org Confidential

6. Year 0 Costs Cost by functional area for Year 0 (2011) Note that ASG’s base case assumes year 0 work is priced at a 50% margin, which is typical of “transition year” programs. Year 0 work also is shown at a 30% margin. Assessment Solutions Group www.assessmentgroup.org Confidential

7. 2012 Costs Cost for each functional area and cost per student – Year 1 (2012) Assessment Solutions Group www.assessmentgroup.org Confidential

8. 2012 Cost by Content Area Total cost by content area – Year 1 (2012) Assessment Solutions Group www.assessmentgroup.org Confidential

9. Cost by Grade Total cost for the Kentucky assessment program by grade (2012 – 2014) Note: grades 4 and 8 are tested more frequently; consumable test books are used for grade 3 Assessment Solutions Group www.assessmentgroup.org Confidential

10. Cost by Content Area & Grade Costs for the Kentucky assessment program by content area & grade for 2012 – 2014. Assessment Solutions Group www.assessmentgroup.org Confidential

11. Total Program Costs Total program costs for 2011 - 2014 Assessment Solutions Group www.assessmentgroup.org Confidential

12. Total Costs and Key Metrics by Year Total program costs and key metrics by year for 2012 - 2014 Assessment Solutions Group www.assessmentgroup.org Confidential

13. Summary of Impact of Options/Changes in Costs Chart shows cost savings or increases related to various options/changes to the program for 2012 - 2014 Assessment Solutions Group www.assessmentgroup.org Confidential

14. Summary, Conclusions, & Recommendations • ASG performed a special study analyzing the costs of the new Kentucky 2012 assessment system • Year 0 (2011) activities – item development and field testing • Years 1 – 3 (2012-2014) – full administration of new assessment • A vendor performing efficiently is expected to price the year 0 activities at $3.2M and the total program costs for years 2012-2014 at $12.5M, $12.7M, and $13.2M respectively • Consistent with other NCLB state programs, the key costs are in the item development, scoring, and printing areas • ASG performed a number of scenarios to consider what could change the cost picture significantly Assessment Solutions Group www.assessmentgroup.org Confidential

15. Summary, Conclusions, & Recommendations ASG’s analysis finds the total cost and cost per student of the new Kentucky assessment system appear to be in a reasonable range for a high stakes, NCLB-type state assessment. Metrics and benchmarks used in the analysis represent those of an efficient contractor realistically bidding on the program The proposed item release rate and use of NRT components require only one form per year which significantly controls costs. Assessment Solutions Group www.assessmentgroup.org Confidential

16. Summary, Conclusions, & Recommendations • ASG recommends that the KDE consider ways to improve the cost-effectiveness and efficiency of the new assessment system in the future. • When developing the RFP for the new Kentucky assessment program it will be critical to have a clearly written proposal that calls for the vendor to submit the metrics and specifications outlining their understanding of the program requirements. Assessment Solutions Group www.assessmentgroup.org Confidential