Multiple Perspectives on Computer Adaptive Testing for K-12 Assessments

1. Multiple Perspectives on Computer Adaptive Testing for K-12 Assessments U.S. Department of Education David J. Harmon, Ph.D. Program Specialist Policy implications from the federal perspective June 20, 2010

2. Topics • Part 200 Final Regulations for Standards and Assessments - Analysis of Comments • Early Implications of NCLB Assessment System Requirements on CAT • NCLB Assessment System Requirements • Evidence of Comparability • Beyond NCLB • References

3. Part 200 Final Regulations for Standards and Assessments - Analysis of Comments

4. Part 200 Final RegulationsAnalysis of Comments • NCLB requires that all students be held to the same achievement standards. This means that assessments for a particular grade and subject must elicit comparable content knowledge and understanding, within the framework of the standards, for all students tested.

5. Part 200 Final RegulationsAnalysis of Comments • Individual level assessments (adaptive assessments) would measure the performance of some students at a particular grade level against lower standards. • This would result in some schools being held to lower standards than other schools. • Use of levels assessments would not allow all schools and students to be held to the same high standards required by the NCLB Act.

6. Implications of NCLB Assessment System Requirements on CAT

7. Implications of NCLB Assessment System Requirements on CAT • Early in NCLB “Blended Format” combined • On-grade level section used for AYP and • Computer-adaptive section for instruction • Used on-grade level section for AYP to • Ensure all students and schools were being held to the same content standards and • Same academic achievement standards

8. Implications of NCLB Assessment System Requirements on CAT • Oregon currently is the only State that has received approval from the U.S. Department of Education through the peer review process to use CAT for AYP designation and to meet NCLB assessment requirements. • Some assessment requirements seem more complex and challenging when using CAT!!

9. NCLB Assessment System Requirements

10. NCLB Assessment System Requirements • NCLB assessment system requirements as provided in the Standards and Assessment Peer Review Guidance (January 2009) apply equally to fixed form and adaptive tests. • All forms are aligned with the State’s “grade level” academic content and achievement standards and yield comparable results; • All forms are equivalent to one another in terms of content coverage, difficulty, and quality; • All assessments yield comparable results for all subgroups;

11. NCLB Assessment System Requirements • If the assessment system includes various instruments (e.g., the general assessment in English and either a native-language version or simplified English version of the assessment), does the State demonstrate comparable results and alignment with the academic content and achievement standards;

12. NCLB Assessment System Requirements • When different test forms or formats are used, the State must ensure that the meaning and interpretation of results are consistent; • If the State administers both an online and paper and pencil test, has the State documented the comparability of the electronic and paper forms of the test;

13. NCLB Assessment System Requirements • Are the assessments and the standards aligned comprehensively, meaning that the assessments reflect the full range of the State’s academic content standards; • Do the assessments yield scores that reflect the full range of achievement implied by the State’s academic achievement standards?

14. NCLB Assessment System Requirements • The following are among the requirements that Oregon met to achieve full assessment system approval via ED’s peer review process: • Adaptive tests are comparable to each other and paper/pencil versions at the achievement levels, restricted to grade-level content, and matched to detailed grade-level test blueprints.

15. NCLB Assessment System Requirements • Requirements met Oregon (cont.) • Consistency of strand content among the paper-and-pencil and computer-adaptive versions documented with detailed test form construction rules and test maps. • Comparability of paper-and-pencil and computer-adaptive test difficulties between school years. • Evidence of comparability among English and side-by-side dual-language science test items and forms (for both Spanish and Russian).

16. Evidence of Comparability

17. Evidence of Comparability • Content • Adaptive and paper tests are built to the same grade level test specifications and blueprints with weighted content allocations by strand • Consistency of strand content among the paper-and-pencil and computer-adaptive versions was adequately documented by detailed test form construction rules and test maps

18. Evidence of Comparability • Content • Oregon submitted a consistency of strand content in computer administered tests study • Each test record contains a count of operational items by strand. • Oregon’s adaptive test delivery algorithm was designed to constrain the number of items by strand to match the test blueprints. • Study supported content comparability at the strand level.

19. Evidence of Comparability • Difficulty/Scale Comparability • Research study indicated: • There were no significant differences in the grade levels mean scale scores across adaptive and paper administered tests and • The correlations were moderate-high (0.65 to 0.84) for the double tested group.

20. Evidence of Comparability • Difficulty/Scale Comparability • Comparability of paper and adaptive test difficulties between school years, Oregon submitted external pilot and follow up studies on year-to-year comparability of Rasch parameter estimates. Both studies show highly consistent parameter estimates and support comparability between years.

21. Beyond NCLB

22. Beyond NCLB • ED supports the development and use of a new generation of assessments that are aligned with college- and career-ready standards, to better determine whether students have acquired the skills they need for success. (Blueprint) • New assessment systems will better capture higher-order skills, provide more accurate measures of student growth, and better inform classroom instruction to respond to academic needs. (Blueprint)

23. Beyond NCLB • Race to the Top Assessment Program - The Department published the notice inviting applications (NIA) provides • A flavor of some of the things that will be considered in assessments under the reauthorization blueprint, subject to Congressional approval

24. Beyond NCLB • Adaptive testing is a means not an end. • No one knows the implications of the ESEA reauthorization for assessment systems or what roles CAT may play. • If subsequent ESEA assessment requirements can be achieved using CAT, I believe ED will be encouraging.

25. References • Title I--Improving the Academic Achievement of the Disadvantaged, 34 CFR Part 200, Final Regulations for Standards and Assessment (July 5, 2002) • Standards and Assessments Peer Review Guidance: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001. (January 12, 2009) http://www2.ed.gov/admins/lead/account/saa.html#peerreview • A Blueprint for Reform - The Reauthorization of the Elementary and Secondary Education Act http://www2.ed.gov/policy/elsec/leg/blueprint/blueprint.pdf • Race to the Top Assessment Program... The Department published the notice inviting applications (NIA) for the fiscal year2010 Race to the Top Assessment Program competition (including the ...http://www2.ed.gov/legislation/FedRegister/announcements/2010-1/012710c.html