Models for Aligning Assessments to Standards

Models for Aligning Assessments to Standards Consortia Conference Call August 23, 2005 Regie Stites SRI International

Assessment Limits Some (more) Chinese sayings: “Viewing the heavens through a bamboo tube, Measuring the ocean with a spoon.” “Riding a horse, viewing flowers.” “Frog sitting in a well, viewing the sky.”

Accountability Demands An American saying: “What gets tested, gets taught.” When West meets East, we’ve got a world of trouble unless … • everyone understands the limits of assessment, and • we achieve reasonably good alignment of standards, assessment, and instruction.

Two Directions for Alignment Horizontal – align content standards to assessments, match content and depth Vertical – align accountability tests to educational systems (curricula, materials content, instruction, student outcomes, stakeholder opinions)

How to Align Tests and Standards:Three Methods • Sequential – develop content standards and curriculum, then develop standards-based tests • Expert review – use expert judgments to evaluate alignment, then select/reject test or explicate alignment • Content analysis – use content criteria to guide formal analysis, then select/reject test or explicate alignment

How to Analyze Alignment: Four Expert Models • ‘Webb’ – procedures and criteria developed by Norman Webb • ‘SEC’ – Surveys of Enacted Curriculum • ‘Achieve’ – alignment analysis service provided by Achieve, Inc. • ‘CBE’ – technical assistance from the Council for Basic Education See ‘Dimensions of Comparison’ on page 6 of Models for Alignment Analysis and Assistance to States, CCSSO (2002)

How Alignment Analysis Helps • Provides feedback for revision of assessments or standards • Provides information to help align instruction with assessments and standards • Helps everyone understand the proper limits for interpreting results of accountability tests

Metaphors for Alignment Goals Alignment as Congruence – assessment and standards content mirror each other Alignment as a Set of Correspondences – assessment content samples from standards content Alignment as a Bridge –curriculum content connects standards and assessments Alignment as Gravitational Pull – standards, assessments, and instruction all reflect common elements of a reform agenda [from Baker, 2004]

Achieve’s Alignment Protocol:Four Dimensions • Content centrality – test item match to content of related standard • Performance centrality – test item match to the type of performance (cognitive demand) of related standard • Challenge – source and level of challenge for sets of items • Balance and range – match between emphasis and coverage of sets of items and standards

Achieve’s Alignment Protocol:Three Stages Stage 1: Item-by-item analysis 1(a) Confirming the test blueprint 1(b) Content centrality 1(c) Performance centrality Stage 2: Challenge 2(a) Source of challenge (each item) 2(b) Level of challenge (sets of items mapped to each standard) Stage 3: Balance and range (sets of items mapped to each standard) 3(a) Balance (emphasis) 3(b) Range (coverage)

Achieve’s Alignment Protocol:Ratings Stage 1: Content and performance centrality ratings = 2 clearly consistent; 1A not specific enough; 1B somewhat consistent; 0 inconsistent Stage 2: Source of challenge rating = 1 appropriate source(s); 0 inappropriate source(s) Stage 2: Level of challenge rating = narrative evaluation Stage 3: Balance rating = narrative evaluation Stage 3: Range rating = fraction of total objectives (indicators) mapped to standard assessed by at least one item [above .67 = good, between .50 and .66 = acceptable]

Achieve’s Alignment Protocol:Results Elementary English Language Arts Content centrality (% of items rated 2) State A = 100; State B = 89; State C = 66; State D = 46; State E = 29 Performance centrality (% of items rated 2) State A = 100; State B = 79; State C = 62; State D = 79; State E = 21 Source of challenge (% of items rated ‘appropriate’) State A = 90; State B = 76; State C = 89; State D = 88; State E = 79 Average range scores State A = .27; State B = .75; State C = .73; State D = .52; State E = .31

Some Resources Achieve alignment procedures Robert Rothman et al. (May 2002). Benchmarking and alignment of standards and testing. http://www.cresst.org/Reports/TR566.pdf Metaphors for alignment Eva Baker. (December 2004). Aligning curriculum, standards, and assessments: Fulfilling the promise of school reform. http://www.cresst.org/reports/r645.pdf Test publishers see more alignment than teachers Chad Buckendahl et al. (2000). Alignment of standardized achievement tests to state content standards: A comparison of publishers’ and teachers’ perspectives. http://www.unl.edu/BIACO/NCME/buckendahlcw2.pdf ‘Nine requirements’ for a responsible state assessment system Commission on Instructionally Supportive Assessment. (October 2001). Building tests to support instruction and accountability: A guide for policymakers. http://www.aasa.org/issues_and_insights/assessment/Building_Tests.pdf

Resources Continued • Horizontal and Vertical AlignmentBetsy Chase and Sasha Zucker, Harcourt Assessments, Inc. • Rethinking Issues of Alignment Under No Child Left BehindSri Ananda, WestEd.

Discussion Points • What are your goals for alignment? What do you want to learn from an alignment analysis? What will you do with the results of an alignment analysis? • help educators and stakeholders interpret results of accountability tests? • influence instructional content and practice? • shape policy?

Models for Aligning Assessments to Standards