grades 3-8 ITEM WRITING

Mathematics Assessment grades 3-8 ITEM WRITING

Day 1 Item Writing Agenda • Welcome and Introductions • Forms and Paperwork • Group Norms • Item Writing for the State Assessment • Purpose • Importance of Validity • Procedures Followed and Documents Used to Support Validity • Item Writing—Multiple-Choice

Group Norms • Be on time • Use cell phones and personal electronic devices at breaks; inform us of personal family considerations • Keep side conversations to a minimum • Respect each other’s work styles (need for quiet) • Ask for and give feedback; share expertise • Other norms • Parking Lot

Purpose and Goals of Workshop • Write assessment items that are aligned to Washington mathematics standards and accessible to all students • Increase capacity to build a valid and reliable summative assessment that promotes mathematics achievement and rigor for all students (excellence and equity) • Involve Washington educators in the assessment process • Provide participants with knowledge to share item writing process with others

Purpose of Assessment • To improve student learning of content standards through improved instruction based on assessment results • To inform teachers/students of their progress • To inform the public about school performance (accountability) • To guide decision making about students, teachers, or schools • To provide data comparisons

Importance of Validity • “The most important characteristic of any assessment procedure is its impact on validity.” (p. 6) • Validity is defined as “the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of tests” (AERA, APA, and NCME, 1999, p. 9). • Strong validity allows for making inferences about the meaning of performance from year to year.

Procedures to Support Validity • Start with a clear explanation of what students are intended to know and be able to do (Mathematics Learning Standards) • Control irrelevant sources of variability by: • Standardizing tasks to be performed • Standardizing conditions under which tasks are to be performed • Test window • Tools and manipulatives allowed, etc. • Standardizing criteria used to interpret the results • Accepted statistical practices

Operational Range Finding Reported Results Operational Scoring Standardizing Tasks to be Performed OSPI, WA Educators Expert Consultants WA Bias/Fairness: OSPI, Contractor, WA Citizens Content Reviews: OSPI, Contractor, WA Educators OSPI, WA Educators Contractor Standards Test and Item Specifications Item/Rubric Development Item/Rubric Operational Reviews Item Bank Pilot Item and Pilot Item Data Reviews Pilot Item Scoring OSPI, Contractor WA Educators Pilot Item Range Finding Scoring Contractor Approximately 18 months WA Educators, OSPI, Contractor

Standardizing Tasks: Item/Rubric Development • Alignment to Standards • Test and Item Specifications Document • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Alignment to Cognitive Complexity • Item Writing Guidelines Document

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Alignment to Standards • Alignment Study • Independent contractor evaluated content and cognitive complexity of test items • Findings of Alignment Study and all processes followed for new mathematics standards will be submitted for peer review March 2011 • Use the wording and intent of the Standards; recorded in Item Specifications

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item Specifications Purpose: “The guidelines in this document assist in writing items that match the performance expectations and their associated restrictions. Restrictions are necessary to construct a valid and reliable on-demand assessment. These restrictions are not necessary in classroom based assessments.”

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item Specifications • Test Development Guidelines: • Compare and contrast grades 3-5 document with grades 6-8 document • Cognitive Complexity • Parallel Sections • Paragraphs (summarize intent of standards) • Test organization/Test Map • Areas of Emphasis • Item Types • Test Map (timing) • Item Specifications-organization and key

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item SpecificationsGrade 3 example

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item Specifications Grade 3 example

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item Specifications Grade 7 example

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Test and Item Specifications, cont. • Vocabulary First Used in Assessment Items at Grade Level • Scavenger Hunt • Become familiar with organization, rules, and restrictions in grade level item specifications

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Debrief • Parking lot questions • Questions about content of morning information and/or activities

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Plain Language • Researchers define “plain language” as “text-based” language that is straightforward, concise, and uses everyday words to convey meaning. • The goal of plain language editing strategies is to improve the comprehensibility of written text while preserving the essence of its message.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Plain Language, continued Plain Language Editing Strategies: • Reduce excessive length. • Eliminate unusual or low frequency words and replace with common words. • Avoid ambiguous words and words with varied meanings. • Avoid irregularly spelled words.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Plain Language, continued Edit the following item using Plain Language Strategies. Samantha Marie has a red sandbox in her back yard. She wants to collect shells to put in her sandbox. How much sand does she need to fill the box halfway full? The sand box is 6’ on each side and 16” tall.  A. 36 cubic feet  B. 18 ft  C. 24 feet 3 3

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Vocabulary • Items use language targeted to the previous grade level or lower readability, except for required mathematics terms listed in the Test and Item Specifications. • All mathematics vocabulary and terms allowed in items for all previous grade levels may also be used. • Measurement vocabulary in charts.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines How “Vocabulary List” Is Used Item writer chooses to write for performance expectation 3.4.A “Identify ..parallel..lines.” “Sheila used parallel lines to draw a design. Which of the following lines are parallel?” • Item writer checks the reading level for: “following” • Check “Vocabulary First Used . . . “parallel, 3” picture. 1st “design” 4th 5th “parallel”

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Bias and Sensitivity • Bias is the presence of some characteristic of an item that results in differential performance for two individuals of the same ability but from different groups. • Sensitivity issues are references or language in an item or passage that might cause a student to have an emotional reaction during the test administration and prevent the student from being able to accurately demonstrate knowledge and skills.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Potential Sources of Bias and Stereotyping • Regional and geographic differences in language usage and topic familiarity • Linguistic issues (e.g., idiomatic expressions, figurative language, or words with multiple meanings) • Gender and age stereotypes • Ethnic, cultural, and religious stereotypes • Socioeconomic and occupational stereo types

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Potential Sources of Sensitivity Issues • Passage about abuse or struggles of an individual or group because of minority status • Deployment overseas • Competitive context • Controversial issue unrelated to the content standards: holidays, food, height, animal rights, etc. • Use of patronizing language to describe an individual or group

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Bias & Sensitivity Issues • What are some issues that might affect the students in your district? • Ownership • Houses/apartments • Junk food • Gender: distribute importance evenly, when appropriate • Socioeconomic issues: expensive items, etc.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Topics That Introduce Bias

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Sample Item #1 Which of the following measures could be the length of a typical Par 4 golf fairway?  A. 400 inches  B. 400 feet  C. 400 yards  D. 400 miles Bias issue: SES and/or prior knowledge

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Sample Item #2 At a carnival, Rolando sees a booth that has a prize wheel. The wheel has 20 sections, and 5 of these sections are labeled ‘Winner.’ If Rolando decides to give the wheel booth a shot, what is the probability that he will win a prize?  A. 0.20  B. 0.25  C. 0.33  D. 0.75 Bias issue: Idiomatic Language “… give the wheel booth a shot…”

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Principles of Universal Design • Assessments should be designed to be valid and accessible for use by the widest possible range of students, including students with disabilities and students with limited English proficiency. • Results should not be influenced by disability, ethnicity, gender, or English language ability.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Universal Design Assessments based on universal design principles offer a promising approach to providing optimal, standardized assessment conditions for all students, giving each student a comparable opportunity to demonstrate achievement of the standards being tested.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity • No Child Left Behind (NCLB) requires assessments to “measure the depth and breadth of the state academic content standards for a given grade level” (U.S Department of Education, 2003, p. 12) • NCLB requires states to develop test items aligned with cognitive complexity of performance expectation

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Item Content and Complexity • Content is the “what” of the item. • Complexity is what the student “does” with the content. • Level 1—Recall; perform a rote procedure • Level 2—Application of Skill/Concept • Level 3—Strategic Thinking • Level 4—Synthesizing information and creating new example (Classroom-based; long-term)

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 1 • This category relies heavily on the recall and recognition of previously learned concepts and principles. Items typically specify what the student is to do, which is often to carry out some procedure that can be performed mechanically.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 1 Examples • Recall or identify a fact, term, or property • Identify an example of a concept • Compute a sum, difference, product, or quotient • Identify an equivalent representation • Perform a specified procedure • Draw or measure simple geometric figures • Solve a one-step problem • Perform a single-unit conversion

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 2 • Items in the level 2 complexity category require basic application of skills and concepts to use information, conceptual knowledge, procedures, two or more steps, etc.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 2 Examples • Solve a word problem • Compare figures or statements • Provide reasons for steps in a solution process • Interpret a visual representation • Extend a pattern • Extract information from a graph, table, or figure and use it to solve a problem • Represent a situation mathematically in more than one way

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 3 • Level 3 complexity items require strategic thinking such as reasoning, developing a plan, or determining a sequence of steps to use.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Cognitive Complexity Level 3 Examples • Perform a procedure having multiple steps and multiple decision points • Solve a novel (non-routine) problem • Solve a problem in more than one way • Explain and justify a solution to a problem • Describe how different representations can be used for different purposes • Generalize a pattern • Analyze or produce a deductive argument • Create a representation of a problem situation

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Would you consider yourself a good tennis player if all you had was a killer forehand?

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increasing Cognitive Complexity • Non-routine problems • Application of skills in novel ways • Obtaining a solution from a different direction • Multiple decisions

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increasing Cognitive Complexity 3.2.H Solve single and multi-step word problems involving division • Sally has 25 cookies to share among 6 people. She wants each person to get the same number of cookies. How many cookies will be left after she shares all the cookies? • Sally has 25 cookies. She is storing them in boxes that hold 6 cookies each. How many boxes does she need for all the cookies?

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increasing Cognitive Complexity • Sally has 25 cookies. She is packing them in boxes of 6 cookies. How many more cookies does she need so that the last box she uses will be completely full?

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increasing Cognitive Complexity 6.1.H Solve single- and multi-step word problems involving operations with fractions. • Mr. Davis made 48 cookies. He sold ¼ of them. How many cookies did he sell? • Mr. Davis made 48 cookies. He sold ¼ of them and gave ¼ of the remainder to his neighbor. How many cookies did he have left for himself?

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increasing Cognitive Complexity • Mr. Davis had some cookies. He sold ¼ of them. After giving 5 of the remaining cookies away, he had 40 cookies left. How many cookies did Mr. Davis start with?

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Increase Cognitive Complexity • Warm up your item- writing brain by creating items that increase in cognitive complexity. Work as individuals, share ideas, and then work together. • Teachers grade 3-5 • Write to 4.1.I (or a PE of choice) • Teachers grade 6-8 • Write to 7.1.G (or a PE of choice) Share ideas with whole group.

Alignment to Standards • Test and Item Specs • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Cognitive Complexity • Item Writing Guidelines Why think about cognitive complexity? • What is tested is what gets taught. • Tests must measure what is most important.

We want our students to be powerful mathematical thinkers who can apply mathematics in a variety of situations.

Review of Morning • Importance of Validity • Ways to support validity • Standardize tasks • Review process (item cycle) • Alignment to standards and cognitive complexity • Test and Item specifications-rules and restrictions • Plain Language and Vocabulary • Bias and Sensitivity • Universal Design • Standardize conditions • Testing window, manipulatives allowed, etc. • Standardize interpretation of results

grades 3-8 ITEM WRITING