Bias and Sensitivity Review of Items for the MSP/HSPE/EOC

1. August, 2012 ETS Olympia Bias and Sensitivity Review of Items for the MSP/HSPE/EOC 1

2. General WCAP Development Cycle Develop Item and Test Specifications Item Writing Training and Item Writing Item Reviews for Content by OSPI Item Pilot Testing Item Reviews for Content & Bias/Sensitivity by Committee Pilot Rangefinding Pilot Scoring Training and Pilot Scoring Data Review Operational Scoring Operational Rangefinding Operational Testing

3. Principles of Universal Design • Assessments should be designed to be valid and accessible for use by the widest possible range of students, including students with disabilities and students with limited English proficiency. • Results should not be influenced by disability, ethnicity, gender, or English language ability. 3

4. What is Item Bias? • Language or content in the item that prevents members of a group from demonstrating their knowledge and skills in a content area; and • Systematic error in the measurement process. 4

5. Potential Sources of Bias and Stereotyping • Regional and geographic differences in language usage and topic familiarity • Linguistic issues (e.g., idiomatic expressions or figurative language) • Gender and age stereotypes • Ethnic, cultural, and religious stereotypes • Socio-economic and occupational stereotypes 5

6. Sensitivity Issues Any reference or language in an item or passage that might cause a student to have an emotional reaction during the test administration and prevent the student from being able to accurately demonstrate knowledge and skills. Examples found in previous bias & sensitivity reviews: • Passage about abuse or struggles of an individual or group because of minority status • Change of meaning or structure in a culture’s legend or folk tale • Competitive context in passages or items • Controversial issue unrelated to the content standards • Use of patronizing language to describe an individual or group 6

7. How Might Potential Item Bias be Detected? • Judgmental Procedure: Review of item for issues related to context, language, or stereotyping prior to field-testing • Statistical Procedure: DIF (Differential Item Functioning) analysis following field-tests using item statistics 7

8. Purpose of Bias and Sensitivity Review • Review test items for potential sources of bias, sensitive issues, and gender or cultural stereotypes. • Apply professional test development standards to ensure items are fair and not insensitive or offensive. 8

9. Read the Bias & Sensitivity Review Guidelines Questions? Comments?

10. Guiding Questions • Is there anything controversial, inflammatory, or insensitive in a scenario, reading passage, or item context? • Is the language likely to be familiar to most examinees regardless of socioeconomic status, region, language proficiency, cultural background, or gender • Is there any apparent patronizing language, stereotypical descriptions, or stereotypical actions/beliefs present? • Would an item context or content give students of a particular group, background, or region a distinct advantage or disadvantage? 10

11. Sample Item #1 In the poem “The Bells,” what is the effect of phrases such as the jingling and the tinkling and themoaning and the groaning?  A. They indicate the actual sounds made by the bells.  B. They reflect the beauty of the sounds made by the bells.  C. They provide the required number of syllables in each line.  D. They show that the speaker is gradually losing touch with reality. Bias issue: not accessible to students who are deaf or hard of hearing 11

12. Sample Item #2 Which of the following measures could be the length of a typical Par 4 golf fairway?  A. 400 inches  B. 400 feet  C. 400 yards  D. 400 miles Bias issue: SES and/or prior knowledge 12

13. Sample Item #3 At a carnival, Rolando sees a booth that has a prize wheel. The wheel has 20 sections, and 5 of these sections are labeled ‘Winner.’ If Rolando decides to give the wheel booth a shot, what is the probability that he will win a prize?  A. 0.20  B. 0.25  C. 0.33  D. 0.75 Bias issue: Idiomatic Language “… give the wheel booth a shot…” 13

14. Practice Item Review • Read the items • Decide whether the content of the items have any of these problems: • Stereotyping or negative representation of any group • Context that is controversial or sensitive • Language or context that may be unfamiliar to a student because of region, language proficiency, socio-economic status, etc.

15. Do you need to review for: • Names of the children and adults in math and science items? NO, names are adjusted for gender and ethnic balance once tests are put together • Whether the concepts or skills are appropriate for students at the identified grade level? NO, content review committees check that items only measure on-grade level concepts and skills

16. How To Proceed • Read the passages, keeping in mind the guiding questions. • Make your own judgment. Write your judgment on the review form: • Accept as is (no apparent bias/sensitivity issue) • Edit • Reject due to potential bias or sensitivity issue • As a group, discuss each item flagged for revision or rejection. • Recorder will take detailed notes. • Make a group decision about each item. • Record dissenting position, if appropriate 16