New England Common Assessment Program

1. New England Common Assessment Program Grades 3-8 & 11 Bias/Sensitivity Review- Part 2 March 26 and 27, 2006 Quechee, VT

2. What Is Item Bias? • Bias is the presence of some characteristic of an assessment item that results in the differential performance of two individuals of the same ability but from different student subgroups • Bias is not the same thing as stereotyping (but we don’t want stereotypes in NECAP either) • We need to ensure that ALL students have an equal opportunity to demonstrate their knowledge and skills

3. The “Big Idea” of Bias-Sensitivity Review NECAP should measure the results of learning opportunities NOT life experiences

4. How Do We Prevent Item Bias? • Reading Passages Selected by Item Developers • Bias-Sensitivity Review of Reading Passages – Focus on Sensitivity Issues • Reading and Math Items are Developed • Bias-Sensitivity Review of Reading & Math Items – Focus on Item Bias • Item Review of Reading and Math Items • Field-Testing Feedback • Pilot-Testing Data Analysis (DIF)

5. Role of the Bias-Sensitivity Review Committee • Review reading passages, and then both math and reading items • Complete independent ratings of each passage or item and record comments • Be prepared to discuss “Mixed Reviews” and “Problem Items” with your review team; Finalize independent ratings Please Note: The Bias-Sensitivity Review Committee plays an advisory role. Final decisions on which passages and items to include in NECAP will be made by the NH, RI and VT DOEs

6. Role of the Bias-Sensitivity Review Committee The Bias-Sensitivity Review Committee DOES need to make recommendations that will help ensure… • Sensitivity to different cultures, religions, ethnic and socio-economic groups, and disabilities • Balanced gender roles that are free of gender stereotypes • Use of positive language, situations and images • Use of context, situations, images and examples that are universally understood across groups and cultures • Overall, that the test is free of items and text that may elicit strong emotions in specific groups of students, and as a result, prevent those groups of students from accurately demonstrating their skills and knowledge • Overall, that the test is free of items that measure life experiences instead of learning experiences

7. Role of the Bias-Sensitivity Review Committee • Reading Level • Grade Level Appropriateness • GE Alignment • Instructional Relevance • Language Structure and Complexity • Accessibility • Overall Item Design The Bias-Sensitivity Review Committee DOESNOT need to make recommendations concerning… The Item Review Committees Are Addressing These Issues and Factors

8. The Bias-Sensitivity Review Process • December 12th and 13th – Review All Reading Passages • March 26th and 27th – Review All Reading and Mathematics Test Items Process: • Today – (1) Training, (2) Practice, (3) Clarification of Guidelines and Procedures, (4) Break Into Two Groups for Independent Review of Item Sets – Group 1 will review reading and math for grades 3, 5 and 7 and math for grade 11, and Group 2 will review reading and math for grades 4, 6 and 8 and reading/writing for grade 11. • Tomorrow – (1) Break into Small Groups for Re-Evaluation and Discussion of “Problem Items” (2) Finalize Independent Ratings, (3) Evaluate the Review Process and Provide Feedback

9. General Guidelines • Avoid issues and topics considered sensitive by the general public • Keep in mind that some topics that are appropriate for classroom discussion may not be appropriate for assessment • Consider context and setting; Bias-Sensitivity Review is not a “literal” process • Think of “bias” as an assessment issue as well as a cultural issue, that is, a specific topic or feature of a passage or item that may disadvantage one of two otherwise equally matched groups of students

10. Abortion Birth Control Child Abuse/Neglect Creationism Divorce Incest Occult/Witchcraft Questioning Parental Authority Rape Religious Doctrine Sex/Sexuality Sexual Orientation Sexually Transmitted Diseases Suicide Weight Topics to Avoid in Reading Passages & Test Items

11. Death Drugs, Alcohol & Tobacco Family Issues Guns/ Gun Control Handicapping Conditions Holidays Homelessness Misuse of Animals/ Animal Rights Murder Pregnancy Racism, Sexism & Ageism Religion (Except in Historical Context) Violence Topics Which May or May Not Be Acceptable

12. Awareness Issues • Avoid economic, regional, cultural or gender bias in questions, prompts, identified situations, graphics or reading selections • Avoid profanity unless it has a clear literary or historical context • Keep in mind that the ability to answer a test question should reflect learning opportunities not cultural opportunities or life experience • Remember that we are proud of our states and region and expect to see that pride reflected in assessment passages and items

13. Item Review Rating Form New England Common Assessment Program Bias & Sensitivity Review Committee Item Review Evaluation Form Read the statement below and select the rating that best describes your evaluation of the item for potential bias and sensitivity concerns. “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Comments Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be impacted, and the extent of the potential impact within the subgroup(s).

14. Bias-Sensitivity Examples and Non-Examples Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives. Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.” “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree

15. What’s the Point? “Avoid ECONOMIC, regional, CULTURAL or gender bias in questions, prompts, identified situations…” Okay because the writer’s culture and economic status are not factors in the passage Not okay because the passage assumes a cultural and economic status that won’t be shared by all test-takers

16. Bias-Sensitivity Examples and Non-Examples “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”  Larry buys an iPod ® that has a regular price of $120. The iPod ® is on sale for 30% off. What is the sale price of the iPod ®?  A. $150  B. $84  C. $36  D. $90 Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree  Larry buys a sweater that has a regular price of $40. The sweater is on sale for 30% off. What is the sale price of the sweater? Show your work or explain how you know. Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree

17. What’s the Point? “Avoideconomic, regional, CULTURAL or gender bias in questions, prompts, identified situations…” “(Use) context, situations, images and examples that are UNIVERSALLY UNDERSTOOD across groups and cultures” Not okay because many students can’t afford luxury items such as IPods Okay because clothing is a necessity and $40 is a more reasonable amount

18. Bias-Sensitivity Examples and Non-Examples “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students” • Explain how the Model T differed from earlier attempts to build a “horseless carriage.” Use examples from the story. • (Related to a reading passage called “Henry’s Horseless Carriage.”) Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree • Explain how the antique cars described in the story are different than modern cars. Provide at least 3 examples. • (Related to a reading passage called “Henry’s Horseless Carriage.”) Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree

19. What’s the Point? “Avoideconomic, regional, CULTURAL or gender bias in questions, prompts, identified situations…” “(Use) context, situations, images and examples that are UNIVERSALLY UNDERSTOOD across groups and cultures” Okay because student is only required to use information contained in the passage. Not okay because the student needs both information from the story AND information from personal experience

20. Bias-Sensitivity Examples and Non-Examples “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree

21. What’s the Point? “We need to ensure that ALL students have an equal opportunity to demonstrate their knowledge and skills” “(Use) context, situations, images and examples that are UNIVERSALLY UNDERSTOOD across groups and cultures” Okay because the context is generally understood by all students. Not okay because some students (particularly VT & NH students) may have experience using the trail guide formula

22. Bias-Sensitivity Examples and Non-Examples “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students” • The word hear belongs in which sentence? •  A. People come ____ for vacation. •  B. I want to sit _____ by the window. •  C. Take cover when you _____ thunder. •  D. Please put my books over _____. Rating Descriptions ? 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree • The word hear belongs in which sentence? •  A. People come ____ for vacation. •  B. I want to sit _____ by the window. •  C. She can _____ the loud thunder. •  D. Please put my books over _____. Rating Descriptions 4 Strongly Agree ? 3 Agree 2 Disagree 1 Strongly Disagree

23. What’s the Point? “Avoideconomic, regional, CULTURAL or gender bias in questions, prompts, identified situations…” “(Use) context, situations, images and examples that are UNIVERSALLY UNDERSTOOD across groups and cultures” Is this NOT okay because thunder may not be “universally understood” by persons who are deaf? Is this better? Is it okay because the sentence refers to another person’s hearing, not the reader’s

24. The “Big Idea” of Bias-Sensitivity Review NECAP should measure the results of learning opportunities NOT life experiences