Assessment Literacy Module

1. California Department of Education Tom Torlakson, State Superintendent of Public Instruction Assessment Literacy Module Unit 6: Identifying, Developing, and Implementing Sound, Quality Assessments

2. Welcome to Unit 6 The purpose of this unit is to help educators identify, select, and develop high quality classroom assessments, implement them at multiple levels to continuously monitor student progress, and increase the accuracy of student performance information.

3. Learning Objectives for Unit 6 By the end of this unit, participants will be able to: Evaluate the quality of assessments and their items Recognize when linguistic complexity makes assessment unfair Describe a variety of checking for understanding strategies

4. Identifying, Selecting, and Developing Appropriate Classroom Assessments Refer again to the teaching-assessment cycle and the assessment literacy attribute to be covered first in this unit:

5. Identifying, Selecting, and Developing Appropriate Classroom Assessments Key Concepts in Accurately Measuring Student Performance Educators need accurate and relevant student performance information from high-quality assessments to make informed instructional decisions. The quality of assessments rests on three key concepts: validity, reliability, and fairness.

6. Key Concepts in Accurate Measurement: Validity Validity Validity is the extent to which inferences and actions made on the basis of assessment results are appropriate and backed by evidence (Messick, 1989). It is the use and/or interpretation of assessment data that is key to the concept of validity. The validity of any assessment is the extent to which test scores accurately reflect the relevant knowledge and skills of test takers. Discuss: What are the actions and decisions you make based on students’ assessment results? Do any of the actions/decisions call for more valid assessments than others?

7. Key Concepts in Accurate Measurement: Validity Validity For the Smarter Balanced assessments, the relevant knowledge and skills are defined by the CCSS. These assessments are developed using the principles of Evidence-Centered Assessment Design (ECD). The three basic elements of ECD are: (1) stating the claims to be made about test takers, (2) deciding what evidence is required to support the claims, and (3) administering test items that provide the required evidence (Mislevy, Steinberg, & Almond 1999). Additional information about ECD is available at http://padi.sri.com/downloads/TR9_ECD.pdf

8. Key Concepts in Accurate Measurement: Reliability Reliability Reliability is the degree of confidence that both scores and student performance are repeatable over time and across different circumstances. Repeatability of scores means that different scorers or the same scorer at different times should assign the same score(s) to the same piece of student work (O’Neill & Stansbury 2000). Discuss: How can teachers ensure that their classroom assessments are reliable?

9. Key Concepts in Accurate Measurement: Fairness Fairness “Test scores are fair when they yield score interpretations that are valid and reliable for all students taking the test. Regardless of race, national origin, gender, or disability, academic tests must measure the same knowledge of content for all students who take the test. Test scores must not systematically underestimate or overestimate the knowledge of students of a particular group.” ‒California Education Code Section 60208

10. Key Concepts in Accurate Measurement: Fairness Fairness For the Smarter Balanced assessments, the Bias and Sensitivity Guidelines helps to ensure that the evidence provided by the items means the same thing for various groups of test takers and allows the ECD to work as intended (Educational Testing Service 2010). For additional information, download the Smarter Balanced Assessment Consortium: Bias and Sensitivity Guidelines at http://www.smarterbalanced.org/wordpress/wp-content/ uploads/2012/05/TaskItemSpecifications/Guidelines/BiasandSensitivity/BiasandSensitivityGuidelines.pdf

11. Assessment Fairness and Bias Educators have the greatest opportunity to improve their assessments and directly impact students by increasing the fairness of their classroom assessments. “Assessment bias occurs whenever test items offend or unfairly penalize students for reasons related to students’ personal characteristics, such as their race, gender, ethnicity, religion, or socioeconomic status” (Popham 2003).

12. Assessment Fairness and Bias Fairness includes: 1. Instructional Learning Targets: Communicating clear, specific learning targets to students ensures they know what will and will not be assessed, what method will be used, and how the assessment will be scored. 2. Opportunity to Learn: Providing all students equally with adequate time and appropriate instruction will enable them to obtain mastery. 3.Prerequisite Skills and Knowledge: Ensuring students have the prerequisite skills and knowledge is necessary to complete a task, but are not the learning target itself.If learning targets are NOT related to writing or reading comprehension, this source of unfairness must be reduced by: a.Identifying the prerequisite skills and knowledge of the learning target b. Pre-assessing students’ abilities in the prerequisite skills and knowledge, either formally or informally through assignments, observations, or questioning Source: McMillan 2000

13. Assessment Fairness and Bias Respond to the following questions: 1. How does bias, opportunity to learn, and prerequisite skills and knowledge impact students and the way they are assessed? 2. What groups of students are most impacted when assessments are not fair? 3. What action can you take to ensure that all students have adequate time and appropriate instruction to enable them to achieve mastery? 4. What steps can you take to minimize bias in your assessment?

14. Assessment Accessibility To ensure that assessments are fair and accessible, it is essentialto allow all students to show what they know and can do, particularly students with special learning needs such as students with disabilities and English learners. In California classrooms, a Universal Design approach is being used to modify instruction according to IEPs, 504 plans, and language needs. Applying some of these processes and approaches to assessment takes time and practice.

15. Assessment Accessibility Smarter Balanced uses an “Access by Design” approach, which includes an array of universal digital tools and features available to all students. Examples include: Text-to-speech administration of the mathematics test and ELA items, screen magnification, guided line readers, and color options designed to increase contrast and readability. IEPs or 504 plans may specify tools and features such as keyword translation glossaries, auditory calming, audio captioning, administration in American Sign Language, or braille administration and response formats. Additional information about Universal Design is available on The National Center on Universal Design for Learning Web site at http://www.udlcenter.org/aboutudl.

16. Linguistic Accommodation for English Learners “Linguistic accommodation” is a research-based approach in which the language in test items, directions, and/or selected response choices is changed in ways that clarify and simplify the text without simplifying or significantly altering the learning target. It is especially useful with assessments in content areas such as math, science, social science, and technical subjects when language may be irrelevant to the learning target. To facilitate comprehension, the complexity of text is reduced by shortening sentence length and complexity, using common or familiar words and concrete language. Source: Abedi, Courtney, Mirocha, Leon, & Goldberg 2005; Abedi, Lord, & Plummer 1997; Rivera & Stansfield 2001

17. Linguistic Accommodation for English Learners The Linguistic Accommodation approach is aligned with the Universal Design used in Smarter Balanced assessments. Considering that 37 percent of California’s students come from families that speak a language other than English, it is important for educators to know the impact of linguistics on classroom assessment (CalEdFacts2011).

18. Linguistic Accommodation for English Learners Watch Kenji Hakuta of Stanford University discuss how the needs of English learners are being considered in the CCSS and the creation of new assessments: “Kenji Hakuta on ELLs and the Common Core Standards” http://www.youtube.com/watch? eature=player_embedded&v=PY2fhP8Q0OY

19. Limited English Proficiency Affects Learning and Assessment “While ELL students are struggling to learn English, learning content-based knowledge cannot occur at the same rate as for a native speaker of English when that instruction is offered only in English. Limited English proficiency may also make it difficult for ELL students to benefit fully from the teacher's instructions and to understand assessment questions. Therefore, limited English proficiency affects both learning and assessment. To help close the performance gap between ELL and non-ELL students both learning and assessment conditions must be addressed." ‒Abedi 2007 Key question in deciding when to use linguistic accommodations in creating tests: Is the learning target in ELA or another content area?

20. Limited English Proficiency Affects Learning and Assessment If the learning target is not in language, assessment-literate educators use strategies to reduce the interference of unnecessarily complex language in test directions and test items. Even when language is the learning target, assessment-literate educators make test directions clearer and simpler.

21. Limited English Proficiency Affects Learning and Assessment Refer to the table of interferences and accommodation strategies handout and complete the steps on the next slide: Handout Analyzing Linguistic Interferences and Accommodations

22. Limited English Proficiency Affects Learning and Assessment 1. Look at the Examples of Interferences on the next slide. If an example matches an “Interference” in Column 1 of your handout, write the letter of the example into the adjacent cell in Column 2. Some examples will be used more than once. 2. In Column 4, use the corresponding “Accommodation Strategy” from Column 3 and write in your idea to fix the Interference. 3. Compare your answers to the Completed Table handout.

23. Limited English Proficiency Affects Learning and Assessment Examples of Interferences: A. Alba needed to know about how much the sum of 19.6, 23.8, and 38.4 is. She correctly rounded each of these numbers to the nearest whole number. What three numbers did she use? B. Ted can no longer drive over 40 mph in his truck. C. The weights of two objects were measured. D. As long as you bring your own bedding, you can stay with us. E. According to the article, what role did some Navajo speakers play during World War II?

24. Limited English Proficiency Affects Learning and Assessment Compare your responses to the Completed Table Discuss findings and comparisons within your group. Handout Completed Table: Analyzing Linguistic Interferences and Accommodations

25. Limited English Proficiency Affects Learning and Assessment Important: Linguistic accommodation is not in conflict with the CCSS’ emphasis on text complexity and academic language because it is NOT about “dumbing down” texts. It IS about access to the CCSS for ALL students. The challenges for assessment-literate educators are: 1. To be clear about when language is the learning target and when it is not; 2. To recognize and eliminate unnecessarily complex language. By using linguistic accommodation strategies to develop and improve assessments, assessment-literate educators allow students to dedicate their attention to show what they know instead of translating text (Abedi, Lord, & Plummer 1997).

26. Limited English Proficiency Affects Learning and Assessment To explore accommodations options for your own classroom, use a new copy of the Interferences and Accommodations handout, and try the following: 1. Choose an assessment you are currently using. Look at the directions, questions, and answer choices (if applicable). Using the list of Interferences, how many can you spot? Use the list of strategies to edit the assessment. Be clear about what is being assessed — language or another content area? 2. For students with communication disorders and/or reading disabilities, which linguistic accommodations are appropriate for making assessments more accessible? Discuss how to address linguistic accommodations in IEPs or 504 Plans. This is especially critical for English learners with disabilities. Handout Interferences and Accommodation Strategies Template

27. Variations, Accommodations, or Modifications Becoming familiar with the principles of UDL, the diverse needs of students, and the various accessibility options available is the first step that educators can take to choose fair and accessible assessments — so that ALL students have the opportunity to show what they know and understand. The choice of variations, accommodations, or modifications in classroom assessments must be made thoughtfully, based on the students’ individual needs. If students with disabilities are entitled to variations in state testing, these variations should be used routinely in the classroom. Assessment-literate educators understand the test variations indicated for their students AND are able to carry out these variations in their classroom instruction and assessment.

28. Selecting and Developing Appropriate Classroom Assessments: Assessment Methods Now that you understand the key concepts behind measuring student learning, you are ready to develop sound, quality assessments. In Unit 2, you learned about two basic types of assessment methods that might be used in your classroom to gather information about student learning: selected-response and constructed-response.

29. Selecting and Developing Appropriate Classroom Assessments: Assessment Methods Selected-Response Methods In this short video, W. James Popham explains relevant features of selected-response assessment methods: “Selected-Response Assessment” http://www.k-state.edu/ksde/alp/module6/ Selected-response (SR) methods prompt students to select one or more correct responses from a set of choices. Carefully constructed SR items allow students to demonstrate complex thinking skills such as developing comparisons or contrasts; identifying cause and effect; identifying patterns or conflicting points of view; categorizing, summarizing, or interpreting information.

30. Selecting and Developing Appropriate Classroom Assessments: Assessment Methods Constructed-Response Methods In this short video, W. James Popham explains relevant features of constructed-response assessment methods: “Constructed-Response Assessment” http://www.k-state.edu/ksde/alp/module8/ Constructed-response (CR)methods prompt students to generate a text or numerical response in order to collect evidence about their knowledge or understanding of a given assessment target. CR items can be both short and extended. Short items may require test takers to enter a single word, phrase, sentence, number, or set of numbers, whereas extended items require more elaborated answers and explanations of reasoning. These kinds of CR items allow students to demonstrate their use of complex thinking skills.

31. Selected-Response vs. Constructed-Response Assessment Methods Meet Mr. Bumble. Mr. Bumble is a hard-working “every teacher.” He struggles with how to assess his students and often shows you what not to do. You will learn about Mr. Bumble’s exploits throughout this unit. Mr. Bumble realizes that he needs to improve his classroom assessments. He thinks the way to do this is to choose between selected- or constructed-response methods. So, he compares the two to find out which one is better for classroom assessments.

32. Selected-Response vs. Constructed-Response Assessment Methods EUREKA! Mr. Bumble realizes there are advantages and disadvantages to both methods. Now he’s confused about when he should use which one...

33. Selected-Response vs. Constructed-Response Assessment Methods If Mr. Bumble asked you which method to use, how would you answer him? Refer to the Target-Assessment Method Match table handout from Unit 4. Recall that the key determination in selecting the type of assessment method is how well it measures the type of learning target. Handout Target-Assessment Method Match (from Unit 4)

34. Developing Better Assessments Assessment-literate educators develop assessments that are appropriate and yield accurate information on which they can make sound instructional decisions. Popham (2003) alerts educators to: These roadblocks apply to any type of item.

35. Developing Better Assessments Now let’s look at the directions and some of the items from one of Mr. Bumble’s science tests. As you review them, identify how many roadblocks you find. Read sections of Mr. Bumble's exam on the following slides and then learn about the roadblocks they contain, along with tips for overcoming them. To review the roadblocks and tips after the workshop, refer to the online module at http://www.myboe.org/portal/default/Content/Viewer/Content?action=2&scId=505706&sciId=15350

36. Mr. Bumble’s Science Exam: African Elephant Unit Directions: "This exam consists of four parts. Each part contains different types of items (for example: true/false, multiple-choice, and essay). Each of the exam’s four parts covers one of the major units you have studied. Work your way through the test efficiently because there is a time limit. Don’t write on the test. Good luck!" Discuss the roadblocks that you encounter.

37. Mr. Bumble’s Science Exam: African Elephant Unit True/False: T F "Several research studies show that adult elephants always become domineering toward younger elephants because of their inherited characteristics." Multiple Choice: The commonly recognized example of a pachyderm is an a. elephant b. turtle c. lion d. pigeon

38. Mr. Bumble’s Science Exam: African Elephant Unit Essay Prompt: “Discuss elephants in Africa." Apply the item-writing rules you have learned so far to fix Mr. Bumble’s essay question. Then, compare your response to how the expert would fix it.

39. Mr. Bumble’s Science Exam: African Elephant Unit To develop high quality assessments, assessment-literate educators review all items and directions from the perspective of students. Answering essay questions mentally or in writing can help to uncover problems in advance. For more information on developing assessments, including matching learning objectives with test items and writing test questions aligned with Bloom’s Taxonomy, see “Is This a Trick Question? A Short Guide to Writing Effective Test Questions” at http://www.k-state.edu/ksde/ alp/resources/Handout-Module6.pdf.

40. Instructionally Embedded Formative Assessment: Checking for Understanding Strategies Here is where we are in the teaching-assessment cycle and the assessment literacy attribute that will be covered.

41. Sources of Assessment Data The graphic below shows the varying frequency and duration of teaching-assessment cycles. This section addresses an effective assessment strategy for shorter-cycle, daily, and minute-by-minute formative assessment. Checking for understanding provides useful assessment information for adjusting instruction in real time. Sources of Assessment Data Source: Heritage 2010

42. The Benefits of Checking for Understanding Checking for understanding is a formative assessment practice where educators ask questions or give short assignments and students select or construct their responses. Assessment-literate educators use “end of class” formative assessment results to guide planning for the next day’s lesson, or embed the assessment seamlessly into instruction to make adjustments “in the moment.”

43. The Benefits of Checking for Understanding The benefits of checking for understanding when responses are obtained from all students: Gives educators information quickly about the need to differentiate instruction Helps educators understand students’ background knowledge, skills, and misconceptions Models for students how to monitor their own understanding The biggest payoff is the positive effect on student achievement. Research findings support the boost in student achievement when educators pause to check for understanding and use the information to adjust their teaching (Fisher and Frey 2007).

44. The Benefits of Checking for Understanding Watch this scene in Mr. Bumble's math class right after he completes an explanation of the statistical concept of the mean. Is he really checking for understanding? “Mr. Bumble Checks for Understanding” http://goanimate.com/videos/08KkNc8Zt5_E?utm_source=linkshare Later, when Mr. Bumble reviews the results of the unit test, he is surprised to see that only one or two students understood the concept of the statistical mean. Where did Mr. Bumble go wrong? What assumptions did he make?

45. The Benefits of Checking for Understanding Nonverbal checks for understanding in the formative assessment process include observing facial expressions and body language, as well as establish classroom routines such as hand signals, colored cards, or answers to selected-response questions. What nonverbal cues can students give spontaneously that indicate whether they understand?

46. The Benefits of Checking for Understanding Watch how an assessment-literate educator checks for understanding nonverbally on a minute-by-minute basis. “Show your Cards” http://successatthecore.com/teacher-development/featured-video.aspx?v=41 How is the data gathered from colored cards used to adjust instruction?

47. Checking for Understanding on a Daily Basis The next four videos show how assessment-literate educators can check for understanding on a daily basis where students respond verbally or in writing. • As you watch, think about: • How does the teacher ensure that checking for understanding is a low-stress formative assessment? • How can the data gathered from the assessment be used to adjust instruction? • How are students benefiting from the strategy? What feedback are they getting? • How would you use the strategy?

48. Checking for Understanding on a Daily Basis 1. Stoplight Watch an ELA teacher use exit slips to assess learning and how students self-assess their learning: “The Stoplight Method: An End-of-Lesson Assessment” https://www.teachingchannel.org/videos/daily-lesson-assessment 2. Exit Cards How does this strategy help teachers assess learning and plan future math lessons? “Assess and Plan with Exit Tickets” https://www.teachingchannel.org/videos/teacher-assessment-strategy?fd=1

49. Checking for Understanding on a Daily Basis 3. Tiered Exit Cards Watch the video to see how a secondary math teacher uses differentiated exit cards to group students for the next day. “Daily Assessment with Tiered Exit Cards” https://www.teachingchannel.org/videos/student-daily-assessment 4. Text What You Learned A Secondary English teacher uses texting to check for understanding. “Text What You Learned: Using Technology to Assess” https://www.teachingchannel.org/videos/texting-to-assess-learning

50. Using Think-Pair-Share to Check for Understanding “Think-Pair-Share” is a peer-to-peer discussion strategy where students discuss their constructed responses to a prompt or question before sharing with the whole class. It is a strategy that engages every student so that checking the understanding of the whole class can be done efficiently. The following steps outline the strategy (Fisher & Frey 2007). 1. Pose a question, prompt, reading, visual, or observation. 2. Allow a few minutes to individually THINK about the question. 3. PAIR up students with designated partners to discuss their respective responses, comparing thoughts and agreeing on the best responses. 4. After an appropriate length of time, ask the pairs to SHARE their thinking with the whole class. Students need specific feedback on speaking and listening especially since these skills do not always have products for reviewing.