Scoring Open-Ended Literacy Items

1. ScoringOpen-Ended Literacy Items www.tlionline.net

2. Understand the importance of open-ended literacy items • Analyze prompts and rubrics for these items • Scoring tips • Relating Scoring to Classroom Instruction • Practice scoring open-response reading questions • Practice scoring writing samples • Share best practices Goals for the Session

3. Reading Open-ended Questions • 1 Open Response question for each passage • Total of 3 for ACTAAP tests grades 3-8 • Total of 6 for Grade 11 Literacy Exam • Account for 50% of score (24 points out of 48 for grades 3-8) Importance of Open-Ended Literacy Items

4. Writing Open-ended Tests • 2 questions (prompts) requiring essay-length responses • Account for 40 out of 48 points • That is 83% of total writing score in grades 3-8 Importance of Open-Ended Literacy Items

5. Open-ended questions account for 67% of the total literacy score • Students need to earn significant points in these items in order to achieve a proficient score • It’s critical that we score these items accurately so that the testing experience and data will be relevant in helping students improve Importance of Open-ended Literacy Items

6. Reading open response questions and writing prompts have little in common • Frustrating because teachers want students to express themselves well in all written responses • Difficult for teachers and students to accommodate the differences Few Similarities

7. The two reasons to score open-ended items are often at cross-purposes • To upload scores quickly to get data back • Scoring should be as accurate to the ACTAAP scoring as possible • With practice this can be done with a minimum of marks and time • To give meaningful feedback to students and parents • This scoring is more detailed, in-depth, and takes much longer Reasons to Score

8. District, or at least building, must decide whether to score interim assessments for quick, accurate scores to be uploaded for data purposes only or for meaningful in-depth scoring for feedback and reteaching • To do both will require delayed uploading, double scoring, or some creative time use Scoring Decisions

9. Writing: 5 domains on the state scoring rubric are used to assess writing. Content Style Sentence Formation Usage Mechanics Can the student express himself in written form? Reading: Content-specific rubric is used to assess the reading open response question. Scored for content only Did the student read and understand? Scoring Requirements

10. Reading Open-ended Items

11. Each reading passage has 1 open response question • The question will generally require more than one thinking skill, even at the lower grades: • Analysis • Inference • Evaluation • Comparison Open Response Reading Questions

12. Question is tied to the passage and requires information from the passage as support • It is a more in-depth question than can be covered by a multiple-choice format question. • Short answer • Scored on a 0 to 4 scale • TLI continues to use a B for Blank Open Response Reading Questions

13. Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage. • Grade 3, 2006 Examples of Open Response Questions

14. Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions. • Grade 5, 2006 Examples of Open Response Questions

15. Compare and contrast Sylamore Horse Trail and Ozark Bicycle Trail. Use specific examples from the passage to support your response. • Grade 11, 2006 Examples of Open Response Questions

16. Each open response reading question is scored by a specific rubric • Can be subjective • Was the response thorough or only adequate? • A variety of tasks requested Rubrics

17. Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage. • What does “describe” really mean? • How much description is enough? • How many details do we include? Analysis of Open Response Reading Questions

18. Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions. • Describe or tell? • How many details should support each idea? • What conclusions? Analysis of Open Response Reading Questions

19. Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task. Sample Rubric

20. Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task. Sample Rubric

21. With the 2007 test a new format was begun by the state • Old format is still in use • Most questions in 2007 and 2008 did follow the new format • Continue to use old format in practice • TLI now uses the 4 point format New Open Response Format

22. These questions are written so that there are 4 tasks to perform • The rubrics show score points equal to the number of tasks correctly performed • 4—all 4 tasks are completed correctly • 3—any 3 of the tasks are completed correctly • 2—any 2 of the tasks are completed New 4-point Open Response Format

23. Explain why it is important to guard against germs in the kitchen. Identify three ways to fight against infectious diseases in relation to food and its preparation. • Explain why • Identify way #1 • Identify way #2 • Identify way #3 Analysis of New Format Example

24. Most rubrics seem to accept any 3 tasks to score a 3, any 2 tasks for a 2 • Occasionally a rubric will be specific about the tasks. • For a 4, the response explains and gives 3 examples • For a 3, the response must explain and give 2 examples (3 examples without explanation may not be acceptable) Variations in Rubrics

25. Sample Open Response Items

26. Fragments OK • Charts or Diagrams OK • Lists OK • “Don’t be creative.” --ADE • Just answer the questions • Some questions have parts that should be labeled. Generally at lower grades. • Some questions have charts. The missing information should be listed in the answer box. OR Short Answers Encouraged

28. Some questions are difficult to answer with lists • If the question suggests discussion, sentences may be a better approach • Complete paragraph construction is never necessary • Scoring should never take anything into consideration except what is specifically called for in the rubric (either positively or negatively) Respond to Match the Question

29. Scorers should be very familiar with the passage and the question • Scorers should work the “problem” themselves, preferably more than 1 way • TLI provides possible answer information • Follow the rubric exactly, even if the response is unexpected. Give credit if tasks are accurately performed. Scoring Tips for Open Response Items

30. Scan the response, looking for answer information • Put a check-mark in the margin for each task performed • Tasks do not have to be placed in the order requested • If question divides answer and requests labeling, no points should be removed if student does not label Scoring Tips for Open Response Items

31. Score of B—blank, no attempt to answer • Pull these from the stack and score • Analyze why • Did student run out of time? • Did student waste time? • Was student unaware of question? • Did student refuse to attempt? • Does this reveal a hidden problem? Scoring Tips for Open Response Items

32. Score 0—Student attempted an answer but earned no points • Analyze why • Ran out of time and jotted something down • Did not understand the question • Was uncertain of format

33. For all responses that earn points, look quickly at why points were not earned • Only 3 tasks performed? Why not the other? • 4 tasks performed, but some incorrectly? • What could student do to earn just one more point?

34. Score and share

35. Do you know that open response question and rubric templates are on the TLI Web site? More OR Scoring Tips

36. DO teach students to analyze the Open Response prompt • DO teach the 4 types of Open Response Skills • Do expect “analysis” to be the underlying skill • DO expect 4 tasks on each question • DO teach students to score their own work—great practice • DON’T score for mechanics, sentence formation, and the like The Scoring-Teaching Connection

37. DO teach students to complete tasks in order • DO encourage labeling/numbering tasks • DO work with students to find ways to increase the score by 1 point • DO encourage students to look for clues within the question • DO encourage students to mark the passage to locate answer information for the question The Scoring-Teaching Connection

38. DO teach students to anticipate an OR question • DO teach students to write their own OR questions and rubrics • DO help students understand the difference between the reading open response and a writing assignment: Short box= short answer= score for content only The Scoring-Teaching Connection

39. DO teach students to consider their OR answers to be “prewriting;” do not need to develop the answer past this stage • DO teach students to use information from the passage; prior knowledge is rarely requested • DO encourage reading the questions first with students for whom that might work • Search for answers as they read the text The Scoring-Teaching Connection

40. Tips to Share