210 likes | 353 Views
Designing Rubrics For Classroom Assessment. Professor Timothy Farnsworth, CUNY Hunter College. Communicative Assessment Is Hard!. We are making inferences about what people can and can’t get done in the real world, not: What grammar or vocabulary they can choose from 4 options on a MC test
E N D
Designing Rubrics For Classroom Assessment Professor Timothy Farnsworth, CUNY Hunter College
Communicative Assessment Is Hard! • We are making inferences about what people can and can’t get done in the real world, not: • What grammar or vocabulary they can choose from 4 options on a MC test • Tasks must be realistic but can’t be TOO realistic: • Time constraints • Stakes are different • Interlocutors are different • Setting, format usually different
Speaking Assessment Challenges • Speaking is the hardest skill to assess • Impermanent -> must be rated on the spot or recorded • Interactive -> Real communication takes two persons • Nonverbal -> Much real communication takes place without language use • Topical -> Students’ knowledge base affects their speaking output
Oral Assessment / Testing Tasks • Must get students involved in speaking! • In communication, many things happening • Listening • Nonverbal channeling • Decisions made about what to say, what not to say • Students practice avoidance strategies, game the system, don’t take risks • Assessment tasks take many forms
Types of Assessment Tasks • Many options – some common ones: • Interview, either scripted or unscripted • Picture description • Role Play • Pair or group tasks • Almost any classroom task, really!
Sample task: Pair oral assessment • Each student has a different picture; communicate to find similarities and differences
Scoring tasks and communicating results • We need some way to turn the “observation” into a “score” or “grade” • Usually done via numbers, can be done as letters, smiley faces, etc. – doesn’t matter • Should be clear, simple • Should be “reliable” -> repeatable, dependable • Should be “valid” -> meaningful • Shouldn’t cause any harm
Types of scoring rubric • Holistic • Analytic • Task Fulfillment (“primary trait”) • Performance decision trees • Checklists
Holistic Rubrics • One overall score tied to a “performance descriptor” • Most familiar type of rubric • Assign score of the description that best matches performance • Problem: Performance doesn’t match description at the same level in all areas • Advantage: Simple to use and understand
Analytic Rubrics • Two or more categories of scores • Usually based around linguistic categories such as pronunciation, grammar, etc. • Also quite common • Assign scores for each category per performance • Problem: Can be cognitively challenging for raters • Advantage: provides more detailed information
Task Fulfillment Rubrics • Usually look like holistic rubrics • Instead of describing language, they describe whether the student completed the “task” • Usually used in role play type assessments • Useful in testing for employment, other settings • Useful when there is a specific set of tasks that involve language and can be specified • Example rubric for hotel employee testing: • Job applicant is tested for ability to respond to guest requests and complaints, answer telephone, etc. in role play situation,
Performance Decision Trees • Fairly new to the scene(Fulcher 2011) • Use a flowchart design • Good for clear, simple scoring when the task is very consistent / clear • Give points for each successful “yes” on the flowchart • Lets the test raters answer very specific questions
Self / Peer Assessment • Students have some good ideas about their and their classmate’s performances • In a busy classroom, teacher cannot hear everyone • Students can use any feedback they get • Training students to listen carefully and help each other is good practice in general • But: Students are not always good evaluators of their peers • May not catch pronunciation or other important mistakes • May be LESS proficient than the person they are supposed to assess • Only ask them to assess in areas they can definitely “do” – yes / no checklists are great for this
Checklist Rubrics • Phrased very simply like a can-do statement: • “Student can understand and use vocabulary for classroom objects. Yes / No / Partially • “Student can use formal language in giving oral presentations” Yes / No / Partially • Can be broken into parts: • “Student understands WH questions” Yes / No / Partially • “Student responds to WH questions using appropriate grammar and vocabulary” Yes / No / Partially • “Student responds to WH questions at length and returns questions appropriately” Yes / No / Partially
Checklists for student self-evaluation • Self Evaluation – teacher collects evidence • Checklists are simple and clear (example) • List main objectives for the week, create checklist for students • Have them do and turn in anonymously • Use the info to plan review sesssions in future
Keys to consistent, fair scoring 1: KISS principle for tasks • Keep It Simple, Sweetheart! • Don’t use any more complicated assessment than you really need • One or two tasks is enough as long as you are getting the right kind of interaction • Make sure you get “enough speech” to make a rating using the right rubric • Make a system that everyone can easily understand • Good speaking testing usually can be done in less than five or ten minutes • More “high stakes” testing takes longer, up to an hour
Keys to consistent, fair scoring 2: Use the simplest rubric that works • I like checklists or performance trees for classroom and self assessments • I like analytic rubrics for more high-stakes testing • They give information on what things the student needs to work on • Holistic rubrics don’t provide much information for students • I don’t like task fulfillment rubrics for most situations • It’s not clear from the score what the student needs to work on • It’s sometimes possible to do a task using little language; the task fulfillment setup still allows high scores
Keys to consistent, fair scoring 3: Write the rubric so students can understand it • Use plain and simple language • Use “I can / can’t do X” or “You did X” when possible • Explain the scores and the rubric to them before the test – • no reason for it to be a secret • They should understand why they got their score and what they can work on next time • Do not use a self assessment for a grade • Do not use peer assessment for a grade
Your assignment: Write a scoring rubric • Find a partner next to you • Together, pick one rubric style: Holistic, analytic, decision tree, checklist, task fulfillment • Think about students in a typical class you teach: • What level are they? • What do you need to learn more about in terms of their speaking skills? • Write a rubric with a few levels and clear descriptions • Compare your rubric with a pair near you – does it look similar? Why or why not?
Sample task: Pair oral assessment • Each student has a different picture; communicate to find similarities and differences