1 / 50

Checklists & Rating Scales

Checklists & Rating Scales. Scoring Performances. Goals. understand the importance of observing performances and grade them appropriately understand, spot, compensate for usual errors understand and use appropriately the following observational tools: checklists rating scales exemplars

danil
Download Presentation

Checklists & Rating Scales

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Checklists & Rating Scales Scoring Performances

  2. Goals • understand the importance of observing performances and grade them appropriately • understand, spot, compensate for usual errors • understand and use appropriately the following observational tools: • checklists • rating scales • exemplars • keys Observation Checklists

  3. Observation Checklists

  4. Principles of Effective Rating • Characteristics should be educationally significant: in harmony with department’s vision, goals, and objectives and desired learning outcomes. • Characteristics should be directly observable: occur in the setting and clearly visible to an observer – skills – less so are interests, feelings, and attitudes. Observation Checklists

  5. Principles of Effective Rating • Characteristics and points on the scale should be clearly defined. • Between three and seven rating positions should be provided, and raters should be permitted to mark at intermediate points. • Raters should be instructed to omit ratings when they feel unqualified to judge. Observation Checklists

  6. Principles of Effective Rating • Ratings from several observers should be combined whenever possible -> greater reliability. Observation Checklists

  7. Checklists and Rating Scales • are MONITORING and • RECORD-KEEPING devices • With increased emphasis on self-reflection and self-assessment, individual students should use self-checklists and rating scales to ASSESS THEIR OWN PROGRESS and DEVELOP AN IMPROVEMENT PLAN. Observation Checklists

  8. from adult framework recorded objective & nonjudgmental consider whole learner regular, systematic, frequent observing include spontaneous & ongoing observations efficient recording purposeful monitor product & process before, during, after task from many points of view use spectator and participant modes Effective Observations Observation Checklists

  9. Affective Measurement Problems • Fakability • Self-deception • Semantic problems • Criterion inadequacy Observation Checklists

  10. Checklists & Rating Scales • Checklist: Attitude – Yes or No • Checklist: Attitude – Yes or No? • shows originality Yes No • respects own work Yes No • is easily motivated Yes No • Rating Scale: Attitude -- Degree • Rating: Attitude SA A U D SD • shows originality SA A U D SD • respects own work SA A U D SD • is easily motivated SA A U D SD Observation Checklists

  11. Constructing Checklists • Base on course objectives. • State in clear and precise terms. • Avoid generalities & focus on specifics. • Design them so they can be used as a basis for discussion by instructor and student. Thus, write them so they are easily understood by the student. • May also be used for self- and peer-evaluation. Observation Checklists

  12. Constructing Checklists • Keep them short so that they can focus on a few specifics. • May be designed for either a single evaluation or a cumulative record. • Can be used for diagnostic, formative, and/or summative evaluation as well as a basis for anecdotal comment, as well as an instructional tool for learner to follow in practicing the skill. Observation Checklists

  13. CHECKLISTS psychomotor & social skills indicate presence or absence/mastery or non-mastery of skill may indicate sequence of actions in a performance RATING SCALES for psychomotor and affective skills indicate quality, degree, or frequency of skill or trait useful in summarizing general impressions Differentiating Assessment Tools Observation Checklists

  14. Checklist: Preparation for Varnishing • __ sands and prepares surface properly • __ wipes dust from surface with appropriate cloth • __ selects appropriate brush • __ selects varnish and checks varnish flow • __ pours needed amount of varnish into clean container, etc. Observation Checklists

  15. Rating Scale: Job Attitude • is enthusiastic G Av N Imp • is willing to work & cooperate G Av N Imp • desires to improve G Av N Imp • is a good team worker G Av N Imp • is concerned with safety G Av N Imp • works without supervision G Av N Imp • aware of job requirements G Av N Imp Observation Checklists

  16. T-Form: Encouragement • Looks Like__________Sounds Like_____ Observation Checklists

  17. Why Use Observation Checklists • A quick and easy way to observe and record many of the skills and behaviors rarely assessed prior to final test or summative exam. • Show areas that need work early enough to provide help and correct before student fails area/unit. Observation Checklists

  18. Why Use Observation Checklists • Provide opportunities to change gears” re feedback. • The characteristics of intelligent behavior can be identified, taught, and checked. • Observation checklists let us check off whether student can demonstrate skill or attribute being measured. Develop base; practice; check progress. Observation Checklists

  19. Why Use Observation Checklists • They focus on observable performances or criteria that are often more especially useful for monitoring growth or need for improvement. They are more meaningful or authentic. • Useful re what learners can do and what their learning needs are. Observation Checklists

  20. Why Use Observation Checklists • Useful to discover patterns, assess progress, and make plans to help students continue their learning. Observation Checklists

  21. How to Use Observation Checklists • First, conduct a TASK ANALYSIS to determine specific areas to include in the observation checklist. • Then, ensure students are aware of the areas that will be observed. • Do this by training participants in what the skill “LOOKS LIKE” and “SOUNDS LIKE” – even “FEELS LIKE.” Observation Checklists

  22. How to Use Observation Checklists • Students may be assessed by peers or perform a self-assessment, before an instructor assessment. • Model, Teach, and Practice the skills prior to observation. • Students should work with the instructor to develop the indicators. Observation Checklists

  23. Indicators for Persistence • knows how to access information • tries several approaches • does not give up quickly • has patience • brainstorms alternative solutions • checks own work • Do so in a T-Form. Observation Checklists

  24. Making Valid Observations • Plan in advance what is to be observed and prepare an observation list, guide, or form to make the observations more objective and systematic. • Concentrate on a few significant behaviors at a time. • Use clear, unambiguous terminology and accompanying descriptions in the observation tools. Observation Checklists

  25. Making Valid Observations • Each trait [item] should be mutually exclusive for coding. • The observer must be cognizant of the time-sampling errors: use frequent, short observations distributed over a period of several weeks and at different times of the day and week. Observation Checklists

  26. Making Valid Observations • Coordinate the observations with your teaching: instructional strategies and observations must be coordinated. • Extensive observations should be selective – the most needy cases. • Carefully record and summarize the observation immediately after it has occurred. Observation Checklists

  27. Making Valid Observations • Make no interpretations concerning the behavior until later on. Doing so may interfere with objectivity. • Have categories and coding schemes that are simple to use, that call for behaviors easily observed, and that deal with behaviors that can be conveniently recorded. Inferences should occur only after data have been gathered and recorded. Observation Checklists

  28. Making Valid Observations • Wherever possible, observers and observations should be unobtrusive. When people know they are being observed, they often tend to avoid or overemphasize certain behaviors to gain attention. Observation Checklists

  29. Checklists Basics • for psychomotor processes or products and for personal/social growth • for both processes and products • for analyzing sequences and correct order of actions • for traits present in a product Observation Checklists

  30. Checklists Basics • Limitations • indicates present or absent only – no degree specified: an all-or-nothing scale. • Not useful for summarizing impressions as no degree indicated. Observation Checklists

  31. Checklists Basics • Perform a careful task analysis to determine component behaviors. • Clearly specify the actions/traits to be observed. • Include in the list common errors where they are likely to occur. • Arrange procedures in the order they are expected to occur. Observation Checklists

  32. Checklists Basics • Keep separate checklists for each person. • Transfer results to a master list of all candidates [for making comparisons]. • Give clear directions. • Train observers. Observation Checklists

  33. Common Errors in Rating • Personal Bias: general tendency to rate all at approximately the same position on the scale • generosity error - rates all at high end • severity error – rates all at low end • central tendency error – rates all as average Observation Checklists

  34. Common Errors in Rating • Halo Effect: our general impression of a person influences the rating of individual characteristics • Logical Error: two characteristics [intelligence and achievement; gifted and poor social adjustment] are rated as more or less related than they actually are. Observation Checklists

  35. Common Errors in Rating • Rater’s Attitude: doesn’t view rating as important and doesn’t put much into it. • Not enough time to do a good job: need more extensive opportunities for observation. Observation Checklists

  36. Resulting Problems • Puts in doubt a single rating of an individual – Limits the range of an individual’s ratings. • Obscures strengths and weaknesses on different traits because the individual receives similar ratings on all characteristics. • Higher relationship implied than actually exists. • Not done, invalid, and/or flawed results. Observation Checklists

  37. Overcoming Problems • Proper design, training, and use of the scales. Observation Checklists

  38. Performance Indicators • Quality • Quantity • Waste • Safety • Improvement • Focus: Finished student product • Process: Process student uses to produce product Observation Checklists

  39. Quality Indicators • conformity to overall specifications • dimensions, spacing, and/or strength • suitability for intended purpose • general appearance • error rate during performance • efficiency of steps • choice of tools, equipment, and/or materials Observation Checklists

  40. Time Indicators • time needed to complete product or perform process • number of products produced Observation Checklists

  41. Improvement Indicators • improvement in one or more qualitative characteristics • improvement in one or more quantitative characteristics • reduction in the number of steps for which assistance is required Observation Checklists

  42. Waste Indicators • number of unacceptable products produced • amount of excess material; used during performance Observation Checklists

  43. Safety Indicators • Degree to which completed product is safe for intended use • Safe handling of tools, materials, and/or equipment • Accident rate during performance Observation Checklists

  44. What is QUALITY? • If something occurred occasionally, it would occur __% of the time. • If something occurred sometimes, it would occur __% of the time • If something occurred frequently, it would occur __% of the time • If something occurred usually, it would occur __% of the time • If something occurred often, it would occur __% of the time Observation Checklists

  45. Rating Scale Basics • Use for psychomotor and affective domains • give quality, frequency, degree, or level information • for processes or products Observation Checklists

  46. Rating Scale Limitations • Subject to a number of biases: leniency, strictness, modesty, halo effect. • Subject to social expectations and, hence, to faking and self-deception. • Subject to interpretation regarding semantic problems. • Subject to criterion inadequacy. Observation Checklists

  47. Precautions/Considerations • Begin with a blueprint of the behaviors to ensure valid sampling within the area. • Clearly define traits in most behavioral way possible. • Divide the rating continuum into as many points as needed for clarity: usually between three and seven. Observation Checklists

  48. Precautions/Considerations • Clearly define the anchors on the continuum so there is no question as to what each rating means. • Train and motivate the raters to be as accurate and objective as possible. Observation Checklists

  49. Sample Rating Scale: Degree of Confidence • 1: little or no; 2 some; 3 great confidence • inquiry teaching 1 2 3 • instructional modules 1 2 3 • simulations 1 2 3 • lecture 1 2 3 • leading a discussion 1 2 3 • questioning 1 2 3 Observation Checklists

  50. Defining Steps on a Scale • Numeric Anchors: numbers, often accompanied by verbal cues • Degree of Agreement Anchors: Completely Disagree to Completely Agree; Never to Always • Adjectival Anchors: Bipolar Adjectives define endpoints with numbers in between: valuable 7 6 5 4 3 2 1 worthless passive 1 2 3 4 5 6 7 active Observation Checklists

More Related