Gregory Light, PhD

1. Brazil Science, Technology, Humanities, Engineering and Mathematics (STHEM) Consortium 1ST WORKSHOP: LORENA, BRAZIL, MAY 26‐30, 2014 Changing Teaching, Transforming Learning Session II: Assessing Learning Gregory Light, PhD

2. Overview Session I: Design Principles & Learning Outcomes • Global & National Challenges (Interactive Presentation) • Teaching for Learning: a Framework (Interactive Presentation) • Constructing Learning Outcomes (Activity) Session II: Assessing Learning • Assessing Student Learning (Discussion) • Dimensions of Assessment (Interactive Presentation/Activity) • Aligning Assessment (Activity)

3. “The quickest way to change student learning is to change the assessment system.” -- Elton & Laurillard, 1979, p. 10

4. ASSESSMENT Group Question: Why do we assess students? At Tables Share

5. Purposes of AssessmentSummative • To pass or fail a student • To grade or rank a student • To license to proceed • To select for future courses • To license to practice • To predict success in future courses • To predict success in employment • To select for future employment

6. Purposes of AssessmentFormative • To provide feedback to students to improve their learning • To motivate students • To diagnose a student’s strengths and weaknesses • To help students develop their skills of self-assessment • To provide a profile of what a student has learnt

7. Purposes of AssessmentEvaluative • To provide feedback to teachers • To improve teaching • To evaluate a course’s strengths and weaknesses • To make the course appear ‘respectable’ and credit worthy to other institutions and employers

8. Dimensions of Assessment

9. Dimensions of Assessment Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group

10. Dimensions of Assessment Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group

11. Dimensions of Assessment Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group

12. Dimensions of Assessment Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group

13. Dimensions of Assessment Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group

14. Dimensions of Assessment LEARNING-CENTERED Self-Referenced Criterion Referenced Based on knowledge and skills learned on course Based on self-reflection of learning achieved on the course Validity The extent to which the assessment methods reflect student learning and the learning goals of the course Summative Formative Practicality Essentially designed to sum up someone’s achievement Essentially designed for use in helping the learning process Reliability The extent to which the results of the assessment method can be trusted Norm-Referenced Peer-Referenced Based on peer appraisal of learning achieved on the course Based on comparisons with others in the group TEACHING-CENTERED

15. Activity: Assessment MethodsChoose an assessment method(s) you currently use and “map” it on to 1) a learning outcome2) these dimensions.Do individually, then share at tableThen with whole group

16. Assessment Methods(Some Examples) • final exam (written) • Homework • projects • reports • Presentations • observed discussion groups • office hours • final exam (multiple choice) • quizzes • cold calling • career performance • standardized tests • personal response systems Higher validity – less reliable Higher reliability – less validity

17. Aligning AssessmentUsing Rubrics(Making valid assessments more reliable) Quick Share

18. Aligning Assessment with a Learning Outcome (Biology Example) Knight, (2011)

19. A Typical Rubric Format Adapted from Beauchamp et al 1996

20. Assessing Writing: Example Rubric Adapted from Beauchamp et al 1996

21. Benefits of Rubrics • Used for grading (Summative) or feedback (Formative) • Clear criteria (Criteria Ref.) • Ensure grading aligns with learning outcomes (Validity) • Shared with multiple teacher graders/situations (Reliability) • Can be used with students (Self Ref.) and peers (Peer Ref.)

22. Activity In small group, identify and design a rubric for a specific Learning outcome Large Group Discussion

23. Aligning AssessmentUsing Multiple Choice Questions (MSQ)(Making reliable assessments more valid) Quick Share

24. MCQs as formative instruction Answer and provide your rationale: 1. Which of the following has/have intrinsic pacemaker characteristics? a) Medulla c) Sinoatrial node b) Pons d) Atrioventricular node Use rationales (short open-ended explanation for choice) to let students demonstrate their learning - can be graded or not.. Stanford Learning Lab: Nash & Shaeffer, 1999; Schaeffer et al., 1999

25. Ideal rationale: SA node is the normal pacemaker for the entire heart. AV node also has pacemaker potential, but is overshadowed by SA node. Medulla has pacemaker potential for breathing rhythm as well. Pons helps refine rhythm, but does not have pacemaker potential. Less-than-ideal rationales: Offering an incomplete answer: Normally the SA node is responsible for generating heart rate, and it is able to do this because of its intrinsic rhythm. The AV node also has an intrinsic rhythm, but it is “overshadowed” by that of the SA node. Providing a book definition: The sinoatrial node is the pacemaker of the mammalian heart. Providing irrelevant information: Stretch receptors are located in the aortic arch and the carotid sinus. They have the ability to respond to changes in pressure. Restating the question: The SA node, AV node, and medulla all possess intrinsic pacemaker characteristics as they all serve as intrinsic pacemakers. Blind appeal to authority: This answer is right because Professor Heller said that it was, and Professor Heller is cool.

26. Activity In small group, share other ways in which the validity of MCQs might be enhanced. Share with Large Group

27. Testing low-level knowledge Purely economic loss is recoverable in a product liability action. • True • False Purely economic recovery will be barred in which of the following causes of action? • Negligence • Fraud • Defamation • Product liability “What’s the rule?” No context, not allowing for interpretation/analysis from Case & Donohue, 2008

28. Revision: Higher-level understanding A restaurant hired an exterminator to eliminate cockroaches from the basement under the restaurant. Around midnight, the exterminator applied to the basement floor and walls an effective pesticide that he had purchased from the manufacturer. A toxic gas released by the pesticide penetrated into the restaurant kitchen and did not disperse by the next day. As a result, the restaurant was required to close that day. The restaurant brought a tort action based on product liability against the pesticide manufacturer for lost profits. Will the restaurant prevail? • No, because in this action purely economic loss in not recoverable. * • No, because the exterminator was the proximate cause of the restaurant's damages. • Yes, because the manufacture of pesticides is an abnormally dangerous activity. • Yes, because the pesticide was being used as intended. from Case & Donohue, 2008

29. 3 ideas to take with you

30. FinalQUESTIONS?

31. References • Entwistle, N. & Tait, H. (1990) ‘Approaches to learning, evaluations of teaching and preferences for contrasting academic environments’, Higher Education, 19 (2): 169–94. • Knight, J. (2011) University of Colorado. NAS/HHMI Summer Institute on Undergraduate Biology Education, Madison, WI. • Light, G, & Micari, M. (In press) Making Scientists: Six Principles for Effective College Teaching, Harvard University Press. • Beauchamp, McConaghy, Parsons & Sanford. (1996) Teaching From the Outside In. Duval: 1996, 37.