Identification and Articulaton of Enduring Student Learning Outcomes

1. Identification and Articulaton of Enduring Student Learning Outcomes • Karl A. Smith • STEM Education Center / Technological Leadership Institute / Civil Engineering – University of Minnesota & • Engineering Education – Purdue University • ksmith@umn.edu - http://www.ce.umn.edu/~smith • King Fahd University of Petroleum and Minerals • Design and Implementation of Cooperative Learning • August 19-21, 2013

2. Session 2 Layout • Reflection on Session 1 • Enduring student learning outcomes (BIG ideas) • Taxonomies of learning outcomes • Connecting outcomes and assessment strategies 2

3. Five Minute University http://www.youtube.com/watch?v=kO8x8eoU3L4 Streveler and Smith

4. Content-Assessment-Pedagogy (CAP) Design Process Flowchart Understanding by Design (Wiggins & McTighe, 2005) Start Context Backward Design Content Assessment Pedagogy Streveler, Smith & Pilotte (2012) C & A & P Alignment? No Yes End

5. 3 Stages of Backward Design What should learners know, understand, and be able to do? Identify the Desired Results Determine Acceptable Evidence Plan Learning Experiences Are the desired results, assessments, and learning activities ALIGNED? Streveler and Smith

6. 3 Stages of Backward Design Identify the Desired Results How will we know if the learners have achieved the desired results? What will be accepted as evidence of learners’ understanding and proficiency? Determine Acceptable Evidence Plan Learning Experiences Are the desired results, assessments, and learning activities ALIGNED? Streveler and Smith

7. 3 Stages of Backward Design Identify the Desired Results Determine Acceptable Evidence What activities will equip learners with the needed knowledge and skills? What materials and resources will be useful? Plan Learning Experiences Are the desired results, assessments, and learning activities ALIGNED? Streveler and Smith

8. Elements of Content • Knowledge-centered aspects of Content • Focuses on the information, topics, “stuff” of the learning unit • Curricular priorities • Learning-centered aspects of Content • Focuses on how the learner interacts with the content Streveler and Smith

9. Designing Learning Environments Based on HPL (How People Learn) Streveler and Smith

10. What do you want learners to learn? Knowledge-Centered Streveler and Smith

11. Understanding Big Ideas To understand a topic or subject is to use knowledge in sophisticated, flexible ways. Knowledge and skill are necessary elements of understanding, but they are not synonymous with understanding. Matters of understanding require more: [Learners] need to make conscious sense and apt use of the knowledge they are learning and the principles underlying it. -Understanding by Design Wiggins and McTighe (1998) Streveler and Smith

12. Wiggins & McTighe Curricular Priorities Good to be familiar with Important to know Enduring outcomes Streveler and Smith

13. Streveler and Smith

14. Understanding Misunderstanding • A Private Universe – 21 minute video available from www.learner.org • Also see Minds of our own (Annenberg/CPB Math and Science Collection – www.learner.org) • Can we believe our eyes? • Lessons from thin air • Under construction

15. Exercise Determine for your curriculum (re)design Good to be familiar with Important to know or understand Enduring outcomes Streveler and Smith

16. How do people learn? Learner-centered Streveler and Smith

17. Information Processing Model of Learning • Input via attention – to short term memory – to long term memory – retention and retrieval Streveler and Smith

18. Information Processing Model of Learning • Key areas for instruction • Attention and processing power = cognitive load (bandwidth) • LIMITED – need to be careful how one uses the learner’s bandwidth • Connected nature of memory • Link to what is already known • PRIOR KNOWLEDGE • Need for organization • Structural knowledge – ADVANCED ORGANIZERS • Multiple inputs (both in timing and in modes of input) = number and strength of connections = richer learner that will be retained longer • PRACTICE • MULTIPLE MODES OF INPUT (visual, audio, explanation to yourself and others) Streveler and Smith

19. What are your initial ideas about how you can help your learners achieve the enduring outcomes of the course you are redesigning? Exercise Streveler and Smith

20. How will you know learners have learned what you want them to learn? Assessment Streveler and Smith

21. Objectives of AssessmentWhat is the Purpose of the Assessment? • We cannot measure learning directly. Instead we must make inferences from evidence. • Sound inferences depend on assessments that are aligned with content.

22. Key questions to ask yourself about assessment • What should I be assessing? • What is the best assessment to use? • How can I be sure I am consistently interpreting the results of the assessments? Streveler and Smith

23. 1. What should I be assessing? • Assessments should be aligned with your curricular priorities. • Enduring outcomes should ALWAYS be assessed. • Important to know outcomes are usually assessed. • Good to be familiar with information might not be assessed. Streveler and Smith

24. 2. What is the best assessment to use?Assessments Aligned to Curricular Priorities • Traditional Quizzes and Tests • Selected Response • Closed-ended, convergent • Well-structured • Scored via “answer key” Familiar with • Academic Prompts • Constructed Response • Open-ended, ill-structured, divergent • Academic conditions (e.g. exams, drills) • Requires analysis, evaluation, and/or synthesis • Judgment-based scoring (e.g., rubrics) Important to know Enduring Understanding • Performance Tasks • Papers, projects, design tasks, etc. • Open-ended, complex, ill-structured, divergent • Approximation of practice, specific audience • Higher autonomy, more personalized • Judgment-based scoring (e.g., rubrics) Adapted from: Wiggins, G. & McTighe, J. (1997). Understanding by Design. Alexandria, VA: ASCD

25. 3. How can I be sure I am consistently interpreting the results of the assessments? • For close-ended and well-structured assessment prompts. • Construct an accurate grading key • Determine what level of performance is acceptable for demonstrating “mastery” [i.e. “pass”] • For more open-ended prompts and for performance tasks • Construct a matrix what will link key features with how those features will be demonstrated to show mastery. This is called a RUBRIC. • Determine what level of performance is acceptable for demonstrating “mastery.” Streveler and Smith

26. Some steps for creating a rubric • Look at work you feel is of varying levels of quality. • What characteristics make some work “better” than others? • What are markers within the work point to better performance? • Exercise: Work together to create a rubric for assessing • A memo • A weld Streveler and Smith

27. Learning objectives are the bridge between what you want learners to learn and how you know they learned it. Learning Objectives and Taxonomies Streveler and Smith

28. Constructing Learning Objectives Learning Activities vs. Learning Objectives Streveler and Smith

29. Constructing Learning Objectives Using Verb-Noun Format Streveler and Smith

30. Mapping Learning Objectives to Curricular Priorities Streveler and Smith Annotated Example: Ruth Wertz - High Level Weekly Planning (Annotated Example).docx

31. Taxonomies • Bloom’s taxonomy of educational objectives: Cognitive Domain (Bloom & Krathwohl, 1956) • A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001). • Taxonomy of significant learning (Fink, 2003) • Evaluating the quality of learning: The SOLO taxonomy (Biggs & Collis, 1982) • Facets of understanding (Wiggins & McTighe, 1998) Streveler and Smith

32. Revised Bloom’s Learning Taxonomy Streveler and Smith Source: http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdf

33. Revised Bloom’s Learning Taxonomy Streveler and Smith Source: http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdf

34. Revised Bloom’s Learning Taxonomy Streveler and Smith Source: http://www.celt.iastate.edu/pdfs-docs/teaching/RevisedBloomsHandout.pdf

35. Mapping Learning ObjectivesRange of Low-Order & High-Order Objectives Annotated Example: Ruth Wertz - Map of Weekly Learning Objectives (Annotated Example).docx Streveler and Smith

36. Exercise • For the course you are (re)designing: • Starting with your enduring outcomes – write learning objectives for each of your 5 most important curricular priorities. • Place those learning objectives in Bloom’s revised taxonomy. Streveler and Smith

37. Session 3-4 Preview • Pedagogies of Engagement – Cooperative Learning and Challenge Based Learning • Informal – Bookends on a Class Session • Formal Cooperative Learning • Key Resource • Review Smith, Sheppard, Johnson & Johnson, “Pedagogies of engagement.” 37

38. Kinds of Assessment Streveler and Smith

39. Objectives of AssessmentWhat is the Purpose of the Assessment? • We cannot measure learning directly. Instead we must make inferences from evidence. Sound inferences depend on assessments that are aligned with content.

40. Assessment ExamplesFormative and Summative Assessments Examples above are not exhaustive.

41. Glossary of important terms in summative assessment • Criterion model of assessment • Determine a level of learning considered acceptable and then measure every learner against that criterion. • All could pass, none could pass. • Normative model of assessment • Learners are measured against each other. • “Grading on a curve.” • Rubrics Streveler and Smith

42. Examples of formative feedback Streveler and Smith

43. Quick Thinks • Reorder the steps • Paraphrase the idea • Correct the error • Support a statement • Select the response Johnston, S. & Cooper,J. 1997. Quick thinks: Active- thinking in lecture classes and televised instruction. Cooperative learning and college teaching, 8(1), 2-7 Streveler and Smith

44. Minute Paper • What was the most useful or meaningful thing you learned during this session? • What question(s) remain uppermost in your mind as we end this session? • What was the “muddiest” point in this session? • Give an example or application • Explain in your own words . . . Angelo, T.A. & Cross, K.P. 1993. Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey Bass. Streveler and Smith

45. Session Summary (Minute Paper) Reflect on the session • Most interesting, valuable, useful thing you learned. • Things that helped you learn. • Question, comments, suggestions. • Pace: Too slow 1 . . . . 5 Too fast • Relevance: Little 1 . . . 5 Lots • Instructional Format: Ugh 1 . . . 5 Ah Streveler and Smith

46. MOT 8221 – Spring 2009 – Session 1 Q4 – Pace: Too slow 1 . . . . 5 Too fast (3.3) Q5 – Relevance: Little 1 . . . 5 Lots (4.2) Q6 – Format: Ugh 1 . . . 5 Ah (4.4) Streveler and Smith

47. Exercise • For your 5 most important learning objectives: • What kind of assessments might you use to measure learners’ attained the level of mastery of the learning objectives? • Where can you insert formative assessment to check learners’ progress? Streveler and Smith

48. Taxonomies of Types of Learning Bloom’s taxonomy of educational objectives: Cognitive Domain (Bloom & Krathwohl, 1956) A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001). Facets of understanding (Wiggins & McTighe, 1998) Taxonomy of significant learning (Fink, 2003) Evaluating the quality of learning: The SOLO taxonomy (Biggs & Collis, 1982) 49

49. The Six Major Levels of Bloom's Taxonomy of the Cognitive Domain(with representative behaviors and sample objectives) • Knowledge. Remembering information Define, identify, label, state, list, match • Identify the standard peripheral components of a computer • Write the equation for the Ideal Gas Law • Comprehension. Explaining the meaning of information Describe, generalize, paraphrase, summarize, estimate • In one sentence explain the main idea of a written passage • Describe in prose what is shown in graph form • Application. Using abstractions in concrete situations Determine, chart, implement, prepare, solve, use, develop • Using principles of operant conditioning, train a rate to press a bar • Derive a kinetic model from experimental data • Analysis. Breaking down a whole into component parts Points out, differentiate, distinguish, discriminate, compare • Identify supporting evidence to support the interpretation of a literary passage • Analyze an oscillator circuit and determine the frequency of oscillation • Synthesis. Putting parts together to form a new and integrated whole Create, design, plan, organize, generate, write • Write a logically organized essay in favor of euthanasia • Develop an individualized nutrition program for a diabetic patient • Evaluation. Making judgments about the merits of ideas, materials, or phenomena Appraise, critique, judge, weigh, evaluate, select • Assess the appropriateness of an author's conclusions based on the evidence given • Select the best proposal for a proposed water treatment plant 50

50. The Cognitive Process Dimension The Knowledge Dimension 51 (Anderson & Krathwohl, 2001).