Developing Course-Level Learning Outcomes: Enhancing Learning Through Shared Expectations

1. Developing Course-Level Learning Outcomes: Enhancing Learning Through Shared Expectations A Workshop Hosted by: The College of Arts and Science, the Thompson Center for Learning and Teaching, and the Assistant Provost for Assessment and Institutional Accreditation December 1, 2006 University of Michigan-Flint

2. Agenda: • Introduction (Definitions and Objectives) • Working Lunch (Critique and Share) • Writing Well Stated Outcomes • One Framework: Bloom’s Taxonomy • Independent Work 1: • Articulating your course objectives and outcomes • Course Alignment • Independent Work 2: • Specifying Teaching & Learning Activities • Stating Assessments of Student Learning • Beyond Course Alignment

3. Why transform to a language of assessment? • Specific learning outcomes lead to: • More measurable outcomes • Better assessment • Higher quality feedback • Improved courses and programs • Improved student learning and achievement

4. Design Backward Intended Learning Outcomes of the Lesson Intended Learning Outcomes of the Unit Intended Learning Outcomes of the Course Deliver Forward Alignment Within Courses

5. The Language of Assessment Outcomes Based Assessment: A process by which you • determine the indicators of an effective program, • use those indicators as criteria for assessing the program, and • apply the results of the assessment toward the ongoing and continuous improvement of the program.

6. Objectives vs. Outcomes1 • Program/course objectives are general goals that define what it means to be an effective program/course. They are general, indefinite, and not intended to be measured. They set the overall agenda for the program/course. • Student learning outcomes are specific results the program/course seeks to achieve in order to attain the general goals defined in the objectives. Outcomes are definite and intended to be measured. They establish the particular means by which the agenda (as defined by objectives) is achieved. The achievement of outcomes is evidence that our students are learning.

7. Direct vs. Indirect Assessment1 • Direct assessment of learning: gathers evidence, based on student performance, which demonstrates the learning itself. • Examples: most classroom testing for grades or evaluation of a research paper on specific criteria • Indirect assessment of learning: gathers reflection about the learning or secondary evidence of its existence. • Examples: student, alumni, employer surveys

8. Assessment is not always an “add-on”1 • Embedded assessment: a means of gathering information about student learning that is built into, and is a natural part of the teaching-learning process. • Example: as part of a course, expecting each senior to complete a research paper that is graded for content and style, but is also assessed for advanced ability to locate and evaluate Web-based information (as part of a program level, or a college-wide outcome to demonstrate information literacy).

9. Formative vs. Summative Assessment 1 • Formative assessment: the gathering of information about student learning - during the progression of a course or program and usually repeatedly - to improve the learningof current students. • Summative assessment: the gathering of information at the conclusion of a course, program, or undergraduate career to improve learning of the next cohort of students or to meet accountability demands.

10. Accountability and Improvement1 • Assessment for accountability: assessment of some unit (could be a department, program or entire institution) to satisfy stakeholders external to the unit itself. Results are often compared across units, compared to state and national norms, and always summative. • Assessment for improvement: assessment that feeds directly, and often immediately, back into revising the course, program or institution to improve student learning results. This can be formative or summative.

11. Levels of Assessment1 • Assessment of individuals: uses the individual student, and his/her learning, as the level of analysis. • Assessment of programs: uses the department or program as the level of analysis. Ideally program goals and objectives would serve as the basis for the assessment. • Assessment of institutions: uses the institution as the level of analysis. Ideally, institution-wide goals and objectives would serve as a basis for the assessment. At this level it is essential to examine institutional documents such as mission and vision statements, as well as strategic plans.

12. Lunch Activity • Share your experiences with writing learning outcomes for courses or programs with those at your table. • Describe questions/challenges you face • Review the syllabi provided • What do you like/not like with respect to the learning outcomes in each syllabus? • Use Worksheet 1 (blue) to make individual observations and the same worksheet (buff) for collective observations

13. Characteristics of Well Stated Learning Outcomes2 • student-focused rather than professor focused • focused on the learning resulting from an activity rather than on the activity itself • focused on skills and abilities central to the discipline and based on professional standards of excellence • general enough to capture important learning but clear and specific enough to be measurable • focused on aspects of learning that will develop and endure but that can be assessed in some form now

14. Common Problems with Learning Outcomes • Using vague terms, such as: • Appreciate • Become aware of • Become familiar with • Develop • Know • Learn • Understand • Describing action taken by someone other than the learner. • “The program will...” or • “The course will…”

15. Students will understand Erikson’s developmental stages. Students will be familiar with the major sociological perspectives and how they relate to their daily lives. Students will develop the skills necessary for conducting research in the natural sciences. Students will identify and summarize each of Erikson’s stages of development. Students will describe each of the major sociological perspectives and will illustrate how each perspective relates to events in their daily lives. Students will design, conduct, and analyze a research project using appropriate scientific theory and methodology A Comparison of Poorly and Well Stated Outcomes

16. Why Classify Learning Outcomes? • All learning outcomes are not developed, delivered, or measured equally

17. Learning Domains • Three primary domains for classifying educational goals: • Cognitive (knowledge) • Affective (attitudes) • Psychomotor (skills)

18. Bloom’s Taxonomy of Educational Objectives3(Cognitive domain) • A now classic system that classifies educational goals to facilitate the development and evaluation of college and university curricula. • A hierarchical taxonomy of student behaviors that reflect the development of increasingly complex cognitive abilities and skills as a result of instructional experiences.

19. Bloom’s Taxonomy of Cognitive Levels • There are 6 categories, listed hierarchically from simplest to most complex • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

20. Bloom’s Taxonomy: Knowledge4 • Definition: • ability to remember information from simple (facts, terminology) to more complex/abstract (theories, principles) • Student Learning Verbs: • List, name, identify, show, define, recognize, recall, state, describe, label, match, outline, reproduce, select

21. Bloom’s Taxonomy: Knowledge4 • Examples:Knowledge of dates, events, places, major ideas, and mastery of subject matter • The student will… • Define the 6 levels of Bloom’s taxonomy of the cognitive domain

22. Bloom’s Taxonomy: Comprehension4 • Definition: • ability to understand material at a level sufficient for grasping its meaning and inferring its implications • Translating, comprehending, or interpreting information based on prior learning • Student Learning Verbs: • Summarize, explain, interpret, describe, compare, paraphrase, differentiate, demonstrate, restate, illustrate

23. Bloom’s Taxonomy: Comprehension4 • Examples:Translates knowledge into next context, interprets facts, compare and contrast, order, group, infer, predict • The student will… • Explain the purpose of Bloom’s taxonomy of the cognitive domain

24. Bloom’s Taxonomy: Application4 • Definition: • ability to correctly and independently bring to bear abstractions (e.g., theories, principles, methods) in solving concrete problems • The selection, transfer, and use of data and principles to complete a task with a minimum of direction • Student Learning Verbs: • Solve, illustrate, calculate, compute, use, interpret, relate, manipulate, apply, classify, modify, demonstrate, construct, discover, predict

25. Bloom’s Taxonomy: Application4 • Examples:Use information, methods, concepts or theories in new situations, solve problems using required skills or knowledge • The student will… • Write an instructional objective for each level of Bloom’s taxonomy of the cognitive domain

26. Bloom’s Taxonomy: Analysis4 • Definition: • ability to parse information into is constituent elements and to identify the relationships between those elements • Student distinguishes, classifies, and relates the assumptions hypotheses, evidence, or structure of a statement or question • Student Learning Verbs: • Analyze, organize, categorize, deduce, choose, contrast, compare, distinguish, separate, differentiate, discriminate

27. Bloom’s Taxonomy: Analysis4 • Examples:Seeing patterns, organization of parts, recognition of hidden meanings, identification of components • The student will… • Compare and contrast the cognitive and affective domains as specified by Bloom

28. Bloom’s Taxonomy: Synthesis4 • Definition: • ability to combine elements into new wholes (e.g., ideas, plans of action, abstract relations) that are more than the sums of their respective parts • Student originates, integrates, and combines ideas into a product, plan, or proposal that is new to him or her. • Student Learning Verbs: • Design, create, hypothesize, invent, develop, support, schematize, write, report, discuss, plan, devise, compare, construct, compose, generate

29. Bloom’s Taxonomy: Synthesis4 • Examples:Use old ideas to create new ones, generalize from given facts, relate knowledge from several areas • The student will… • Design a classification scheme for writing educational objectives that combines the cognitive, affective, and psychomotor domains.

30. Bloom’s Taxonomy: Evaluation4 • Definition: • Ability to offer quantitative and qualitative judgments about the value of ideas and methods • Student appraises, assesses, or critiques on a basis of specific standards and criteria • Student Learning Verbs: • Evaluate, choose, estimate, judge, defend, criticize, justify, recommend, critique, interpret, support

31. Bloom’s Taxonomy: Evaluation4 • Examples:Compare and discriminate between ideas, assess value of theories or presentations, make choices based on reasoned argument, verify value of evidence, recognize subjectivity • The student will… • Judge the effectiveness of writing objectives using Bloom’s taxonomy

32. Learning Outcomes by Bloom’s Taxonomy Bloom's Taxonomy of Cognitive Categories Course Goals/Objectives Unit/Lesson Learning Outcomes Student Learning Outcomes Know-ledge Com-prehen-sion Appli-cation Eval-uation Anal-ysis Syn-thesis 1.a.1. Define the three tenets of the Central Limit Theorem 1. Introduce students to inferential statistics  X 1.a learn the conceptual foundations of inference 1.a.2. Describe three key distributions  X 1.a.2. Combine to explain the relationship between the three distributions  X 1.b Apply to confidence intervals 1.b.1. Outcome 1.c Test for difference between means 1.c.1. Outcome 2. Introduce students to descriptive statistics 2.a Outcome 2.a.1. Outcome 2.b Outcome 2.b.1. Outcome Note: While this worksheet accommodates 7 learning outcomes, your specific course will most likely have more than this single worksheet can accommodate. The purpose of the worksheet is to provide a framework and not set parameters.

33. Alignment within a Given Course Course-Level Learning Outcomes Teaching and Learning Activities Assessments of Student Learning

34. The Next Step: • Given your student learning outcomes, what specific tasks or activities will you have students complete to promote learning? • Given these student learning outcomes, how will you know when your students have achieved the outcomes for that lesson or course (what assessments will you use)?

35. Teaching & Learning Activities by Outcomes and Bloom’s Taxonomy Student Learning Outcomes Bloom's Taxonomy of Cognitive Categories Knowledge Comprehension Application Analysis Synthesis Evaluation Class lecture and students will read assigned chapter. 1.a.1 Define the three tenets of the Central Limit Theorem Class lecture and students will read assigned chapter. 1.a.2. Describe three key distributions In class, students will calculate sample means and construct a sampling distribution. Homework will reinforce lesson. 1.a.3 Combine to explain the relationship between the three distributions 1.b.1 Outcome 1.c.1 Outcome 2.a.1 Outcome 2.b.1 Outcome

36. Assessments by Outcomes and Bloom’s Taxonomy Student Learning Outcomes Bloom's Taxonomy of Cognitive Categories Knowledge Comprehension Application Analysis Synthesis Evaluation 1.a.1 Define the three tenets of the Central Limit Theorem Homework and examination. 1.a.2. Describe three key distributions Homework and examination. 1.a.3 Combine to explain the relationship between the three distributions Homework and examination. 1.b.1 Outcome 1.c.1 Outcome 2.a.1 Outcome 2.b.1 Outcome

37. Beyond Course Alignment • Thinking “Globally”: Alignment of course level learning outcomes: • Sequentially • How do my outcomes fit with those of more advanced courses? • What do I expect students to have learned by the time they when they enter this class? • Programmatically: • How do all of the courses within a program’s curriculum fit together? • How do the program curricula contribute to the institutional outcomes?

38. Design Backward Intended Learning Outcomes of the Academic Program Intended Learning Outcomes of the Lesson Intended Learning Outcomes of the Unit Intended Learning Outcomes of the Course Intended Learning Outcomes of the Institution Deliver Forward Alignment Between Course Outcomes and Institutional Outcomes

39. Want to learn more? Plan to participate in the Assessment Summit, January 12, 2007

40. References • 1 Leskes (2002) “Beyond Confusion: An Assessment Glossary,” Peer Review. • 2 Huba and Freed. 2000. Learner-Centered Assessment on College Campuses. • 3 Bloom.1956. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain • 4 From Don Clark, http://www.nwlink.com/~dpmc;arl/hrd/bloom.html, the Learning Skills Program at the University of Victoria (htttp://www.coun.uvic.ca/learn/program/hndouts/bloom.html) , the Faculty Roles and Rewards Program at Portland State University (http://edtech.clas.pdx.edu/presentations/frr99/blooms.htm), and W. Huitt, http://chiron.valdosta.edu/whuitt/col/cogsys/bloom.html.