Bio-Link Industry Research Methods Workshop: Thinking about Learning Outcomes

1. Bio-Link Industry Research Methods Workshop: Thinking about Learning Outcomes Louise Yarnall SRI International June 5, 2008

2. Why Does Defining Learning Outcomes Matter for Biotechnology? • Clarifies communication around “what knowledge and skills matter” in emerging biotech fields • Forms a foundation for documenting college achievement to industry audience (e.g., certification) • Helps define, revise, and expand program goals

3. Brief Overview of Talk • Background of our project • Description of our process for working with biology experts • Presentation of preliminary “big ideas” in the field of biology • Possible implications of our work for community college biotechnology fields

4. Background • SRI International received $1.4 million grant from the U.S. Dept. of Education • Goal: To develop a prototype post-secondary assessment measuring a new type of sophomore-year learning outcome

5. Background • What we assess drives what is taught • To improve instruction, assessments should test high-value knowledge and skills • Tests often measure memorized knowledge and procedures, not reasoning and problem solving

6. Background We want to measure: • Deep, foundational, and flexible forms of knowledge that form the “big ideas” that experts use in a domain • Such knowledge and skill was said to be highly desirable, but highly unusual, in beginning lab techs • Ultimate goal: Measure how well students in first two years of college begin to develop such knowledge

7. Background • Developing prototype assessments in: • Biology • Economics

8. Description of Process • Biology Expert Panel, April 5-6 • Members: • Eric Jakobsson, University of Illinois, computational biology education • John Jungck, Beloit College, computational biology education • Paul Kassner, Amgen, pharmaceutical development • Patricia Morse, University of Washington, field biology • Rick Vosburgh, Nekton, biomechanics

9. Description of Process Presented “goals” of meeting: • Conduct domain analysis: identifying a collection of fundamental knowledge and skills critical in the biology domain • Prioritize the “big ideas” that experts use in a domain

10. Description of Process Representation of knowledge we seek to measure

11. Description of Process Experts generated lists of: • Schematic knowledge • Based on ~ 50 starter ideas* • Strategic knowledge • Based on ~ 15 starter ideas* Experts also listed: • tools, representations, & resources used in work * Sources of “starter ideas”: • Improving Undergraduate Education, National Academy of Sciences • BIO2010, National Research Council • Research articles and Core Curriculum Lists

12. Description of Process • Experts organized knowledge, skills, tools & resources under key “big ideas” • Experts generated possible work-related activities that elicit the use of such “big ideas”

13. Description of Process: Summary and Preview • Elicit and prioritize important knowledge and skills in a domain • List possible tasks through which students may demonstrate important knowledge and skills • Generate new and important “big ideas” • The “big ideas” will be developed into new student learning outcomes

14. Preliminary Big Ideas in Biology • Two levels: • Fundamental concepts: • Evolution • Bioenergetics • Systems Biology/Form & Function • Biological reasoning process: • Hypothesis generation and testing

15. Preliminary Big Ideas in Biology • Different levels of application of big ideas: • Ecosystem • Population • Species • Cells

16. Preliminary Big Ideas in Biology Schematic Knowledge

17. Preliminary Big Ideas in Biology Schematic Knowledge

18. Preliminary Big Ideas in Biology Strategic Knowledge • Cognitive reasoning processes • model-based reasoning • design within constraints • cause and effect reasoning • reasoning from evidence • Phases of problem solving • framing a problem • planning the solution • executing the solution • evaluating the solution process

19. Preliminary Big Ideas in BiologyStrategic Knowledge:Design within Constraints

20. Preliminary Big Ideas Summary • Overview of key schematic knowledge that helps in framing problems like a biologist • Overview of range of strategic knowledge modes and associated skills in using with tools, representations, and resources like a biologist

21. Preliminary Big Ideas in Biology Next steps: • What level of performance in schematic and strategic knowledge in biology is reasonable to expect for college sophomores? • Those who will work as biotechnology technicians? • Those who will transfer to biology programs in 4-year colleges?

22. Preliminary Big Ideas in Biology • Those who will work as biotechnology technicians? • Familiarity with how big ideas inform the work they do • Use of tools, representations, and resources • Those who will transfer to biology programs in 4-year colleges? • Familiarity with how big ideas inform the specialized areas they study in upper division • Use of tools, representation, and resources

23. Preliminary Big Ideas for Biology A reasonable performance level for sophomores? • SCHEMATIC: Understanding the concept of evolution provides a necessary framework to understanding biology. • STRATEGIC: Always realize that whenever you observe, experiment on, model, or think about, any biological system in isolation, you have broken important connections between that system and its context.  So you don't fully understand the system until you have reconstructed the connections between the system and its context. • STRATEGIC: Biology is a series of our current understandings about the patterns in nature and how they work based on evidence from observations and experimentation.

24. Possible Implications • Different way of framing important learning outcomes based on: • Cognitive science studies how experts apply their knowledge • Reasoning that expert industry professionals value

25. Possible implications • Defining a new kind of learning outcome: • Threshold idea: Once you learn the schematic and strategic knowledge of a domain, you never see problems in the real world the same way again.

26. Possible Implications By defining new learning outcomes based on "big ideas" in a field, we can: • Engage with industry regarding the problems these workers need to solve • Define the knowledge and skills that will make students into problem solvers • Assess their performance as problem solvers • Change the perception of their work: problem solvers