Assessing an Integrated Science & Math Curriculum

1. Assessing an Integrated Science & Math Curriculum Bruce Callen Don Deeds Vickie Luttrell Mark Wood Charles Allen The Division of Natural Sciences Drury University Springfield, Missouri

2. Outline • Our context & curriculum • Our initial assessment protocol • Recent efforts & the SaM-VI • General thoughts on effective assessment

3. Drury: Our context • Small college in SW Missouri founded by Congregational church • Liberal arts heritage, now with professional & master’s programs • General education requirements include 12 hours of integrated math & science • 40% growth in campus-wide and science enrollment over the last decade

4. Our Curriculum: Global Perspectives 21 • Goal: prepare students for responsible & productive lives in 21st century • 57 hours in Global Studies, arts, history, foreign language, ... • Includes 12-hour math/science sequence • Math & Inquiry (3 hrs) • Science & Inquiry (6 hrs) • Undergraduate Research Experience (3 hrs)

5. Our Goal: Science Literacy • Appreciate relevance of science & math • Understand and use science & math • Skills to solve real-world problems • Evaluate validity of information • Access sources of knowledge • Excitement for life-long learning

6. Part I: Our Curriculum Math & Inquiry Science & Inquiry Undergraduate Research Experience

7. Key Features • Connected series of courses • Emphasis on group work • Using process of science to address real-world problems • Learn science by doing science • Interdisciplinary approach integrates faculty as well as material

8. Math & Inquiry: 3 hours • Combination of algebra, trig, stat, calc • Group projects on real world problems • Essays on nature and relevance of math • Oral presentation of problems • Exams consist of real world problems • Use Derive software in graphing & algebra

9. Science & Inquiry: 6 hours • Team taught: biology, physics, chemistry • Three modules: often • Scientific method & atomic theory • Waves & Light • DNA & Cancer • Designed to meet goals, not cover specific content

10. Continually Evolving • Assessment, Evaluation, and Student Feedback has shaped this course • Presently, two-hour sessions three days a week • Section size 30 to 48 students

11. Undergraduate Research Experience: 3 hours • Learn & appreciate scientific process by doing science • Unique approach • Exactly what science teaching ought to be • Tremendously successful for students and faculty

12. Research courses • Up to 16 students • Research teams of 2 to 4 students • Projects designed by students & faculty • Student teams carry out background research, experimental design, analysis • Individual journal article & group poster presentation at end of semester

13. Research options • Human genetics • CCD Astronomy • Aquatic ecosystems of the Ozarks • Diabetes & cell membrane research • Multimedia in the classroom • Exercise physiology • Organic chemistry of household products • Human memory & learning • Planetary astronomy

14. Part II: Evaluation Assessment Revision

15. Initial Assessment Protocol • Pre- and Post-testing • Science and math attitudes • Math literacy • Science knowledge • Evaluating validity of scientific quantitative information

16. Other Assessment Forms • Student evaluation of course • Student evaluation of faculty • Focus groups • Outside evaluation program

17. Student Attitudes: over 40% of students thought • Math is boring • A strong math background not required to understand science • Math not important unless required for graduation • Learning math and science primarily involves memorization of facts

18. Student Attitudes: over 40% of students thought • Everyday life requires little understanding of math or science • Everyday life requires little understanding of math or science • An understanding of science and technology is not required to be a good citizen Deeds et al., J. College Science Teaching, 1999

19. Math Assessment (‘96-’99)

20. Math Assessment

21. Student Evaluations • Overall course/instructor rating, 1–5 scale • Math & Inquiry • Spring ‘97 (6 sections) 2.10 • Fall ‘97 (7 sections) 1.84 • Science & Inquiry • Spring ‘97 (3 faculty) 1.97 • Fall ‘97 (4 faculty) 1.75 • Undergraduate Research • Fall ‘97 (3 sections) 1.42

22. Survey: Science & Inquiry • 73% agreed: improved understanding methods of scientific discovery • 85% agreed: the labs improved their understanding of science • 77% agreed: discussions improved their understanding of science • 56% agreed: course helped them appreciate relevance of science

23. Outside Evaluation • Two visits from team of faculty & administrators from Spelman, Southwestern, Dickinson, Harvey Mudd • First visit: Spring ‘97 • Math & Inquiry sections should be consistent • Stronger connection between math, science • Stronger link with other general education courses

24. Revisions: Math & Inquiry • Improved consistency among sections • Essays included to strengthen attitudes • Students may retake tests to improve the learning process • Lecture time reduced to increase student involvement

25. Revisions: Science & Inquiry • Increased connections to Math & Inquiry • Cancer added to DNA module; connection to student interest and relevance • Additional exam for early feedback • More structured discussion material, often using Scientific American articles • Bi-weekly quizzes & other assignments to improve student use of texts

26. Successful Reform: Assessment is one piece • Clear goals and objectives (Think Big) • Nucleus of committed faculty • Other faculty on board • Resources (internal/external) • Assessment/revision

27. Part III: Recent Assessment New Tools

28. Math Assessment • With baseline established, pre/post testing suspended for a few years • To check, re-instituted 2002-03 • Initial results: outcomes appear consistent with earlier years

29. New Student Evaluations • Now evaluate every class, every semester • Beginning in Spring 2000, use IDEA from Kansas State • Comparison with both local and national database controlled for similar courses & students • Allows instructors to be evaluated on what they judge to be important • ‘Progress on Essential & Important Objectives’

30. Results: Last 3 semesters • 6 sections Science & Inquiry, 12 instructors • 13 sections Undergraduate Research • Four main aspects to evaluation: • Progress on Essential & Important Objectives • Improved Student Attitude • Excellence of Teacher • Excellence of Course • On each item, instructor is given a percentile rating as compared to other similar courses

31. Results: Last 3 semesters

32. Scientific Reasoning • A. Lawson’s (ASU) “Classroom Test of Scientific Reasoning” (project w/B. Hinrichs) • Administered pre- and post-test to • 75 Intro physics students (juniors) • 100 General chemistry students (freshmen) • 80 Science & Inquiry students • Repeat for Spring ‘04 Science & Inquiry

33. New Assessment: SaM-VI • Science & Math Values Inventory • Importance of affective goals • Do students value literacy in science & math? • Valid & reliable instrument with four domains of value: • Attainment Utility • Interest Personal Cost

34. Interest in Science • Importance a student places on science because of genuine interest • Items ask students to reflect on the intrinsic satisfaction they receive from learning about science, either in or out of the classroom

35. Utility of Understanding Science • Importance a student place on understanding science because it helps accomplish short- or long-term goals • Items ask students to reflect on what they have to gain personally by understanding scientific concepts, or reflect on personal benefits of taking science courses

36. Personal Cost • Sacrifices a student believes are required to develop an understanding of science, or to do well in science courses • Items ask students to reflect on what may be lost, given up, or compromised in order to master scientific concepts

37. Attainment in Science • Importance a student places on doing well in science • Items ask students to reflect on importance of developing a good understanding of science or achieving at high levels in science courses (not why it is important)

38. Useful in course design • Is what we’re planning useful to students? • Interesting to students? • Have a cost that is reasonable to them?

39. SaM-VI timetable: Science • Develop in two parts: science, math • Expert evaluation of domains (done) • Expert evaluation of item match (done) • Student evaluation of item clarity (now) • Large-scale tryout #1 (Spring ‘04) • Analysis & development of tryout #2 • Large-scale tryout #2 (Fall ‘04) • Test/retest, social influence (Spring ‘05) • Final science inventory: Fall 2005

40. Example format

41. Part IV: Effective Assessment for General Education Science & Math

42. Assessment Principles • Your goals must be clear • Your assessment plan must focus on determining if your goals are met • Your analysis of assessment data must help you decide how to modify your courses so your goals are met

43. Three questions • What are the goals for this course/curriculum? • How will we design the course to ensure that we meet our goals? • What will we do to determine if we’ve met our goals?

44. Pitfalls • Ignoring assessment at the outset • Developing tools that don’t assess things central to your goals • Assessing and then setting aside the data • Let what you learn change what you do

45. Original plan (and NSF proposal) • Science & Math Values Inventory • Scientific Reasoning Instrument • Scientific Knowledge Instrument

46. Content goals • Easier in disciplinary courses: • Often, general agreement on outcomes • Often, instruments developed (FCI, ACS) • In non-science courses, no ‘societies’, no agreement, few tools • Will there ever be national content agreement or instruments? Or always local?

47. Our goals: affective • Initial tool: student attitudes • Our waste of time: ‘science knowledge’ instrument • Our objectives: content is not one of them