Thinking and Acting Like a Scientist: Investigating the Outcomes of Introductory Science and Math Courses

1. Thinking and Acting Like a Scientist:Investigating the Outcomes of Introductory Science and Math Courses Kevin Eagan Jessica Sharkness Sylvia Hurtado Higher Education Research Institute, UCLA Association for Institutional Research 49th Annual Forum May 30 - June 3, 2009 Atlanta, Georgia

2. Background • Relatively few students earn degrees in natural science or engineering in the U.S. • 15% of U.S. BA degrees are in science/engineering • Compared to 67% in Singapore, 50% in China, 47% in France, 38% in South Korea • U.S. needs more undergraduate science majors to maintain achievement and innovation in science and engineering • Also need to diversify the scientific workforce and increase representation of women and minorities

3. Background • To graduate more bachelor’s degrees in science, U.S. needs students to choose science majors and to maintain interest in science majors • National increases in proportion of freshmen indicating interest in science, technology, engineering and math (STEM) majors • However, low proportion of students who intend to major in STEM actually graduate with STEM majors • One obstacle to STEM major completion: Introductory “gatekeeper” courses • Mechanism for sorting students • What is rewarded?

4. Introductory “Gatekeeper” Courses • First course in a series of courses in which knowledge is cumulative • In science – relatively high drop-out and failure rates in gatekeeper courses • Large lectures • Un-engaging • Highly competitive • Grading on a curve

5. Classroom Environments & Instructor Pedagogies • Classroom climates have an impact on learning and performance • Competitive environments have negative impact on learning, performance, retention, self-confidence • Collaborative environments that emphasize group work can mitigate negative effects of large lectures and competitive environments • Can also promote critical thinking about scientific concepts and their applications

6. Supportive Learning Environments and the Skills Needed for Scientific Success • Six necessary conditions for a supportive learning environment: • Quality of instruction, Teacher’s interest, Social relatedness, Support of competence, Support of autonomy • Engender greater self-motivation, encourages self-directed learning • Two primary pedagogical techniques in science • Domain-specific learning = memorization of facts and causal relationships • Domain-general learning = reasoning strategies and critical thinking skills

7. Additional influences on student success in STEM courses • Experiences external to the classroom environment • Participation in research projects • Peer Tutoring • Prior academic achievement and preparation • Most significant influence on outcome of introductory courses?

8. Goals of Current Study • Untangle effect of prior preparation and background factors from performance assessment in introductory STEM courses • Identify how students develop in introductory courses the critical thinking dispositions necessary for science careers, and whether these dispositions are reflected in student grades

9. Conceptual model High School Science Achievement Tutoring & Research Participation Course Grade Amount of student effort expended on course Demographic variables Course Learning Environment & Pedagogy Ability to act and think like a scientist (Post-test) Ability to act and think like a scientist (Pre-Test) Critical thinking dispositions

10. Data & Sample • Data collected via online survey from students in 12 introductory science and math courses at 5 institutions • Two surveys – one at beginning of course (pre-survey) and one at end (post-survey); final analytic sample = 255 • Final longitudinal sample: • 70% female • 34% White, 43% Asian, 8% Black, 13% Latino • 86% majoring in STEM field • 64% first-year students, 27% second-year

11. Thinking and Acting like a Scientist Measurement model fit statistics: χ2=300.69 (305, N=255), NNFI = 0.98, CFI = 0.98, RMSEA = 0.03, reliability = 0.82. *All items were asked as part of a questions stem that read, “Rate your ability in the following areas as it pertains to your academic learning in the sciences.” Response options were Major Strength (5), Above Average (4), Average (3), Below Average (2), Major Weakness (1)

12. Variables • Independent Variables: • Demographic characteristics • Prior preparation • Course pedagogy • Classroom environment • College experiences • Critical thinking dispositions (CCTDI subscales) • Dependent Variables • Final course grade • Dispositions toward science

13. Analysis Plan • Identification of latent constructs (factors), representing acting like a scientist and thinking like a scientist • Exploratory Factory Analysis  Confirmatory Factor Analysis (measurement model in SEM) • Structural Equation Modeling (SEM) to model how student experiences in introductory courses affect three outcomes

14. Structural Modeling Procedure • Added hypothesized predictors and paths to structural equation model • Used prior research, theory and Wald and LaGrange multiplier tests to add and remove paths that did not contribute to model fit • When all paths were deleted from a variable, variable was removed from the model

15. Results (non-significant paths not shown) External Support/Science Experiences Course Learning Environment Student Effort Consistently got support needed HPW Engage in lab activities Felt competition in course Course Pedagogy HPW prof’s research project (college) Crammed for exams Felt Overwhelmed Course employed group activities Sought tutoring on campus Format primarily lecture Grade in intro course Participated in HS research program Think like a scientistPost-test Demographics Pretests Income Act like a scientistPost-test Avg HS GPA in STEM courses URM Student BBS Major Female Act like a scientistPre-test AP Chemistry Score Tutored Student in HS Think like a scientistPre-test Analyticity Open-Mindedness Critical Thinking Confidence Critical thinking dispositions

16. Discussion of findings: Thinking and Acting Like a Scientist • CCTDI – Openmindedness negatively predicted both dispositions • Relativism in conflict with objective, empirical perspective of science • CCTDI – Critical thinking self-confidence positively predicted both dispositions • Indicative of students’ development of domain-general science skills • CCTDI – Analyticity positively predicted thinking like a scientist • Indicative of students’ ability to think more carefully and critically during problem-solving activities • Students who felt overwhelmed scored lower on both dispositions • May be indicative of naïveté about what is required of science majors

17. Discussion of findings: Final Course Grade • Thinking and acting like a scientist and CCTDI subscales unrelated to final grade • Suggests that introductory courses focus too much on the acquisition of knowledge rather than development of higher-order thinking skills • Final grade in large part predicted by prior preparation (high school grades, research) • Suggests that failure to earn top grades may merely lack the preparation necessary for success in these courses rather than science-related skills

18. Discussion of findings: Final Course Grade • Cramming for exams positively predicted course grades • Underscores that grades reward rote memorization rather than development of higher-order thinking skills • Group work positively predicted final course grade • Supports prior research that concluded that peer learning provides learning reinforcement, which may have longer-term benefits • Tutoring (receiving and providing) positively predicted final course grade • Reinforces the benefits of peer-to-peer teaching and learning

19. Implications and Conclusions • Can we afford to cram content into courses at the expense of development of scientific skills and thinking? • Adjust grading practices so that they more accurately reflect learning rather than prior preparation • Further examination of pedagogical practices and interventions in large, lecture-based gatekeeper courses is needed

20. Contact Information Faculty and Co-PIs: Sylvia Hurtado Mitchell Chang Graduate Research Assistants: Kevin Eagan Lorelle Espinsoa Christopher Newman Administrative Staff: Aaron Pearl Acknowledgments: This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R01 GMO71968-01 and R01 GMO71968-05 as well as the National Science Foundation, NSF Grant Number 0757076. This independent research and the views expressed here do not indicate endorsement by the sponsors. Jessica Sharkness Minh Tran Paolo Velasco Papers and reports are available for download from project website: http://heri.ucla.edu/nih Project e-mail: herinih@ucla.edu