Brittany Smith, Ginger Holmes Rowell, Jennifer Yantz , Chris Stephens,

1. Using Bloom’s Taxonomy to Examine the Depth of a Three-week, Introductory Research Experience for College STEM Undergraduates Brittany Smith, Ginger Holmes Rowell, Jennifer Yantz, Chris Stephens, Tom Cheatham, Don Nelson, Elaine Tenpenny (Middle Tennessee State University) Tennessee STEM Education Conference February 2013

2. Introduction • Need for increase of STEM graduates

3. Introduction Mathematics as a FirstSTEP to Success in STEM • $2 M, 5-year NSF STEM Talent Expansion Program (STEP) grant (DUE: 096571) • Participants are first-time, full-time freshman with a strong desire to study STEM and an ACT-Mathematics scores between 19 and 23, inclusively • Yearly cohorts of up to 50 students • Major program components include: • Summer Mathematics Bridge • Academic Year Seminar (Freshman & Sophomore Year) • Summer Immersion – Introductory Research Experience This material is based upon work supported by the National Science Foundation under Grant No. 0969571.Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

4. Introduction Cohort 1 Cohort 3 Cohort 2

5. Introduction • MTSU FirstSTEP Summer Immersion • Three-week, inquiry-based science exploration • Small student teams (5-6) • Led by faculty and upper division or graduate student mentors • Based on research by Schwartz, et al. (2008)

6. Introduction Schwartz, et al. 8 (2008) found that students who covered one science topic in-depth in high school experienced greater success in college science courses.

7. Introduction • Traditional Successes • One team of FirstSTEP Scholars submitted results of their Summer Immersion project for publication, and it has been accepted. • Two teams will present posters at NCUR 2013.

8. Introduction Research Questions: • Does a three-week, introductory, science research program at the end of their freshman year provide an in-depth experience for at-risk STEM majors? • If so, does this in-depth experience help at-risk STEM majors experience greater success in future science courses?

9. Methods “Depth of Science Experience” Study Participants • Faculty • MTSU FirstSTEP Scholars (Cohort 2) • Completed one year of college

10. Methods 63.6% Minorities 63.6% Female 31.8% African American or Black Female 50% First Generation Averages: ACT-Math = 20.9 ACT-Science = 20.9 ACT-Composite = 21.0

11. Methods FirstSTEP Participation: • Six Teams 3-4 undergraduates - at end of freshman year 1-2 undergraduate assistants - at end of sophomore year 1 upper-division or graduate student mentor 1 faculty mentor • Approximately 40 hours/week for 3 weeks • Authentic research • Students give mid-project presentation • Students give final presentation and poster session • After completion, some students continue to work on manuscripts or conference presentations

12. Methods

13. Methods • Measure Cognitive Depth of Experience • Search for established instrument not successful • Developed new instrument using Bloom’s taxonomy • Categorizes cognitive domain which includes the development of knowledge • One level must usually be mastered before progressing to the next

14. Methods The Instrument: Depth of Science Experience (DOSE) • Used verbs from revised Bloom’s taxonomy matrix4 to develop items • Six categories with five items each = total 30 items • Possible responses were never, sometimes, and often with assigned scores of 0, 1, and 2, respectively • Ten possible points for each Bloom’s category • Statements in random order

15. Methods • Delivered paper and pencil assessment to students, faculty, and student team leaders one day prior to the end of Summer Immersion • Students and faculty were asked to report how frequently they participated in activities matching the six categories in Bloom’s taxonomy • Students and faculty completed the assessment without discussing the items with one another • Assessments were collected and data was keyed into an Excel spreadsheet • Students’ performance in science coursework before Summer 2012 was collected • Fall 2012 student schedules were collected (performance in this and future semesters will be monitored)

16. Methods • The points on each individual assessment were summed by Bloom’s category • Each category had 5 items = 10 possible points • Separated data by teams • Found average student rating for each Bloom’s category • Found average faculty and student leader rating for each Bloom’s category

17. Results Validity of the Instrument 0.70 is an acceptable value for Cronbach’s alpha

18. Results

19. Results

20. Results

21. Discussion • Depth of Introductory, Science Research Experience • Teams reported activities in all Bloom’s taxonomy categories • As expected, in most cases the frequency of activities decreases as the level of cognitive demand increases. • Mentors tend to report fewer Evaluating and Creating activities than do students

22. Discussion Limitations • Validity of instrument • Small sample size Problems that remain unsolved • Who’s perception of “depth” do we correlate with future grades? Student or Faculty?

23. Discussion • Next Steps: • Correlate depth of science experience with students’ success in future science courses • Compare “depth” of introductory, science research experience with typical science classroom or lab experience to determine “depth” benchmark • Test DOSE instrument with larger sample • Analyze qualitative data (interviews, observations, lab notebooks)

24. Discussion Questions?

25. Acknowledgements FirstSTEP Project Team • Tom Cheatham (PI, dean, advisory boards, weekly input) • Ginger Rowell (Co-PI, day-to-day organization, seminars) • Don Nelson (Co-PI, department chair, curriculum) • Chris Stephens (Co-PI, immersion, apps of math in bridge) • Elaine Tenpenny (Co-PI, bridge) • Brad Rudnik (Co-Coordinator, budget, payments, web) • Teresa Wall (Co-Coordinator, daily activities, students) • Jennifer Yantz (PhD student, assisting with evaluation) • Brittany Smith (PhD student, assisting with evaluation) • Kathryn Weisbaum (External Evaluator)

26. References 1 Achieve (May 2008) The Building Blocks of Success: Higher-Level Math for All Students. Achieve Policy Brief. www.achieve.org/files/BulidingBlocksofSuccess.pdf. 2 Adelman, C. (June 1999). Answers in the Toll Box: Academic Intensity, Attendance Patters, and Bachelor's Degree Attainment. Office of Educational Research and Improvements, US Department of Education. 3 Adelman, C. (2006). The Tool Box Revisited. Office of Educational Research and Improvement, US Department of Education. 4 Bloom’s taxonomy verb matrix. [PDF document] (n.d.) Retrieved from Western Carolina University website: http://www.wcu.edu/7536.asp 5 Conley, D. (2003). Understanding University Success. A Report from Standards for Success. A project of the Association of American Universities and the Pew Charitable Trusts. www.s4s.org/UUS_Complete.pdf. 6 Kardash, C. M. (2000). Evaluation of an undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors, Journal of Educational Psychology, 92, 191-201.

27. References 7 Sadler, P.M. and Tai, R.H. (2007). The Two High-School Pillars Supporting College Science. Science, 317 (5837), p. 457-458. 8 Schwartz, M. S., Sadler, P. M., Sonnert, G., & Tai, R. H. (2009). Depth versus breadth: How content coverage in high school science courses relates to later success in college science coursework. Science Education, 93, 798-826. 9 Stemler, S. & Tsai, J. (2008). Best practices in interrelater reliability: Three common approaches. In J. Osborne (Ed.), Best practices in quantitative methods (pp 29-49). Thousands Oaks, CA: Sage Publications. 10 Tai, Robert H., Sadler, P.M., and Loehr, John F. (2005). Factors Influencing Success in Introductory College Chemistry. Journal of Research in Science Teaching, Vol. 42, No. 9, p. 987-1012. 11 Wilson, B., Shrock, S. (March 2001). Contributing to success in an introductory computer science course: a study of twelve factors. ACM SIGCSE Bulletin, Volume 33, Issue 1, p 184 – 188.