Collecting and Analyzing data: measuring Student Success

1. Collecting and Analyzing data: measuring Student Success Rebecca Orr, Ph.D. Professor of Biology

2. Agenda • Efforts made to improve student success: • Online homework using MasteringBiology • Peer tutoring • Online quizzing • Strategies for data collection: • Preparing to begin • Modifying as collection progresses • Involving your students • Case study: Impact of pre-exam quizzing on student outcomes • Lessons learned

3. Student Success Challenges • Have you ever heard: • “I studied so hard, and it just didn’t work!” • My question: • How can student success be improved in the introductory biology (majors) course?

4. My Evolution of Mastering Use • Began using Mastering to create assignments and made them available for optional student use. • Student feedback indicated a trend: • Successful students were using Mastering • Course evaluation surveys by students said: “Make Mastering mandatory!” • Less successful students did not tend to utilize Mastering • Moved to making Mastering mandatory by assigning a portion of lecturecredit (10%) to Mastering.

5. Is There a Significant, Positive Correlation Between Completing MasteringBiology and Exam Scores? • Students required to complete MasteringBiology homework assignments (10% lecture grade). • Overall exam average was compared to percentage MasteringBiology completed by end of semester. • Spearman's Rank Order correlation was run: (rs(138) = 0.571, P = .000) • A statistically significant, positive correlation between percentage of MasteringBiology homework completed and exam average was found.

6. Results • Students completing more work in MasteringBiology were more successful on exams.

7. Problem: How to Increase Student Success in Introductory (Majors) Biology? • My Strategy, Spring 2011 semester: • Evaluate the efficacy of providing peer tutors vs. requiring MasteringBiology homework (using exam scores to measure) • Offered optional, online pre-exam quizzes to all using MasteringBiology • Results: • Peer tutoring appears to have no benefit over offering required MasteringBiology homework. • Significant, positive correlation between participating in optional quizzing and increased exam score observed.

8. Cognitive Science of Learning • Introduction of Desirable Difficulties: • “Conditions of instruction that appear to create difficulties for the learner, slowing the rate of apparent learning, often optimize long-term retention and transfer.”

9. Testing Serves as Learning Events • Testing as a learning vehicle vs. an assessment vehicle. • Evidence shows: Practice of recalling information is more powerful than re-presentation of information. • Reality is: Students re-present information to themselves (i.e., reading a chapter over and over again, underlining things, etc.), focusing on storage rather than retrieval.

10. Can An Online Homework Platform Be Used For Quizzing to Increase Student Success? • Introduced required pre-exam quizzing delivered via MasteringBiology • Students required to take pre-exam quizzes during the Fall 2011 semester (n=199) and Spring 2012 semester (n=174) • My Question: Does taking pre-exam quizzes using Mastering result in significantly higher exam scores?

11. Preparation • Before beginning, set up spreadsheet to collect data necessary to answer questions. • My predesigned columns included: • Name (deleted prior to sending to statistician) • Exam # and grade • Number of pre-exam quizzes taken • Grade on each pre-exam quiz • Set up color coding to designate “drops” and “not in my course” • Data gathered by exporting grade book from Mastering.

12. Student Participation Compliance in Required Pre-exam Quizzing Fall 2011 & Spring 2012

13. Comparison of Exam Averages100% Quiz Takers to 0% Quiz Takers Fall 2011 Spring 2012

14. Comparison of Exam Averages by Percentage of Quiz Taking: Spring 2012

15. Analysis by Groups • Scores on Exams 1 and 2 were averaged, and students were grouped by these averages. • Groups Established: • Pass: 70%+ • Fail: <70% • Low (“weak student”): <60% • Middle(“average student”): 60%-80% • High(“strong student”): >80% • Student scores were compared by group for 100% quiz takers vs. 0% quiz takers.

16. Performance of Passing vs. Failing Students: 100% Quiz Takers vs. 0% Quiz TakersExam 3Exam 4

17. Performance by Low, Middle and High Students: 100% Quiz Takers vs. 0% Quiz Takers Exam 3 Exam 4

18. Data Analysis Often Generates New Questions! • Can students’ compliance with pre-exam quiz requirement be increased? • If percent pre-exam quiz compliance increases, will significant gains in student success in the course be observed?

19. Impact of Data Collection on Course Approach • Tell the students about the data! • Ask them to predict! Who earns higher exam scores: • Quiz takers or non quiz takers? • Quiz takers that fail their quizzes or non quiz takers? • Show them the data, and ASK: • “What group do you want to be in?!” The blue group? Or the red group?!

20. Pre-exam Quiz Participation Rate Increased Significantly in Fall 2012 vs. Spring 2012

21. Percentage of ABC vs. DFW Grades Earned in Spring 2012 vs. Fall 2012

22. Study Results • Student participation in pre-exam quizzing results in significantly higher exam grades. • Benefit realized to student is based on participation (rather than performance) in quizzing. • Participation in pre-exam quizzing results in significantly higher exam scores for students of diverse abilities. • Stronger students may see slightly greater benefits from pre-exam quizzing (as evidenced by exam averages). • Significant increase in student compliance with required quizzing was accompanied by significantly higher percentage of ABC rates vs. DFW rates.

23. Student Perception of Mastering: Homework and Pre-Exam Quizzing (% Student Agreement)

24. Study Conclusions • Pre-exam quizzing using online platform seems to provide a viable testing environment for student learning. • Using online platform allows instructors more time to teach! • All students showed positive gains in exam scores as a result of participating in pre-exam quizzing.

25. Lessons learned • Begin process of gaining IRB approval right away. • Consult with your statistician about what you want to investigate before you start. • Speak with colleagues about your investigation to get a feel for questions they may raise. • If introducing a new teaching method or technology, consider giving a student survey to gather their thoughts and comments.

26. Lessons learned (cont.) • Design your data collection tools before beginning the study. • Gather and organize more than you think you need! • Stay current in updating master spreadsheet throughout the semester: • Falling behind can become overwhelming! • New questions often arise as you review the data. • Modifications in what you record may be necessary. Much easier to do this “as you go” than to go back and wade through the entire semester’s worth of data! • Tell students about your study- involve them in the process when possible.

27. Benefits of Studying Student Success Data • Allows you to identify what you are doing that is truly effective. • Sharing results with students motivates them and helps them to feel that your requirements will yield positive gains for them. • Provides intellectual challenge to you that provides additional purpose and interest!

28. Acknowledgements • Shellene Foster Professor of Mathematics, Collin College • Pushpa Ramakrishna, Ed.D. Professor of Biology, Chandler-Gilbert Community College References • 1. Bjork, E. L., & Bjork, R. A.  (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning.  In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-64).  New York: Worth Publishers. • 2. Halamish, V., & Bjork, R. A. (2011). When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 801-812.