1 / 52

Researching for Better O nline I nstructional M ethods using AB Testing

Researching for Better O nline I nstructional M ethods using AB Testing. --- Zhongzhou Chen, Christopher Chudzicki , Peter Bohacek , Neset Demirci and David Pritchard. Teaching is an old practice…. We enter the age of the Internet. Now we can teach like this.

jabari
Download Presentation

Researching for Better O nline I nstructional M ethods using AB Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Researching for Better Online Instructional Methods using AB Testing --- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, NesetDemirciand David Pritchard

  2. Teaching is an old practice…

  3. We enter the age of the Internet

  4. Now we can teach like this

  5. Can we find better ways to teach in an online environment?

  6. In a normal MOOC Entire MOOC Population Course Material

  7. A/B Testing Traditional Teaching Method Experimental Teaching Method

  8. A/B Testing Simple Design of an A/B Expriment Common Assessment Treatment Control

  9. Multiple A/B Experiments in a Course Unit 1 Experiment 1

  10. Multiple A/B Experiments in a Course Independent Random Splits Unit 2 Unit 1 Experiment 2 Experiment 1

  11. Multiple A/B Experiments in a Course Unit 2 Unit 1 Unit 3 Experiment 2 Experiment 1 (part 1) Experiment 1 (part 2)

  12. Summary of A/B Experiment Features • No limit on the number of groups, nor on the number of experiments in a course. • Each group can receive a different number of materials. • Group assignment is random. • No student can "opt-out" of any of the experiments.

  13. Examples of A/B Experiments in our MOOC

  14. Our MOOC: 8.MReV Content: College introductory level mechanics (calculus based) Length: 14 Weeks (12 required, 2 optional) May 29th ------- Sep 14th Initially Registered: 10709 Attempted Second Week’s Quiz: ~4100 (estimated)

  15. Our MOOC: 8.MReV Total Number of AB experiments: 7

  16. Experiment 1: Does length of video make a difference?

  17. Does length of video make a difference? Office hour videos: ~10+ minutes Talk about a challenging homework problem

  18. Does length of video make a difference? Short Video ~2 minute video segments

  19. Does length of video make a difference? Short Video ~2 minute video segments

  20. Does length of video make a difference? Short Video ~2 minute video segments Long Video ~10 minutes whole video

  21. Does length of video make a difference? Short Video ~2 minute video segments Long Video ~10 minutes whole video

  22. Experiment Procedure Challenging Homework Problem (Graded)

  23. Experiment Procedure Challenging Homework Problem (Graded) Long Video Short Video

  24. Experiment Procedure Challenging Homework Problem (Graded) Long Video Short Video

  25. Does length of video make a difference? • Research Questions: • Difference in time spent watching video? • Difference in number of questions answered? • Difference in successfully solving the challenging homework problem?

  26. Increasing Interactivity of Course Material

  27. Interactivity of Course Material Traditional problem vs. Direct measurement video

  28. Traditional Physics problem Provides all necessary information

  29. Direct measurement video (by Peter Bohacek) Distance measured by on screen ruler. Time measured by counting frames Player can play frame by frame

  30. Direct Measurement Video Common Homework Problem Survey question Direct Measurement Video Traditional Homework

  31. Direct Measurement Video • Research Questions: • Does the extra interactivity result in more participation, or does the extra work lead to less participation? • Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?

  32. Interactivity of Course Material Multiple Choice vs. Drag and Drop Problem

  33. Multiple Choice

  34. Multiple Choice

  35. Multiple Choice

  36. Traditional Multiple Choice question Extraneous Cognitive Load

  37. Drag and Drop Example

  38. Drag and Drop Example 2

  39. Drag and Drop and Deliberate Practice Deliberate Practice: (Erickson)

  40. Drag and Drop and Deliberate Practice • Deliberate Practice: • Focus on practicing one skill at a time

  41. Drag and Drop and Deliberate Practice • Deliberate Practice: • Focus on practicing one skill at a time • Providing immediate specific feedback

  42. Drag and Drop and Deliberate Practice • Deliberate Practice: • Focus on practicing one skill at a time • Providing immediate specific feedback • Repetitive and highly focused training

  43. Drag and Drop and Deliberate Practice Domain Specific Problem Design • Deliberate Practice: • Focus on practicing one skill at a time • Providing immediate specific feedback • Repetitive and highly focused training Problem Design/Auto Grading/ Customized Feedback Drag and Drop format

  44. Experimental Setup Entire MOOC population

  45. Experimental Setup

  46. Experimental Setup

  47. Multiple Choice vs. Deliberate Practice • Research Questions: • Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?) • Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments? • Do “deliberate practice” result in any detectable learning gain?

  48. Thank you Comments and Questions?

  49. Discussion Questions • Other innovative instructional methods worth putting online  and testing via AB experiment?

  50. Discussion Questions • How should we design our experiments to suit the (unique) MOOC population? • How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?

More Related