Scientifically Informed Digital Learning Interventions

1. Scientifically Informed Digital Learning Interventions Financial and Intellectual Support: • The William and Flora Hewlett Foundation • The National Science Foundation • A.W. Mellon Foundation • Carnegie Mellon University One Example: The Open Learning Initiative at Carnegie Mellon

2. The Challenge • To learn ways to design and build fully web-based courses which by rigorous assessments are proven to be as good or better than traditional teaching methods • Why? • Increasing access • Improving effectiveness • Providing flexibility for faculty and students • Containing costs

3. A Flaw and an Opportunity • Current structure of higher education presents substantial roadblocks to the application of proven results and methodologies from the learning sciences • eLearning interventions, developed by teams rather than individuals, are more conducive to making the practice of education more scientific and effective

4. OLI Guiding Assumptions • Digital learning interventions can make a significant difference learning outcomes • Designs grounded in contemporary learning theory and scientific evaluation have the best chance of achieving that goal • A possible, acceptable outcome of the OLI efforts is failure or mixed failures and successes – we are doing “action research,” not promoting eLearning for its own sake

5. OLI Guiding Assumptions • Formative assessment will be a major feature (and a major component of the cost) of the designs and iterative improvements of the courses • IT staff working with faculty is too limited a partnership – learning scientists, HCI experts, and assessment experts must be part of design, development, production and iterative improvement

6. Open Learning Initiative Courses • Statistics • Modern Biology • Chemistry • French • Engineering Statics • Causal and Statistical Reasoning • Economics • Logic and Proofs • Physics • Empirical Research Methods • Computational Discrete Mathematics

7. Try it Yourself • http://www.cmu.edu/oli • Don’t expect an “OCW experience”…this project has a different set of goals than OCW • “Clicking around” will be unsatisfying: these interventions are designed to support a novice learner in acquiring knowledge working on their own

8. Key Elements in OLI Courses • Theory Based: Course and individual lesson designs based on current theories in the learning sciences • Feedback Loops: Courses record student activity for robust feedback mechanisms • Diversity of Perspectives, Roles and Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

9. Theory Based: Build on Prior/Informal Knowledge

10. Theory Based: Provide Immediate Feedback in the Problem Solving Context

11. Theory Based: Promote Authenticity, Flexibility & Applicability • Learning environments with ambiguous problems that require flexible application of procedural knowledge

12. Feedback Loops in Learning

13. Evaluation • Chemistry: Post-test scores by treatment group show significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out. • Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

14. Evaluation • Statistics 1st Study:

15. Evaluation • CAOS Sample: Increase: 7.9% [t(487) = 13.8, p <.001] • CMU OLI Course Sample: Increase: 11.7% [t(23) = 4.7, p <.001]

16. Evaluation

17. Accelerated Learning Study • Taught Carnegie Mellon Introductory Statistics course in a blended mode (one in class meeting per week) in half a semester • The OLI Statistics course was the “textbook” • OLI course provided the professor immediate feedback on students’ performance • We compared learning outcomes in the two different treatments

18. Accelerated Learning Study • OLI students significantly outperformed Traditional “control” students on the CAOS post-test.

19. Accelerated Learning Study • OLI students showed significantly greater gains (pre to post) than the Traditional “control” students on the CAOS test.

20. Student Satisfaction • End of course survey for online section: • All students reported at an increase in their interest in statistics. • 75% Definitely Recommend 25% Probably Recommend 0% Probably not Recommend 0% Definitely not Recommend

21. Feedback Loop – Current Research Instructors can use such data to adjust their teaching to students’ needs.

22. Learning Curve Analysis on Stoichiometry Data

23. The Vision – Digital Dashboard for Teaching and Learning: • Instructor assigns students to work through online instruction • System collects data as students work • System automatically analyzes and organizes the data to present instructor with the students’ current “learning state” • Instructor reviews this data summary and adapts instruction accordingly

24. The Anticipated Benefits • Instructors get a window onto students’ progress • They can adapt their teaching accordingly • Students get better feedback to monitor and adjust their learning • Strengthens the student-instructor connection

25. Core OLI Community • Faculty Content Experts • Learning Scientists • Human Computer Interaction • Software Engineers • Evaluation/Assessment Specialists • Learners • A community of scholars from diverse disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

26. “Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity” Herbert Simon www.cmu.edu/oli joelms@cmu.edu cthille@cmu.edu (Candace Thille – Director)