In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning
An attempt to find meaning in three acts • Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher) • Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker) • Personalization: Word problem content (Candace Walkington)
Contiguity Early Version Commercial Version (Carnegie Learning) Research Version (Carnegie Mellon) Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum. Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
Geometry Contiguity • Design and field experimentation • Butcher and Aleven (2008) • Diagram interaction led to better transfer and retention • Analysis of impact • Hausmann and Vuong (2012) • Unit-level effects mixed • Advantage for harder skills
Lessons • Change is constant • Transition from research to production always requires adaptation
Bayesian Knowledge Tracing Cognitive tutor traces these skills differently
Learning Curve Parameter Fitting • Field study looking at learning area of geometric figures • One group used adjusted learning parameters based on previous year’s data • Optimized group took 12% less time to reach same performance • Significant learning gain in both groups • No difference in learning gain between groups (p = 0.772 )
Lessons • Learning efficiency is a great outcome • Small, systemic changes can have big impact • Optimizing skills requires appropriate skill model • Koedinger, McLaughlin and Stamper (2012) - LFA
Personalization field study • Students who got problems related to their interests made fewer errors • Also affected subsequent unit • Interaction with readability
Lessons • Content matters • Challenge for knowledge component modeling • Are we personalizing preferences, reading level or both?
Summary • It’s not about whether A is better than B • It’s about whyA is better than B