Educational Data Mining Success Stories

1. Educational Data Mining Success Stories • Jack Mostow • Project LISTEN(www.cs.cmu.edu/~listen) • “Home run”: demonstrable increase in learning • “Base hit”: likely to improve learning by informing: • Educational researchers • Teachers • Students • Tutor developers • Automated tutors • IERI PI Meeting panel: Data Mining and Analysis • Funding: National Science Foundation 1

2. 1. Informing (Reading) Researchers: Microanalysis of repeated vs. wide reading • Mostow & Beck (SSSR2005) • Predicted a student’s speedup on a word • Reduction in word reading time • From one (“practice”) encounter of a word • To the next (“test”) encounter in a new context • By training a linear model based on: • What was the student’s reading level? • How many letters long was the word? • How often had the student seen the word before? • Had the student seen the practice sentence before? • (More predictor variables …) 2

3. Results based on Reading Tutor data N = 243,172 speedup opportunities for 352 gr 1-6 students. Speedup averaged 18 ms per encounter (for the first seven). Higher readers sped up less: 3 ms less per grade level. Longer words sped up more: 2 ms more per letter. A new practice sentence helped 27 ms more than an old one. • Wide reading beat rereading! 3

4. Scheines et al. (JECR 2005): TETRAD related variables logged in “Causal and Statistical Reasoning” (N = 47 students): pre: pre-test % quiz: average % on quizzes final: % on final exam print: % of modules printed Instrument to estimate effects of voluntary questions so as to infer causality from observation voluntary questions: % attempted Positive effect on performance Online-only, inhibited by printing Telling next-year students helped! (Standardized regression coefficients of variables regressed on parents) 2. Informing Teachers: What influences student outcomes in an online course? 4

5. 3. Informing students:Proactive help to prevent likely mistakes • Merceron & Yacef (AIED 2005): • Induced prediction rules “if missed X, likely to miss Y” for web-based logic tutor (N = 860 students) • Warn students before predicted mistakes occur • Warning phrased by teacher = tutor designer 5

6. 4. Informing tutor designers: Does explaining new vocabulary help more than just reading in context? • Aist (2001 PhD): Explain some new words; later, test all. • Did kids do better on explained vs. unexplained words? • Overall: NO; 38%  36%, N = 3,171 trials • Rare, 1-sense words tested 1-2 days later: YES! 44% >> 26%, N = 189. 6

7. 5a. Informing tutors: Infer knowledge from behavior Average learning curve for 21 cognitive rules • Corbett et al. (1995): Knowledge tracing updates at each step the probability that the student has learned the relevant rule. • Helps tutor decide what to teach next • Predicts error rate with r = 0.85 (0.90 after refining the rules) 7

8. 5b. Informing tutors: learn what to do Data from prior users of tutor Teaching policy Tutor action “try again” Simulated student (predicts effects of tutor actions) Tutorial agent Result “correct answer, took 15 sec.” • Beck et al. (AAAI 2000): learned a teaching policy for a given goal, e.g. “problems average 30 sec” • Learned policy cut time per problem by 30% (p<.001) • N = 58 students using middle school math tutor in classroom 8

9. Summary • Educational data mining can inform: • Educational researchers • Wide reading apparently beat rereading (Mostow & Beck, SSSR2005) • Teachers • Infer causal effects of observed student choices (Scheines et al., JECR 2005) • Students • Warn of likely mistakes to prevent them (Merceron & Yacef, AIED 2005) • Tutor developers • Discover not just whether an intervention works, but when (Aist PhD 2001) • Automated tutors • Infer student knowledge from student behavior (Corbett et al., 1995) • Learn teaching policies that improve outcomes (Beck et al., AAAI 2000) 9