A Trace-based Framework for Analyzing and Synthesizing Educational Progressions

1. A Trace-based Framework for Analyzing and Synthesizing Educational Progressions Erik Andersen SumitGulwaniZoranPopović

2. Procedural Learning

3. Software Learnability Dong et al. CHI 2012

5. Flow Anxiety Challenge Boredom Skill M. Csikszentmihalyi

6. Zone of Proximal Development can’t do yet can do with guidance can do now L. Vygotsky

8. Procedural Traces • What are traces? • What can we do with traces?

10. NO NO report “less” same # of digits? greater # of digits? YES report “greater” YES NO NO report “less” first digit same? first digit greater? YES report “greater” YES NO NO report “less” second digit same? second digit greater? YES report “greater” YES ...

11. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES ...

12. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES L ...

13. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES H L, ...

14. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES DS L, H, ...

15. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES DG L, H, DS, ...

16. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES DDS L, H, DS, DG, ...

17. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES L, H, DS, DG, DDS, DDG… ...

18. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES Compare 313 to 324. ...

19. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES Compare 313 to 324. ...

20. What can we do with traces? • Rank their difficulty • Analyze and compare progressions • Synthesize new progressions

21. What can we do with traces? • Rank their difficulty • Analyze and compare progressions • Synthesize new progressions

22. Partial Ordering • Loop executes longer = harder • more digits in number comparison • more columns in subtraction • Exceptional conditions = harder • if boolean expression:statement • borrowing in subtraction • carrying in addition

23. n-grams

24. 1-grams the boy threw the ball

25. 2-grams the boy threw the ball

27. n-gram based Partial Ordering for some integer n if T1contains all of the n-grams of T2 then T1> T2 1-grams → statement coverage -grams → path coverage

28. What can we do with traces? • Rank their difficulty • Analyze and compare progressions • Synthesize new progressions

29. if length(n1) < length(n2): returnless L iflength(n1) > length(n2): returngreaterH fori ← 0 to length(n1) – 1: D ifn1[i]< n2[i]: returnlessS ifn1[i]> n2[i]: returngreaterG returnequal

30. JUMP Math

31. JUMP Math Singapore Math

32. JUMP Math Singapore Math

33. What can we do with traces? • Rank their difficulty • Analyze and compare progressions • Synthesize new progressions

34. if length(n1) < length(n2): returnless L iflength(n1) > length(n2): returngreaterH fori ← 0 to length(n1) – 1: D ifn1[i]< n2[i]: returnlessS ifn1[i]> n2[i]: returngreaterG returnequal DDS 11 < 12

36. DragonBox

37. DragonBox

39. Validating Our Partial Ordering • 20 participants • 20 level pairs • Users ranked pairs: easier, harder, the same • Calculated Root-Mean-Squared-Error • our ordering: 0.35 • random baseline: 1.00

40. Summary new representation of problem difficulty allows us to: • rank problem difficulty • analyze and compare progressions • generate an entire progression pragmatic way to make content fit theory

41. Acknowledgments • Help: David Yamanoha and RyokoNozawa • Funding: Bill and Melinda Gates Foundation, Officeof Naval Research, Hewlett Foundation, Adobe, and Intel • Contact me: eland@cs.washington.edu

44. From here • automate other things • tutorials • diagnosis of student errors

45. Flow Anxiety Challenge Boredom Skill

46. Implications • Games • Education

47. “Games are a series of interesting decisions.” – Sid Meier

48. Artifacts

49. Artifacts Progressions

50. L NO NO report “less” same # of digits? greater # of digits? YES D H report “greater” YES S NO NO report “less” first digit same? first digit greater? YES D G report “greater” YES S NO NO report “less” second digit same? second digit greater? YES D G report “greater” YES Compare 32 to 314. ...