Evaluation of Multi modal Input for Entering Math ematical Equations on the Co mputer

1. Evaluation of Multimodal Input for Entering Mathematical Equations on the Computer Lisa Anthony, Jie Yang, Kenneth R. Koedinger Human-Computer Interaction Institute Carnegie Mellon University, Pittsburgh, PA

2. Computer-Based Math Tools Maple V, Mathematica, Matlab; Microsoft Equation Editor, … • Programming constructs or special syntax • Linearization of mental representation • Difficult to revise structure in template-based editors MATHEMATICA MICROSOFT EQUATION EDITOR

3. Computer-Based Math Tools Maple V, Mathematica, Matlab; Microsoft Equation Editor, … • Problems: • Large learning curve • Tied to keyboard and mouse input MATHEMATICA MICROSOFT EQUATION EDITOR

4. Pen-Based Computing • Notations already exist for paper-based math • Affordance for spatial representation • Especially good for students learning math • Problem: recognition accuracy

5. Solution: Multimodal Input • Increased robustness • Better recognition accuracy with multiple input streams (Oviatt, 1999; pen gestures + speech) • Consider both repair-only and simultaneous input in both streams

6. Motivation for User Study • Literature says typing is faster (Brown, 1988) • Compared inputting paragraphs of English text • Math is different domain • Little evaluation done of pen-based equation input • Systems constrained by recognition accuracy (Smithies et al, 2001)

7. User Study: Design • Input method • Equation complexity • Number of characters in equation • Number of “complex” symbols (e.g., √ and ∑) “Recognition” means system tries to interpret user input.

8. User Study: Participants • 48 paid participants (27 male, 21 female) • Undergraduate/graduate, full-time/part-time students at Carnegie Mellon • Native English speakers only • Most (33 of 48) had no experience with MSEE before study

9. User Study: Procedure • Entered math equations on TabletPC • 36 equations (7 + 2 practice per condition) • Conditions counterbalanced across participants • Instructions for each condition • No prompting for specific ways of expressing equations • 5 min “explore time” for MSEE

10. User Study: Measures • Time per equation • Number of errors per equation (corrected and uncorrected) • User preferences before and after session Equation entry screen in handwriting condition.

11. User Study: Sample Stimuli

12. User Study: Results • Average time in seconds per equation by condition. Error bars show 95% confidence interval (CI).

13. User Study: Results • Mean number of user errors made per equation by condition. Error bars show 95% CI.

14. User Study: Results • Preference questionnaire rankings of each condition on a 5-point Likert scale. Error bars show 95% CI. Pre-test Post-test

15. Conclusions • Handwriting faster, more efficient, and more enjoyableto novice users than standard keyboard-and-mouse • Handwriting-plus-speech faster and better liked than keyboard-and-mouse • Handwriting-plus-speech not much worse than handwriting alone, so multimodal may be a winner for technology reasons

16. Further Analyses • Transcription of spoken input as corpus for generation of language model • Consistency across and within users in handwriting and speech • Ambiguity resolution • Self correction • Pausing and synchronization in multimodal input • Greater variability within speech condition than within handwriting condition

17. Further Analyses • Transcription of spoken input as corpus for generation of language model • Consistency across and within users in handwriting and speech • Ambiguity resolution • Self correction • Pausing and synchronization in multimodal input • Greater variability within speech condition than within handwriting condition (Supported by Oviatt et al, 2005)

18. Questions? • Project Webpage: • http://www.cs.cmu.edu/~lanthony/research/multimodal/ • Pittsburgh Science of Learning Center: • http://www.learnlab.org/index.php

19. References • Brown, C.M.L.: Comparison of Typing and Handwriting in “Two-Finger Typists.” Proceedings of the Human Factors Society (1988) 381–385. • Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576–583. • Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53–67. • Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4–14.

20. Other References in Paper • Anderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.: Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167–207. • Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557–582. • Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63–68. • Microsoft.: Microsoft Word User’s Guide Version 6.0 (1993), Microsoft Press. • Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257–285.