Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

1. Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities Girijesh Prasad Intelligent Systems Research Centre (ISRC) University of Ulster, Magee Campus, Derry, N. Ireland, United Kingdom Vijit Prabhu Computer Science & Engineering Indian School of Mines Dhanbad, Jharkhand, India

2. Outline • Introduction • Proposed virtual keyboard • A. The working of the VK • B. Modes of input accesss • C. The layout design • Evaluation and testing • A. Evaluation of the layout design • B. User trials • 1. Types of errors • 2. Testing methodology • IV. Results and discussions • V. Conclusion Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

3. Introduction • Persons with speech and motor disorders face problems in expressing themselves in an easy and intelligible way. • Computer based augmentative and alternate communication (AAC) systems are developed to assist them. • A Virtual Keyboard (VK), also called as On Screen Assistive Keyboard is a commonly used AAC system. • A VK is characterized by the input modalities and the layout. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

4. Introduction …Continued. • Scanning and access switches require lesser motor control. • Brain-computer interface (BCI) translates the intent of a subject measured from brain signals directly into control command. • Eye tracking which uses fixed gaze or blinking to activate the switch or triggers. • “Hex-o-Spell” [1] virtual keyboard is an EEG based Brain Computer Interface that uses machine learning techniques to identify brain signals. • Dasher[2] is an information-efficient text-entry interface, driven by natural continuous pointing gestures. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

5. The Proposed Virtual Keyboard: The Working Fig. 1(a): Virtual Keyboard: pointer pointing at the sub circle Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

6. The Proposed Virtual Keyboard: The Working Fig. 1(b) Expanded Sub Circle: pointer points at the character F Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

7. The Proposed Virtual Keyboard: Modes of Input • Any one or the combination of three incorporated technologies can be used for access. • The Brain Computer Interfacing (BCI) based on EEG uses ‘vivid imagination’ of a motor activity as a trigger. • The Eye Tracker technology uses prolonged gaze and/or eye blink as a trigger. • Access switches use any active body part such as hand, foot, mouth or head as a trigger Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

8. The Proposed Virtual Keyboard: Layout Design Different alphabets have different frequency of occurrence in English Text. [3] [4] Table I. Relative frequency of the alphabets in English Text Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

9. The Proposed Virtual Keyboard: Layout Design • Different positions of characters in the layout , require different amount of access activity. • Design Principle- higher the frequency of occurrence of character, lower should be the amount of activity required to access it. • Clashes in positioning of character were resolved based on the probability of the character being the first letter of the word. [5] Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

10. The Proposed Virtual Keyboard: Layout Design More probable of being the first letter of word -assigned the lower sub-circle. Table II: Relative frequencies of the first letters of a word in the English language Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

11. The Proposed Virtual Keyboard: Layout Design Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

12. Evaluation & Testing: Evaluation of Layout Design Given a random variable x, with the probability of occurrence p(x), the expected (mean) value of x: E(x) = Σi xip(xi) x -‘total activity’ associated with a character p(x)-probability of the occurrence of the character in the text based on their relative frequencies Alphabetical layout, the expected amount of activity required is computed to be 6.47. For the proposed layout, the same activity drops down 4.55. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

13. Evaluation & Testing: User Trials • Types of Errors : • Wrong Character Selection (WCS):user types a wrong character-needs to use backspace to remove the incorrect character -then type in the correct character. • Wrong Sub Circle Selection (WSCS):user selects the wrong sub-circle-needs to use the ‘Go Back’ option provided in the sub circle • Failed Selection (FS): user realizes that the pointer (arrow) has already moved beyond the correct sub circle /character Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

14. Evaluation & Testing: User Trials • Testing Methodology • Users given the layout design in advance in print • Explained the working of the VK • One sentence to type using the soft switch at four different scan periods 2100ms, 1800ms, 1500ms and 1200ms. • Just soft switch to select button as the pointer rotated and no prediction help was allowed. • Only error free words and sentences with full-stops were considered for evaluation • At the end of each typing session an informal chat • THE QUICK BROWN FOX JUMPS OVER A LAZY DOG. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

15. Results and Discussions: TABLE IV: Result of user trials. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

16. Conclusion: • The VK developed has multiple modes of access, hence people suffering from different levels of disability may be able to use one or more of the access methods • Evaluation of the layout design of the proposed VK clearly establishes the superior performance measured in terms of average per character activity required for composing text • The user trials not only helped prove the robustness of the VK but also helped in performance evaluation of the layout Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

17. Conclusion: Continued • The various scan periods were also evaluated and an attempt was made to determine the most suitable scan times. • User trials with the actual intended users making use of all three modalities is an utmost requirement. • Comparative study with other existing VK like the flat VK will help in better evaluation of design • Work is ongoing to combine BCI and eye-tracker modalities together to create a hybrid modality and evaluate its performance. Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

18. References • [1] Blankertz et al., The Berlin Brain-Computer Interface Presents The Novel Mental Typewriter Hex-O-Spell, In: Proc. of the 3rd International Brain-Computer Interface Workshop and Training Course 2006, September 2 1-24 2006, Verlag der Technischen Universität Graz.. • [2] Dasher-Information-Efficient Text Entry-Hanna Wallach, University of Cambridge/University of Pennsylvania. http://www.mendeley.com/research/dasher-an-efficient-keyboard-alternative/ • [3] Lewand, Robert (2000). Cryptological Mathematics. The Mathematical Association of America. p. 36. ISBN 978-0883857199 • [4] Lee, E. Stewart; Essays about Computer Security; University of Cambridge Computer Laboratory, p. 181 • [5] Letter Frequency, Wikipedia, http://en.wikipedia.org/wiki/Letter_frequency Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

19. Thank You!