Introduction

1. Teaching Reading to Children with Learning and Cognitive Disabilities Using Intelligent Animated Characters Sponsored by the National Science Foundation Cole, R., Schwartz, S., Corson, L., Corson, K., Davis, G., Doxas, I., Gavin, W., Jie, Y., Jiyong, M., Ngampatipatpong, N., Snyder, L., Tuantranont, J., Wade-Stein, D. and Wise, B. University of Colorado-Boulder Center for Spoken Language Research Introduction A five year multi-agency, multi disciplinary project is currently underway to create computer-based interactive books and tutors to teach reading and comprehension skills to children with learning and cognitive disabilities. The books and tutors use state of the art speech and language technologies including speech synthesis, speech recognition, 3-D facial animation and highly accurate visible speech production. • The Interactive Books • Appear like a book on the screen. • Books are selected to match child’s reading level. • Children select animated tutor or “coach”. • Teacher or child selects appropriate books and interactive features. • Animated coach guides child through book. • Coach can read words, sentences paragraphs or whole page. • Coach asks comprehension, inference and summary questions using speech recognition to determine accuracy of responses. • Each level of book has the capability of invoking specific reading tutors if a child needs more foundational skill training. • Animation Team • The animated characters have been integrated into the reading tutors and interactive books. • The animators have created realistic characters with accurate tongues and mouth movements for visual speech perception, and believable and personalized facial expressions and hand gestures. • The animators have created a working system that controls the family of full-body animated characters. • The character’s tongues move to specified targets for each phoneme, and their lips move to specified viseme targets. • Each eye, eyeball and eyebrow can be controlled independently, as can head movements, body movements, and gestures that combine movements of different body parts. • Characters can be placed into a mode where they look around, blink and change facial expressions during interactions. • Agencies Involved in Project • University of Colorado-Center for Spoken Language Research • Interactive book and tutor development; research, animation, evaluation, project management • Boulder Valley School District • Participatory design; design, test, pilot, and evaluate books and tutors • Oregon Graduate Institute • Children’s speech recognition • University of California at Santa Cruz • Facial animation research and tools • University of California at San Diego • Computer vision research; head tracking, speech reading and facial expressions • Universidad de las Americas; Puebla, Mexico • Spanish recognition research and Spanish tutors Feature Selection Choose your coach. Select what you want the coach to read. AVI or still pictures? Should the coach ask questions? • The Vision • To provide teachers and students with learning tools that are engaging, effective, well informed and will facilitate reading improvement. • To have intelligent animated tutors interact with learners as an effective teacher would through face to face conversational interaction. • To have animated tutors that can perceive, understand and produce auditory and visual behaviors. • Authoring Your Own Book • Children and teachers can create and edit their own books. • Animated coach can accurately translate new text to speech. Current cast of characters Feature Selection Add text. Add picture. The 3-D parametric tongue model for Ms. Gurney. Add AVI. Change arrow. • The Reading Tutors • The reading tutors will employ the animated characters in a variety of motivating and reinforcing activities to teach foundational skills such as: • Letter identification • Phonological awareness • Decoding and encoding • Prefixes/suffixes • Word derivations • Etc. • The reading skills can be taught in a teacher prescribed and adaptable sequence. • Mastery of a skill will automatically move the student to the next level. • Data is automatically logged as students proceed through the tutor. Create questions. Opening screen for authoring your own book. Arm and hand gestures for Percy and Rowdy. • Basic Research • Speech recognition for children’s speech • Currently, hundreds of speech samples are being collected from children in the BVSD. • These samples will be used to improve recognizers for children’s speech. • Visual recognition systems • Video samples are concurrently being collected in BVSD during the speech sample collection. • These samples are being analyzed to produce recognizers that can track the child’s head and eye movement, recognize visible speech and recognize facial expressions so as to infer the reader’s affective state. Sample Reading Tutors Text screen allows you to add text for a story. • Applied Research • Currently 6-8% of children have a learning or cognitive disability that interferes with reading achievement. • These students are often not identified until 3rd grade; an age when catching up is difficult. • Children from six demographically matched schools are receiving kindergarten literacy screeners to identify learning difficulties early. • Children from three of the schools will use the reading tutors and interactive books. • Children from the other three schools will receive an alternative treatment. • A longitudinal study using standardized reading measures will assess the progress of the students under the two conditions. Four Square- The animated character asks the child to find the correct word. Sequences of letters, words or pictures can be entered into the boxes. Picture screen allows you to import a picture and AVI screen allows you to import an AVI. The Future All technologies, tools and applications developed during the project will be placed in the public domain so that teachers and parents everywhere can use them at school and at home. Let’s Build- The animated character asks the child to build and change words. The character then gives feedback and reads the child’s words. Question screen allows you to create comprehension questions and determine acceptable answers.