Download
creating a multimodal design environment using speech and sketching n.
Skip this Video
Loading SlideShow in 5 Seconds..
Creating a Multimodal Design Environment Using Speech and Sketching PowerPoint Presentation
Download Presentation
Creating a Multimodal Design Environment Using Speech and Sketching

Creating a Multimodal Design Environment Using Speech and Sketching

474 Views Download Presentation
Download Presentation

Creating a Multimodal Design Environment Using Speech and Sketching

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Creating a Multimodal Design Environment Using Speech and Sketching Aaron Adler Student Oxygen Workshop September 12, 2003

  2. Goals for System • Create a natural user interface for a design environment • Not command based • Create a natural multimodal UI by combining speech and sketching • Some things more easily expressed with sketching and speaking

  3. ASSIST • Natural sketching tool for mechanical engineering designs • Stylus-style input devices

  4. Motivating Example • Newton’s Cradle

  5. Natural Language • Need to determine how users naturally talk about the devices • Videotaped 6 users sketching 6 drawings at a non-interactive whiteboard • Transcribed data and produced time-stamped speech and sketching events

  6. Video of People Sketching

  7. Segmenting the Data • Once the data was transcribed, graphs and charts were created to help analyze the data • Rules were created to encapsulate the knowledge about segmentation

  8. Rules • Three types of rules • Rules about the text of the speech • Repeated words, mumbled words, key words • Rules about gaps between speech and sketching • Long pauses, timing of speech and sketching events • Rules about groups of sketched items • Similarly shaped objects

  9. And And then Then So Next Also mumbled words, ahhh and ummm, are important We have There is We’ve got It’s I’ll Some Key Words from the Speech

  10. WATCH • Rule output too large, need tool to view relationships between rules • WATCH created to view output of rules as a timeline

  11. Rule Layout

  12. Results • Software matched 24 of 29 break points • Found an additional 18 break points, 10 which were harmless, 7 were ambiguous, and 1 was wrong • Hand segmentation had all events to examine at once, spatial relationships • Rules kept general to avoid over fitting

  13. Harmless “<hmm>” “I’m puzzled as to how to indicate that” <<extra break>> “equal size of” “the suspended balls”

  14. Ambiguous [draws top anchor] “The slopes are fixed in position” [draws middle ramp] [draws middle anchor] <<extra break>> [draws bottom ramp] “slope”

  15. Speech System • Speech done by SLS Sapphire system • The transcribed speech was used as a basis to generate a recognizer (missing words were added) • Speaker independent • Open microphone, continuous recognition

  16. ASSIST Modifications • ASSIST needed some modification to allow the system to manipulate the widgets • Identical, touching, equally spaced functions • Also needed to send the current widgets to the rule system to be combined with the speech input

  17. System Overview • Combines ASSIST and speech recognizer using the developed rules

  18. Ambiguity • Need some inherent knowledge of pendulums, wheels, etc. • Car on ramp example • “Two identical wheels” • Need to know what a wheel is! • Where should this knowledge go? • Top down view – speech triggers search for pendulum

  19. How it Finds the Pendulums • Based around nouns and adjectives • Speech like: “There are three identical touching pendulums.” • Look though widgets around that time • Extract pendulums from group of possible widgets • Looking for an attached rod and circle • If the speech and the sketch disagree about the number of pendulums, don’t do anything

  20. The System in Action

  21. Related work • Work at OGI by Oviatt and Cohen • ASSISTANCE • Several other command-based systems

  22. Future Work • Larger vocabulary • Using Joshua instead of JESS • Learning new vocabulary and corresponding sketches • Next generation Blackboard-based system