1 / 25

Speech, Ink, and Slides: The Interaction of Content Channels

Speech, Ink, and Slides: The Interaction of Content Channels. Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman. Background. Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.)

jill
Download Presentation

Speech, Ink, and Slides: The Interaction of Content Channels

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman

  2. Background • Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.) • Our focus is on the use of digital ink in the classroom setting • We want to capture/playback/analyze these channels intelligently

  3. Why do we want to analyze content channels? • We want to make it easier to interact with electronic materials • Better search and navigation of presentations • Accessibility for the hearing/learning/visually impaired • Generating text transcripts • Recognizing high level behaviors

  4. Distance Learning Classes

  5. Classroom Presenter • General tool for giving presentations on the Tablet PC • Many similar systems – our findings applicable to all such systems • Enables writing directly on the slides • Tablet PC enables high-quality digital ink • Used in over 100 courses so far • Allows us to collect real usage data

  6. Questions We Wanted to Explore • High Level Question: What is the potential for automatic analysis of archived content? • Other Questions: • How well can digital ink be recognized by itself? • How closely are different content channels tied together? • Speech and Ink? • Ink and Slide Content? • Can we identify high level behaviors by analyzing the content channels?

  7. Research Methodology • We wanted to understand what real presentation data is like • We collected several 100’s of hrs. of recorded lectures from distance learning classes • Analyzed the data in various ways to help answer our guiding questions. • Note: All examples given here are from real presentations!

  8. Outline • Motivation • Handwriting Recognition • Joint Writing and Speech Recognition • Activity Inference: Recognizing Corrections • Attentional Mark Identification

  9. Handwriting Recognition • Classroom lectures on Tablet PC offer interesting challenges for handwriting recognition • Somewhat Awkward • Small Surface to Write On • Bad Angle to the Tablet PC • Hastily Written • Concentrating on Speaking • Excited / Nervous

  10. Recognition Examples • The Good: • The Bad: • The Ugly:

  11. Recognition Procedure • Studied isolated words/phrases written on slides • Removed all non-textual ink • Fed through the Microsoft Handwriting Recognizer • No training done!

  12. Handwriting Recog. Results

  13. Outline • Motivation • Handwriting Recognition • Joint Writing and Speech Recognition • Activity Inference: Recognizing Corrections • Attentional Mark Identification

  14. Joint Writing and Speech Recognition • Co-expression of ink and speech • Is digital ink spoken as it is written? • Yes, but how often? How “closely” to the written text? • Can speech be used to disambiguate handwriting? • Can handwriting be used to disambiguate speech? (incl. deictic references)

  15. Experiment • Examined instances of isolated word writing • Selected word writing episodes at random but uniformly from the various instructors • Generated transcripts manually from the audio • Checked whether the instructor spoke the exact word written • Measured the time between the written and spoken word

  16. Speech/Text Co-occurrence Results

  17. Examples • Difficult for Speech and Ink Recognition • Difficult Written Abbreviations • Speech/Ink Used to Disambiguate Ink/Speech

  18. Outline • Motivation • Handwriting Recognition • Joint Writing and Speech Recognition • Activity Inference: Recognizing Corrections • Attentional Mark Identification

  19. Recongizing Corrections • Why? • Want to answer the broad question: - “Can we recognize patterns of activity by analyzing the ink and speech channels?” • Useful for Presenters -Occurs frequently (about 1-3 per lecture) • But Non-trivial

  20. Recognizing Corrections • Identified Six Types of Corrections

  21. Example Results

  22. Attentional Marks (In Brief)

  23. Wrap-up • We wanted to understand the nature of real data to direct our focus when building tools for automatic analysis • Our studies provided the necessary understanding to accomplish this

  24. Wrap-up (Cont.) Specific Results: • Basic handwriting recognition is surprisingly good • Very strong co-occurrence of written and spoken words • Activity Recognition: There are certain high-level activities that we can identify

  25. Questions? E-mail Craig Prince - cmprince@cs.washington.edu Jonathan Su- jonsu@cs.washington.edu Classroom Presenter Website http://www.cs.washington.edu/education/dl/presenter/

More Related