speech ink and slides the interaction of content channels n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Speech, Ink, and Slides: The Interaction of Content Channels PowerPoint Presentation
Download Presentation
Speech, Ink, and Slides: The Interaction of Content Channels

Loading in 2 Seconds...

play fullscreen
1 / 32

Speech, Ink, and Slides: The Interaction of Content Channels - PowerPoint PPT Presentation


  • 111 Views
  • Uploaded on

Repeat Intro of Self. Speech, Ink, and Slides: The Interaction of Content Channels. Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman. Mention: -Richard -Jonathan In Audience. Background.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Speech, Ink, and Slides: The Interaction of Content Channels' - gates


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
speech ink and slides the interaction of content channels

Repeat Intro of Self

Speech, Ink, and Slides: The Interaction of Content Channels

Richard Anderson

Crystal Hoyer

Craig Prince

Jonathan Su

Fred Videon

Steve Wolfman

Mention:

-Richard

-Jonathan

In Audience

background
Background
  • Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.)
  • Our focus is on the use of digital ink in the classroom setting
  • We want to capture/playback/analyze these channels intelligently
why do we want to analyze content channels
Why do we want to analyze content channels?
  • We want to make it easier to interact with electronic materials
    • Better search and navigation of presentations
    • Accessibility for the hearing/learning/visually impaired
    • Generating text transcripts
    • Recognizing high level behaviors

Conversion to:

Braille/Screen Reader

classroom presenter
Classroom Presenter
  • General tool for giving presentations on the Tablet PC
  • Many similar systems – our findings applicable to all such systems
  • Enables writing directly on the slides
  • Tablet PC enables high-quality digital ink
  • Used in over 100 courses so far
  • Allows us to collect real usage data
questions we wanted to explore
Questions We Wanted to Explore
  • High Level Question: What is the potential for automatic analysis of archived content?
  • Other Questions:
    • How well can digital ink be recognized by itself?
    • How closely are different content channels tied together?
      • Speech and Ink?
      • Ink and Slide Content?
    • Can we identify high level behaviors by analyzing the content channels?
research methodology
Research Methodology
  • We wanted to understand what real presentation data is like
  • We collected several 100’s of hrs. of recorded lectures from distance learning classes
  • Analyzed the data in various ways to help answer our guiding questions.
    • Note: All examples given here are from real presentations!
outline
Outline
  • Motivation
  • Handwriting Recognition
  • Joint Writing and Speech Recognition
  • Attentional Mark Identification
  • Activity Inference: Recognizing Corrections
handwriting recognition
Handwriting Recognition
  • Classroom lectures on Tablet PC offer interesting challenges for handwriting recognition
    • Somewhat Awkward
      • Small Surface to Write On
      • Bad Angle to the Tablet PC
    • Hastily Written
      • Concentrating on Speaking
      • Excited / Nervous
recognition examples
Recognition Examples

Mark: Success/Failure

  • The Good:
  • The Bad:
  • The Ugly:
recognition procedure
Recognition Procedure
  • Studied isolated words/phrases written on slides
  • Removed all non-textual ink
  • Fed through the Microsoft Handwriting Recognizer
  • No training done!
handwriting recog results
Handwriting Recog. Results

Mention That These Results Are Surprisingly Good!

Each Row Represents a Different Lecturer

outline1
Outline
  • Motivation
  • Handwriting Recognition
  • Joint Writing and Speech Recognition
  • Attentional Mark Identification
  • Activity Inference: Recognizing Corrections

Look at Potential

joint writing and speech recognition
Joint Writing and Speech Recognition
  • Co-expression of ink and speech
    • Is digital ink spoken as it is written?
      • Yes, but how often? How “closely” to the written text?
    • Can speech be used to disambiguate handwriting?
    • Can handwriting be used to disambiguate speech? (incl. deictic references)

In Time/Accuracy, Wanted Empirical Evidence

examples
Examples

Eswaran, Gray, Loric, Traiger

  • Difficult for Speech and Ink Recognition
  • Difficult Written Abbreviations
  • Speech/Ink Used to Disambiguate Ink/Speech

DigiMon

Java 2 Enterprise Edition

corn flakes

experiment
Experiment
  • Examined instances of isolated word writing
  • Selected word writing episodes at random but uniformly from the various instructors
  • Generated transcripts manually from the audio
  • Checked whether the instructor spoke the exact word written
  • Measured the time between the written and spoken word
speech text co occurrence results
Speech/Text Co-occurrence Results

Each Row Represents a Different Lecturer

outline2
Outline
  • Motivation
  • Handwriting Recognition
  • Joint Writing and Speech Recognition
  • Attentional Mark Identification
  • Activity Inference: Recognizing Corrections
attentional mark identification
Attentional Mark Identification
  • Attentional Marks are…
  • First step is to Identify a stroke as a mark
  • Tying Attentional Marks to slide content is important
  • Attentional Ink provides a concrete link between speech and slide content!
method
Method
  • Segmentation
    • Few strokes
    • Close spatial and temporal proximity
  • Mark Recognition
    • Created hand tuned classifiers for: Circles, Lines, Bullets/Ticks
  • Matched with slide content
experiment1
Experiment
  • Identified and Classified Attention Marks by Hand
    • Two different people per slide
    • Identified type of mark as well as slide content mark referred to
  • Identified Attention Marks Automatically
  • Compared Resulting Identification
content matching issues
Content Matching Issues
  • Hard to determine exactly what content a mark refers to

Not just a recognition Issue, but also related to HOW people draw

content matching cont
Content Matching Cont.
  • Granularity of content parsing can be an issue
outline3
Outline
  • Motivation
  • Handwriting Recognition
  • Joint Writing and Speech Recognition
  • Attentional Mark Identification
  • Activity Inference: Recognizing Corrections
recongizing corrections
Recongizing Corrections
  • Why?
    • Want to answer the broad question:

- “Can we recognize patterns of activity by analyzing the ink and speech channels?”

    • Useful for Presenters

-Occurs frequently (about 1-3 per lecture)

    • But Non-trivial

Our vision allows false

positives

recognizing corrections
Recognizing Corrections
  • Identified Six Types of Corrections

Looked through large # of lectures, wide range of marks

example results
Example Results

No Table Because:

1. Not a robust experiment

2. Proof of Concept

wrap up
Wrap-up
  • We wanted to understand the nature of real data to direct our focus when building tools for automatic analysis
  • Our studies provided the necessary understanding to accomplish this
wrap up cont
Wrap-up (Cont.)

ALL OPEN for Refinement

Specific Results:

  • Basic handwriting recognition is surprisingly good
  • Very strong co-occurrence of written and spoken words
  • We were able to identify attentional marks and the content associated with them
  • Activity Recognition: There are certain high-level activities that we can identify
questions
Questions?

E-mail

cmprince@cs.washington.edu

jonsu@cs.washington.edu

Classroom Presenter Website

http://www.cs.washington.edu/education/dl/presenter/