Interaction of Content Channels in Classroom Presentations

Repeat Intro of Self Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman Mention: -Richard -Jonathan In Audience

Background • Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.) • Our focus is on the use of digital ink in the classroom setting • We want to capture/playback/analyze these channels intelligently

Why do we want to analyze content channels? • We want to make it easier to interact with electronic materials • Better search and navigation of presentations • Accessibility for the hearing/learning/visually impaired • Generating text transcripts • Recognizing high level behaviors Conversion to: Braille/Screen Reader

Distance Learning Classes

Classroom Presenter • General tool for giving presentations on the Tablet PC • Many similar systems – our findings applicable to all such systems • Enables writing directly on the slides • Tablet PC enables high-quality digital ink • Used in over 100 courses so far • Allows us to collect real usage data

Questions We Wanted to Explore • High Level Question: What is the potential for automatic analysis of archived content? • Other Questions: • How well can digital ink be recognized by itself? • How closely are different content channels tied together? • Speech and Ink? • Ink and Slide Content? • Can we identify high level behaviors by analyzing the content channels?

Research Methodology • We wanted to understand what real presentation data is like • We collected several 100’s of hrs. of recorded lectures from distance learning classes • Analyzed the data in various ways to help answer our guiding questions. • Note: All examples given here are from real presentations!

Outline • Motivation • Handwriting Recognition • Joint Writing and Speech Recognition • Attentional Mark Identification • Activity Inference: Recognizing Corrections

Handwriting Recognition • Classroom lectures on Tablet PC offer interesting challenges for handwriting recognition • Somewhat Awkward • Small Surface to Write On • Bad Angle to the Tablet PC • Hastily Written • Concentrating on Speaking • Excited / Nervous

Recognition Examples Mark: Success/Failure • The Good: • The Bad: • The Ugly:

Recognition Procedure • Studied isolated words/phrases written on slides • Removed all non-textual ink • Fed through the Microsoft Handwriting Recognizer • No training done!

Handwriting Recog. Results Mention That These Results Are Surprisingly Good! Each Row Represents a Different Lecturer

Outline • Motivation • Handwriting Recognition • Joint Writing and Speech Recognition • Attentional Mark Identification • Activity Inference: Recognizing Corrections Look at Potential

Joint Writing and Speech Recognition • Co-expression of ink and speech • Is digital ink spoken as it is written? • Yes, but how often? How “closely” to the written text? • Can speech be used to disambiguate handwriting? • Can handwriting be used to disambiguate speech? (incl. deictic references) In Time/Accuracy, Wanted Empirical Evidence

Examples Eswaran, Gray, Loric, Traiger • Difficult for Speech and Ink Recognition • Difficult Written Abbreviations • Speech/Ink Used to Disambiguate Ink/Speech DigiMon Java 2 Enterprise Edition corn flakes

Experiment • Examined instances of isolated word writing • Selected word writing episodes at random but uniformly from the various instructors • Generated transcripts manually from the audio • Checked whether the instructor spoke the exact word written • Measured the time between the written and spoken word

Speech/Text Co-occurrence Results Each Row Represents a Different Lecturer

Attentional Mark Identification • Attentional Marks are… • First step is to Identify a stroke as a mark • Tying Attentional Marks to slide content is important • Attentional Ink provides a concrete link between speech and slide content!

Example

Method • Segmentation • Few strokes • Close spatial and temporal proximity • Mark Recognition • Created hand tuned classifiers for: Circles, Lines, Bullets/Ticks • Matched with slide content

Experiment • Identified and Classified Attention Marks by Hand • Two different people per slide • Identified type of mark as well as slide content mark referred to • Identified Attention Marks Automatically • Compared Resulting Identification

Content Matching Issues • Hard to determine exactly what content a mark refers to Not just a recognition Issue, but also related to HOW people draw

Content Matching Cont. • Granularity of content parsing can be an issue

Attentional Ink Recognition Accuracy

Recongizing Corrections • Why? • Want to answer the broad question: - “Can we recognize patterns of activity by analyzing the ink and speech channels?” • Useful for Presenters -Occurs frequently (about 1-3 per lecture) • But Non-trivial Our vision allows false positives

Recognizing Corrections • Identified Six Types of Corrections Looked through large # of lectures, wide range of marks

Example Results No Table Because: 1. Not a robust experiment 2. Proof of Concept

Wrap-up • We wanted to understand the nature of real data to direct our focus when building tools for automatic analysis • Our studies provided the necessary understanding to accomplish this

Wrap-up (Cont.) ALL OPEN for Refinement Specific Results: • Basic handwriting recognition is surprisingly good • Very strong co-occurrence of written and spoken words • We were able to identify attentional marks and the content associated with them • Activity Recognition: There are certain high-level activities that we can identify

Questions? E-mail cmprince@cs.washington.edu jonsu@cs.washington.edu Classroom Presenter Website http://www.cs.washington.edu/education/dl/presenter/

Interaction of Content Channels in Classroom Presentations