auto summarization of audio video presentations l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Auto-Summarization of Audio-Video Presentations PowerPoint Presentation
Download Presentation
Auto-Summarization of Audio-Video Presentations

Loading in 2 Seconds...

play fullscreen
1 / 33

Auto-Summarization of Audio-Video Presentations - PowerPoint PPT Presentation


  • 169 Views
  • Uploaded on

Auto-Summarization of Audio-Video Presentations. Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and Multimedia Group Microsoft Research. Motivation. On-demand multimedia is becoming pervasive Corporate training and communication

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Auto-Summarization of Audio-Video Presentations' - teenie


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
auto summarization of audio video presentations

Auto-Summarization of Audio-Video Presentations

Li-wei He, Elizabeth Sanocki

Anoop Gupta, Jonathan Grudin

Collaboration and Multimedia Group

Microsoft Research

ACM Multimedia 99

motivation
Motivation
  • On-demand multimedia is becoming pervasive
    • Corporate training and communication
      • At Microsoft, over 360 courses online in two years
    • Research seminars
      • Microsoft Research archives about 2 talks daily

ACM Multimedia 99

motivation cont
Motivation (Cont.)
  • Effective summarization and browsing techniques can help viewers utilize time better
    • Audio-video different from text
    • Many approaches possible
      • Time-compression, indexes, highlights, …
  • This talk focuses on:
    • Informational presentations
    • Automatic summarization methods

ACM Multimedia 99

what is a video summary
What Is a Video Summary?
  • Assembled from segments of the original

ACM Multimedia 99

the 4 c s of a good summary
The 4 C’s of a Good Summary
  • Conciseness: as short as possible
  • Coverage: covers key points
  • Context: defines terms before using them
  • Coherence: flows naturally and fluidly

ACM Multimedia 99

talk outline
Talk Outline
  • Introduction
  • Automatic summarization
    • Sources of information in A/V presentations
    • Three algorithms
  • Evaluation

ACM Multimedia 99

sources of information
Sources of Information
  • Audio and video
    • Pitch and pause information
  • Speaker actions
    • Slide-transition points
  • End-user actions
    • Video segments watched by earlier viewers

ACM Multimedia 99

1 slide based method s
1. Slide-based Method (S)
  • Rationale
    • Beginning of a slide marks a new topic
    • Time devoted to slide indicates its importance
  • Algorithm
    • First N% of video for each slide

ACM Multimedia 99

2 pitch based method p
2. Pitch-based Method (P)
  • Rationale
    • Pitch activity indicates the speaker’s emphasis
  • Algorithm (based on Arons ISSLP 94)
    • Compute pitch for every 1ms frame
    • Count the number of frames above a threshold in 15 second windows
    • Select the windows with the most count

ACM Multimedia 99

3 combined method spu
3. Combined Method (SPU)
  • The amount of time that previous viewers spent on a slide indicates importance

ACM Multimedia 99

3 combined method spu12

Average Viewer Count of Slide N

Importance of Slide N

=

Average Viewer Count of Slide N-1

3. Combined Method (SPU)
  • Algorithm
    • Compute importance measure for each slide
    • Allocate summary time for each slide according to the importance measure
    • Use pitch-based algorithm to pick the segments in each slide

ACM Multimedia 99

talk outline13
Talk Outline
  • Introduction
  • Automatic summarization
  • Evaluation
    • Experimental design
    • Results

ACM Multimedia 99

experimental design
Experimental Design
  • To compare summarization techniques
    • Original presenters (authors) created summaries (A) as gold standard
    • Authors wrote quiz questions that covered the content of summaries
    • Objective measure: quiz score improvement after watching a summary
    • Subjective measures: user survey

ACM Multimedia 99

experimental design cont
Experimental Design (Cont.)
  • 4 summary types (S, P, SPU, A)
  • 4 talks chosen from Microsoft training site
  • 24 Microsoft employees were subjects
    • Summary types and talks are counter-balanced within each subject

ACM Multimedia 99

demo summary
Demo Summary

ACM Multimedia 99

quiz score improvement
Quiz Score Improvement
  • As expected, author-created summaries did best
  • No significant difference among the automatic methods

ACM Multimedia 99

survey rating results
Survey Rating Results
  • A >> SPU > P = S

ACM Multimedia 99

percent of value derived
Percent of Value Derived
  • From slide content: 46%
  • From audio content: 36%
  • From video content: 18%

ACM Multimedia 99

conclusions
Conclusions
  • Ability to skim/browse will be key to wide use
  • Automated methods can add significant value
    • Add domain knowledge is important
    • Increasing acceptance over time
  • Evaluation is a key but very difficult

ACM Multimedia 99

conclusions cont
Conclusions (Cont.)
  • Getting the human into the loop
    • Speakers
    • End-users as a group
      • E.g. collaborative filtering
    • End-users as an individual
      • E.g. interactive browsing
  • Visit us at: http://research.microsoft.com/coet

ACM Multimedia 99

interface of a typical talk
Interface of a Typical Talk

Video

VCR-like

controls

Slides

Table of

content

ACM Multimedia 99

summary characteristics
Summary Characteristics
  • Talks were from MS internal training site
    • UI Design, Internet Explorer, Dynamic HTML, Microsoft Transaction Server
  • Average length
    • 20% to 25% of the original
    • 10 to 14 minutes
  • Overlap with author-created summaries was no better than chance

ACM Multimedia 99

survey on the summary just watched
Survey on the Summary Just Watched
  • Concise: It captured the essence of the talk without using too many sentences
  • Coverage: My confidence that it covered the key points of the talk is …
  • Context: It is clear and easy to understand
  • Coherent: It provided reasonable context, transitions, and sentence flow

ACM Multimedia 99

survey rating results26
Survey Rating Results
  • A >> SPU > P = S

ACM Multimedia 99

information not used
Information Not Used
  • Spoken text content
  • Speaker gestures

ACM Multimedia 99

talk outline28
Talk Outline
  • Introduction
    • Motivation
    • Definition of a video summary
    • Attributes of a good summary
  • Automatic summarization
  • Evaluation

ACM Multimedia 99

viewers over time for one talk
Viewers Over Time for One Talk
  • Viewer number decreases overall and within each slide

ACM Multimedia 99

importance measure

Average Viewer Count of Slide N

Importance of Slide N

=

Average Viewer Count of Slide N-1

Importance Measure

ACM Multimedia 99

author created summary a
Author-created Summary (A)
  • Original presenters (authors) were asked to produce summaries of the talks
    • Author marked the text transcript
    • Video summaries were generated manually by aligning the video with the marked portions

ACM Multimedia 99

summary
Summary
  • Automatic algorithms performed respectably
    • “That’s pretty cool for a computer. I thought someone had sat down and made them”
    • SPU was preferred over S and P
  • Will viewers get used to auto summary?

ACM Multimedia 99

future work
Future Work
  • Compare audio/video and text summaries
  • Interactive and intelligent video browser
  • Visit us at http://research.microsoft.com/coet

ACM Multimedia 99