1 / 16

Multimodal Analysis

Video Summarization. Video Highlights Extraction. Video Browsing. Video Retrieval. Video Representation. Multimodal Analysis. content access. data flow. Video Summarization. Highlights based Summarization. Video Browsing. Table of Contents based Summarization. Top-down.

bobbier
Download Presentation

Multimodal Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Video Summarization Video Highlights Extraction Video Browsing Video Retrieval Video Representation Multimodal Analysis

  2. content access data flow Video Summarization Highlights based Summarization Video Browsing Table of Contents based Summarization Top-down Video Representation Multimodal Analysis Video Content Bottom-up Video Retrieval

  3. Highlight Groups Grouping Highlight Candidates Audio-visual markers association A highlight Audio-Visual Markers Visual Marker Audio Marker Key audio-visual object detection Play / Break Play Break Feature extraction & segmentation Video with Audio Track

  4. Browsing Highlights based Summarization ToC based Summarization Retrieval ToC Highlights Index Highlight Groups Scenes Visual Groups Semantic Highlight Candidates Shots Audio Audio-Visual Markers Key frames Camera motion Play/Break

  5. Video Audio Audio Markers Detection Visual Markers Detection Applause Cheers Soccer Goal Post Baseball Catcher Excited Speech Golfer Bending to Hit Which sport is it? A-V Marker Negotiation Highlight Candidates Finer Resolution Highlights Golf Swings Golf Putts Non-highlights Strikes, Balls Ball-hits Non-highlights Corner Kicks Penalty Kicks Non-highlights

  6. Soccer video markers Baseball video marker Golf video markers

  7. Time Domain Frequency Domain Audio Class Recognition (GMM) Classify the input audio into one of 5 Audio Markers using GMM Input Audio Applause Feature Extraction Comparing Likelihoods Cheering Music MFCC Coefficients. Speech Excited Speech Training Audio Clips Audio Class Feature Extraction Training the GMM Classifiers using MFCC features and BIC.

  8. Time Domain Frequency Domain Audio Class Recognition (GMM) Classify the input audio into one of 2 Audio Markers using GMM Input Audio Excited Speech Feature Extraction Comparing Likelihoods Other (Applause,Cheering, Music, Speech) MFCC Coefficients. Task = Sports Highlights Training Audio Clips Audio Class Feature Extraction Training the GMM Classifiers using MFCC features and CV

  9. Time Domain Frequency Domain Audio Class Recognition (GMM) Classify the input audio into one of 5 Audio Markers using GMM Input Audio Applause Feature Extraction Comparing Likelihoods Cheering Music MFCC Coefficients. Speech Excited Speech Training Audio Clips Audio Class Feature Extraction Training the GMM Classifiers using MFCC features and BIC.

  10. Importance Level MDCTs Input Audio Class Label Importance Level Calculation Feature Extraction Audio Classifier Task

  11. Generic Audio Classification Applause Compare Likelihoods Cheering Class Label Music MDCTs Speech Excited Speech Training Audio Clips Feature Extraction Training Data for GMMs

  12. Importance Level MDCTs Input Audio Class Label Importance Level Calculation Feature Extraction Audio Classifier Task

  13. Task Specific Audio Classification Compare Likelihoods Excited Speech Other (Applause,Cheering, Music, Speech) Class Label MDCTs Training Audio Clips Feature Extraction Task = Sports Highlights

  14. Filter 1 Filter 2 Filter 3 Goal post view 1 Filter 3 Filter 1 Filter 2 Goal post view 2 Soccer Models Interpretation Filter 1 Filter 2 Catcher Model Interpretation Filter 2 Filter 1 Filter 2 Filter 1 Golfer view 1 Golfer view 2 Golf Models Interpretation

  15. Play Break Video clip A-V markers Index excited speech catcher bat swing cheer

More Related