1 / 13

Crossing Media for Video Search: enabling usability beyond traditional broadcast & TV

Crossing Media for Video Search: enabling usability beyond traditional broadcast & TV. Katerina Pastra and Stelios Piperidis. Language Technology Applications, Institute for Language and Speech Processing, Athens, Greece. The “Pervasive Digital Video” Era. .

donnan
Download Presentation

Crossing Media for Video Search: enabling usability beyond traditional broadcast & TV

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Crossing Media for Video Search: enabling usability beyond traditional broadcast & TV Katerina Pastra and Stelios Piperidis Language Technology Applications, Institute for Language and Speech Processing, Athens, Greece

  2. The “Pervasive Digital Video” Era  • TV sets extended with “intelligent” DVRs, set-top boxes with PC-like functionalities, linked with PCs that display streamed video and allow interaction through gaming consoles • Video viewing transferred beyond the TV set, to mobiles & i-pods allowing on-the-move viewing • Video broadcast carried through broadband using IP • Video content (professional and/or consumer-generated)exchanged through file swapping and headline syndication technology

  3. A New Era in Video Search ?  From the digital video libraries context to the new pervasive digital video reality: The scope of video search (indexing & retrieval) technologies is broadened and their role is reinforced Does pervasive digital video affect the “search” in video search technologies? (imposes new challenges) Does video search affect the “pervasiveness” of digital video? (affects usability of available video and corresponding new technologies)

  4. Overview • Video search: the market perspective - market players and video search developers - video search in commercial prototypes • Video search research prototypes - lessons from the digital video library scenarios • New technological challenges • Dealing with new challenges - Suggestions from the REVEAL THIS project - Using cross-media decision mechanisms

  5. IPTV software developers Content owners ISP, computer networking & phone companies Video search software developers TV service providers & file-swapping networks Electronics Manufacturers Transitive Dependencies Dependency Trends Web content Aggregators, content service providers, content repackaging companies Video Search & the Market Players

  6. Not always present Find the right keyword problem! Other languages? Video search mechanisms in the market Characteristics : • Use of owner/broadcaster createdmetadata • Text-based searchon closed captions or ASR or speech stream • Processing ofEnglishfiles mostly • Keyword query(restricted semantic expansion) • Retrieval unitis either whole video or short segment where keyword appears (+ few seconds before and after) Not robust Quest for coherence Are such mechanisms efficient?

  7. Video Search in Research • The digital video library access scenarios prevailed in research projects up until the ’00s (cf. Informedia, TRECVid etc.) • Video indexing & retrieval prototypes explored a variety of unimodal and multimodal mechanisms that go beyond commercially offered video search Lessons learned(Hauptmann and Christel 2004): • Fusion of medium-specific retrieval results boosts video retrieval performance slightly (vs. e.g. text-based only retrieval) • Fusion based on linear weights ~ query-type helpful • Text query/ASR enhancement, relevance feedback and feature-concept associations, all helpful MM approaches slightly better. Necessary?

  8. Video search challenges in the new context • From advanced computer users to laymen - type & quality of query - expectations & requirements on retrieval accuracy - length of retrieval unit - domain and language of data • From structured data collections to pervasive video - genre, domain, language, source/structure variation - consumer-generated / noisy / low quality and prof. - broadcast metadata, closed captions etc. availability • From static to dynamic search • - VoD &real-time broadcast data • - re-active &pro-active, personalised search • (push and pull) • The ideal search mechanism?

  9. Video search prototypes in the ’00s • Multimedia approaches for video search suggested by research projects with application scenarios related to the new pervasive digital video context, e.g. leisure and entertainment in the digital home, and/or for the mobile user (e.g UP-TV, BUSMAN, AceMedia) • Image features – language concepts association for video search suggested (Multimedia integration) • REVEAL THIS goes a step further in suggesting the use of cross-media decision mechanisms FP6 funded project (Nov. 2004-April 2007) http://www.reveal-this.org

  10. Media Server (Content Aggregator) REVEAL THIS Use Scenario WEB TV Radio TV, Radio, Web data • A system that offers both types of service : • Multimediaand Cross lingual Information Retrieval (pull) • Multimedia and Cross lingual informationFiltering (push) Media Archive Reveal-THIS technology Search archive Delivery Mobile phone, and Web interfaces EN-EL, European Parliament plenary sessions & press-conferences, national news, travel documentaries & info User profile Web User Mobile Local Archive

  11. Web text radio TV Media Server: Storage Personalisation Browse Query Retrieve Push Notifications REVEAL This System Architecture Media Manager video text audio IAC – keyframe extraction, Image Analysis & image categorization FDIC - face Detection & identification TPC - text processing SPC–speech processing text keyframes Automatically extracted metadata: TPC: named entities, terms, facts Text Categories SPC: speaker turns, speaker names, text IAC: shotcuts, keyframes, image features, image categories FDIC: face regions, names Text categorisation Translation Story Boundary Detection cross-media stories Cross-media Indexing, Cross-media Categorization Multimedia Summarization

  12. equivalence The notion of Cross-Media Decision Mechanisms Mechanisms that decide on the relation that holds between medium specific pieces of information: • across documents (Boll et al. 1999) • within documents (Pastra 2006) The mechanisms decided whether medium-specific pieces of information within the same Multimedia Document are: • associated (multimedia integration) • complementary • semantically compatible/incompatible complementarity independence

  13. Conclusions • The scope of video search technology is broadened & new technological challenges are imposed • The market players consider video search technology indispensable • Commercial video search limited; research in the digital library access context goes beyond such limitations & points to slight benefits in using multimedia fusion techniques • Research with new application scenarios (iTV etc.) emphasizes the necessity of such mechanisms & introduces the notion of x-media decision mechanisms Efficient video search is indispensable for usability beyond traditional broadcast and TV; x-media decision mechanisms may hold the key for achieving it

More Related