Challenges in Multimedia Information Retrieval & Filtering

1. Challengesin Multimedia Information Retrieval & Filtering 薛向阳 xyxue@fudan.edu.cn 复旦大学计算机科学与工程系 上海市智能信息处理重点实验室

2. Outline • Potential Applications, Query Examples & Achievements • Basic Concepts & Architectures • Key Techniques & Problems 薛向阳 － 复旦大学计算机科学系

3. Many Potential Applications • Broadcast media selection (e.g. radio channel, TV channel) • Cultural services (e.g. history museums, art galleries) • Digital libraries (e.g. image catalogue, musical dictionary, bio-medical imaging catalogues, film, video and radio archives) • Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face) • Multimedia directory services (e.g. yellow pages, Tourist information) • …… 薛向阳 － 复旦大学计算机科学系

4. Video Query Examples（TREC） • a specific person • I want all the information you have on Ronald Reagan • a specific thing • I'm interested in any material on Hoover Dam.I'm looking for a picture of the OGO satellite 薛向阳 － 复旦大学计算机科学系

5. Informedia – CMU • Establishment of large video libraries as a searchable information resource • Full content information retrieval in both spoken language and video/image domains • Integration of speech, image and natural language understanding for library creation and exploration • Fully automated transcriptions generated entirely speech recognition or with closed captions • Information summaries at varying detail, both visually and textually 薛向阳 － 复旦大学计算机科学系

6. CueVideo – IBM • Developing fully automatic means for indexing, hyper-linking and preparation of media material for effective searching and browsing by users • Combines several automated indexing，searching and browsing tools • Video analysis and summarization • Use of speech recognition for spoken media retrieval 薛向阳 － 复旦大学计算机科学系

7. Outline • Potential Applications, Query Examples & Achievements • Basic Concepts & Architectures • Key Techniques & Problems 薛向阳 － 复旦大学计算机科学系

8. Document Corpus Query String IR System 1. Doc1 2. Doc2 3. Doc3 . . Ranked Documents An Instance of IR System 薛向阳 － 复旦大学计算机科学系

9. Information Retrieval • Information Retrieval (IR) Deals with: • Representation (or Modeling) • Storage • Organization • Access • of / to Information Items 薛向阳 － 复旦大学计算机科学系

10. Architecture：IR offline Multi - Modal User Interface Representation Modeling ， Relevance feedback Description (MPEG -7/XML) Multimedia ng Query Processi Database Organizing: Index Structure Searching Ranking 薛向阳 － 复旦大学计算机科学系

11. Information Filtering • Generally, the goal of an Information Filtering (IF) system is to sort through large volumes of dynamically generated information and present to the user those which are likely to satisfy his or her information requirement 薛向阳 － 复旦大学计算机科学系

12. representation Architecture：IF 薛向阳 － 复旦大学计算机科学系

13. Applications using MPEG7 薛向阳 － 复旦大学计算机科学系

14. Comparison：IR & IF • Information Retrieval • User Information Needs or Query –Varying • Database or Collection –Static • Information Filtering • User Information Needs or Profile –Static • Incoming Data –Varying • Common to both • how to represent information • how to select relevant information 薛向阳 － 复旦大学计算机科学系

15. Outline • Practical Applications, Query Examples & Achievements • Basic Concepts & Architectures • Key Techniques & Problems 薛向阳 － 复旦大学计算机科学系

16. Shot • Key-frame • Speech • Ocr • Scene • Face • Motion • … User DVB-S DVB-C MPEG2 TS XML Template Database: >2TB Search Engine Filtering Digital TV Program Filtering & Searching System 薛向阳 － 复旦大学计算机科学系

17. Representation –extract low level features • Text Features • stop word elimination, stemming, index term selection, thesauri, word cut… • Image and Video Features • color, texture, shape, motion, … • Audio (Speech,Music) Features • zero-crossing ratio, short time energy • Spectral, Spectral Flux, Spectral Centroid, LPC, MFCC • Pitch,Rhythm,Timbre,… • Requirements － Good Representation, Fast, Automatic, Robust 薛向阳 － 复旦大学计算机科学系

18. Representation –get high level features • Structured Video Analysis • Video – Scene – Shot – Key frame • Summaries at varying detail, both visually and textually • Audio & Visual Object Recognition • Face,Character,Car,… • Word Spotting,Speech Recognition,Speaker,… • Problem - Low Precision, Infant, Inevitable Incompleteness in the Representation,… 薛向阳 － 复旦大学计算机科学系

19. 薛向阳 － 复旦大学计算机科学系

20. Retrieval Model • Boolean Model • Vector Model • Probabilistic Model • Fuzzy Set Model • Neural Network • …… 薛向阳 － 复旦大学计算机科学系

21. Storage & Organization • Storage • Standardized Descriptors - MPEG-7 • Management of XML Documents • Index Structures – For Fast Query • Inverted File for Text • Index Structure for XML Documents • Index Structures for High Dimensional Vector (Visual Features) - Dimensionality Curse 薛向阳 － 复旦大学计算机科学系

22. D r （0,0） r D （0,0） Curse of Dimensionality An Intuitive Explanation Assume n-dimensional points distributed in super-cubic. Selectivity can be computed： When n increasing, P(n) will go down to zero exponentially. In order to find relevant points, searching window should be enlarged! 薛向阳 － 复旦大学计算机科学系

23. Multi-Modal Interface － 1 • Input Information Needs • Key Word,… • Example Image, Example Face, Example Video Clip,… • Speech, Humming,… • Relevance feedback • How to submit user’s query easily and friendly to IR system? • How can IR system understand user’s query intention? • People are unable to specify that which they don't know • There is inevitable uncertainty in the representation or understanding of information problems 薛向阳 － 复旦大学计算机科学系

24. Multi-Modal Interface － 2 • Output Query Results • Enable user to browse full content in hierarchy or web • Visualization is important for presentation 薛向阳 － 复旦大学计算机科学系

25. How to Compute Relevance? • Relevance is a dynamic and idiosyncratic relationship between person and information object • Information objects mean many different things to different people (or the same person at different times) • There is inherent uncertainty in the relevance relationship 薛向阳 － 复旦大学计算机科学系

26. Comparison：IR & DR 薛向阳 － 复旦大学计算机科学系

27. Conclusion • Many types of data without strict structure in huge multimedia database • Almost all algorithms of intelligent information processing and recognition (audio & visual) are necessary for better representation • Seeking good retrieval model may be key to reduce gap between person and computer • Uncertain & chaotic task – unable to be formulated 薛向阳 － 复旦大学计算机科学系

28. Q/A？ Thank You！ 薛向阳 － 复旦大学计算机科学系