alexander gelbukh www gelbukh com n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Alexander Gelbukh Gelbukh PowerPoint Presentation
Download Presentation
Alexander Gelbukh Gelbukh

Loading in 2 Seconds...

play fullscreen
1 / 29
gregory-anthony

Alexander Gelbukh Gelbukh - PowerPoint PPT Presentation

94 Views
Download Presentation
Alexander Gelbukh Gelbukh
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Special Topics in Computer ScienceAdvanced Topics in Information RetrievalLecture 5 (book chapter 11): Multimedia IR:Models and Languages Alexander Gelbukh www.Gelbukh.com

  2. Previous Chapter: Conclusions • Inverted files seem to be the best option • Other structures are good for specific cases • Genetic databases • Sequential searching is an integral part of manyindexing-based search techniques • Many methods to improve sequential searching • Compression can be integrated with search

  3. Previous Chapter: Research topics • Perhaps, new details in integration of compression and search • “Linguistic” indexing: allowing linguistic variations • Search in plural or only singular • Search with or without synonyms

  4. Motivation • Applications: • office, • CAD, • medical, • Internet • Example: • Artists sings a melody and sees all the songs with similar melody

  5. What’s different • Different from text IR: • Structure of data is more complex. Efficiency is an issue • Using of metadata • Characteristics of multimedia data • Operations to be performed • Aspects: • Data modeling: Extract and maintain the features of objects • Data retrieval: based not only on description but on content

  6. Retrieval process • Query specification • fuzzy predicates: similar to • content predicates: images containing an apple • data type predicates: video, ... • Query processing and optimization • Parsed, compiled, optimized for order of execution • Problem: many data types, different processing for each • Answer • Relevance: similarity to query • Iteration • Bad quality, so need to refine

  7. Modeling

  8. Data modeling • To model is to simplify, in order to make manageable. “We will represent an image as...” • From the user’s point of view • From the system’s point of view (technically) • A problem: very large storage size. Modeling needed • Objects are represented as feature vectors • Images / Video: shape. House, car, ... • Sound: style. Music: Merry, sad, ... • Features are defined directly or by comparison • Degree of certainty is stored

  9. Multimedia support in commercial DBMSs (1999) • Variable length data. • Non-standard • Different and usually very limited sets of operations • SQL3: • provides user-extensible data types • Object-oriented • Implemented partially in many systems • Example: data blades of Informix • Content-based functions on text and images • E.g.: date = 1997 AND contains (car)

  10. Spatial data types • Informix: 2D, 3D data blades • Boxes, vectors, ... • Operations: intersect, contains, center, ... • Text: containWords, .... • Supports query images by content

  11. Example: MULTOS • Multimedia document server • Documents are described by: • logical structure: title, into, chapter, ... • layout structure: pages, frames, ... • conceptual structure: allows content-based queries • Docs similar in conceptual structures are grouped into conceptual types • Example: Generic_Letter

  12. Example of conceptual structure...

  13. ...continued

  14. Image data in MULTOS • Analysis • low level: detect objects and positions • high level: image interpretation • Result of analysis: • description of objects found and their classes • certainty values • Indices are used for fast access to this info • Object index. Includes pointers to objects and certainty values • Cluster index, with fuzzy clusters of similar images

  15. Internet • How Google does it? • No image processing. Textual context! • File names, nearby words • Distance from image to words • “give me images with flower in the file name or near the image”

  16. Languages

  17. Query languages • As a query, either a description of the object or an example object is submitted • “show me images similar to this one” • in what respects similar?! • Exact match is inadequate. Additional means are needed • Content is not a single feature

  18. What defines query language • Interface. How to enter the query • Types of conditions to specify • Handling of uncertainty, proximity, weights

  19. Interface • Browsing and navigation • Search: description or query by example • Query by example: • specify what features are important. Give me all houses with similar shape but different colors • Libraries of examples can be provided

  20. Conditions... • Attribute predicates • structured content – the predefined types extracted beforehand • Exact match. E.g.: size, type (video, audio, ...) • Structural predicates • structure: title, sections, ... • metadata are used. Find objects containing an image and a video clip • Semantic predicates • unrestricted content. • Find all red houses: red = ?, house = ? Fuzzy

  21. ... conditions • Predicates • Spatial: contain, intersect, is contained in, is adjacent to ... • Temporal: Find audio where first politics and then economy is discussed • Spatial and temporal predicates can be combined: Find clips where the logo disappears and then a graph appears at the same place • A predicate can be applied to a part of document • As path expressions in OO databases

  22. Uncertainty, proximity, weights • Similarity function • The user can assign importance weights to individual predicates in a complex query • This gives ranking, as in text IR • The same models can be used, e.g., probabilistic model

  23. Examples of query languages: SQL3 • Functions and stored procedures: user-defined data manipulation • Active database support: database reacts on the events, not only commands. This enforces integrity constraints • Good news: rather standard • Bad news: no ranking supported! • Effort to integrate SQL3 with IR techniques.SQL MM Full Text and other similar languages

  24. ... examples: MULTOS • One of design goals: easy navigation • Paths are supported • Identification of components by type, not by position • All images in the document, not the image in 3rd chapter • Types of predicates: • on data attributes, on textual components, on images (image type, objects contained, ...) • Example:

  25. MULTOS example

  26. Another example of MULTOS

  27. Research topics • How similarity function can be defined? • What features of images (video, sound) there are? • How to better specify the importance of individualfeatures? (Give me similar houses: similar = size?color? strructure? Architectural style?) • How to determine the objects in an image? • Integration with DBMSs and SQL for fast access and rich semantics • Integration with XML • Ranking: by similarity, taking into account history, profile

  28. Conclusions • Basically, images are handled as text described them • Namely, feature vectors (or feature hierarchies) • Context can be used when available to determine features • Also, queries by example are common • From the point of view of DBMS, integration with IRand multimedia-specific techniques is needed • Object-oriented technology is adequate

  29. Thank you! Till ??, 6 pm