Discovering Objects and their Location in Images

1. 0 1 0 2 ... Discovering Objects and their Location in Images Josef Sivic1, Bryan C. Russell2, Alexei A. Efros3, Andrew Zisserman1 and William T. Freeman2 1Oxford University 2MIT 3Carnegie Mellon University Introduction The topic discovery models Improving localization using doublets Form a new vocabulary from pairs oflocally co-occurring regions Goal:Discover visual object categories and their segmentation given a collection ofunlabelled images Probabilistic Latent Semantic Analysis (pLSA) [Hofmann’99] w … visual words d … documents (images) z … topics (‘objects’) Doublet formation Doublet example I Doublet examle II pLSA graphical model pLSA Model fitting: Find topic vectors P(w|z) common to all documents and mixture coefficients P(z|d)specific to each document.Fit model by maximizing likelihood of data using EM. All detected visual words Singlet segmentation Singlet segmentation Doublet segmentation Experiment II: MIT dataset Latent Dirichlet Allocation (LDA) [Blei et al.’03] Approach:1) Represent an image as a collection of visual words 2) Apply topic discovery models from statistical text analysis • 2873 images, learn 10 topics Treat multinomial weights over topics as random variables. Fit model using Gibbs sampling [Griffiths and Steyvers’04]. 4 of the 10 learned topics shown by the 5 most probable images for each topic Image representation Represent an image as a histogram of “visual words” LDA graphical model “Buildings” “Trees / Grass” Results Results shown only for pLSA. LDA had very similar performance. Experiment I: Caltech Dataset “Computers” “Bookshelves” Histogram of visual words Four object categories: faces, motorbikes, airplanes and cars rear (total of 3,190images) and 900 background images Example Images with multiple objects • Detect affine covariant regions • Represent each region by a SIFT descriptor • Build visual vocabulary by k-means clustering (K~1,000) • Assign each region to the nearest cluster centre Image Classification Mikolajczyk and Schmid’02, Schaffalitzky and Zisserman’02, Matas et al. ’02, Lowe’99, Sivic and Zisserman’03 Assign each image to a topic with the highest P(z|d) Learn K = (5,6,7) topics Background is better modelled by multiple topics Pre-learning background topics on a separate bg dataset improves results Performance on novel images is comparable with semi-supervised method of [Fergus et al.’03] Examples of visual words Experiment III: Application to image retrieval Confusion tables (K=5,6,7) learned topics Learn topic vectors on Caltech database Represent new query image in terms of learned topic vectors Segmentation Five samples from a ‘motorbike’ visual word Visual Polysemy. Single visual word occurring on different (but locally similar) parts on different object categories. For a given word wi in document dj examine posterior probability over topics. pLSA Retrieve images within Caltech database Raw word histograms Visual words colour coded according to the topic with the highest probability Query image Faces Five samples from an ‘airplane’visual word Visual Synonyms. Two different visual words representing similar part of an object (wheel of an motorbike). Motorbikes Precision – Recall plot Airplanes Retrieve images in movie Pretty Woman Cars Overview Background I Retrieved images using pLSA ‘object’ coefficients P(z|d) Background II Background III Pretty Woman (6,641 keyframes) Find visual words Form histograms Example face segmentation Retrieved images using visual word histograms Represent each keyframe using topic vectors learned on Caltech database Discover topics Example airplane segmentation Example motorbike segmentation