1 / 57

Bag-of-features for category recognition

Bag-of-features for category recognition. Cordelia Schmid. Bag-of-features for image classification. Origin: texture recognition Texture is characterized by the repetition of basic elements or textons.

sook
Download Presentation

Bag-of-features for category recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Bag-of-features for category recognition Cordelia Schmid

  2. Bag-of-features for image classification • Origin: texture recognition • Texture is characterized by the repetition of basic elements or textons Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

  3. Texture recognition histogram Universal texton dictionary Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003

  4. US Presidential Speeches Tag Cloudhttp://chir.ag/phernalia/preztags/ Bag-of-features for image classification • Origin: bag-of-words • Orderless document representation: frequencies of words from a dictionary • Classification to determine document categories

  5. Bag-of-features for image classification SVM Extract regions Compute descriptors Find clusters and frequencies Compute distance matrix Classification [Nowak,Jurie&Triggs,ECCV’06], [Zhang,Marszalek,Lazebnik&Schmid,IJCV’07]

  6. Bag-of-features for image classification SVM Extract regions Compute descriptors Find clusters and frequencies Compute distance matrix Classification Step 1 Step 3 Step 2 [Nowak,Jurie&Triggs,ECCV’06], [Zhang,Marszalek,Lazebnik&Schmid,IJCV’07]

  7. bikes books building cars people phones trees Bag-of-features for image classification • Excellent results in the presence of background clutter

  8. Examples for misclassified images Books- misclassified into faces, faces, buildings Buildings- misclassified into faces, trees, trees Cars- misclassified into buildings, phones, phones

  9. Bag-of-features for image classification SVM Extract regions Compute descriptors Find clusters and frequencies Compute distance matrix Classification Step 1 Step 3 Step 2 [Nowak,Jurie&Triggs,ECCV’06], [Zhang,Marszalek,Lazebnik&Schmid,IJCV’07]

  10. Harris-Laplace Laplacian Step 1: feature extraction • Scale-invariant image regions + SIFT • Selection of characteristic points

  11. Step 1: feature extraction • Scale-invariant image regions + SIFT • Robust description of the extracted image regions 3D histogram image patch gradient x   y • SIFT [Lowe’99] • 8 orientations of the gradient • 4x4 spatial grid

  12. Step 1: feature extraction • Scale-invariant image regions + SIFT • Selection of characteristic points • Robust description of these characteristic points • Affine invariant regions give “too” much invariance • Rotation invariance in many cases “too” much invariance

  13. Step 1: feature extraction • Scale-invariant image regions + SIFT • Selection of characteristic points • Robust description of these characteristic points • Affine invariant regions give “too” much invariance • Rotation invariance in many cases “too” much invariance • Dense descriptors • Improve results in the context of categories (for most categories) • Interest points do not necessarily capture “all” features

  14. Dense features - Multi-scale dense grid: extraction of small overlapping patches at multiple scales - Computation of the SIFT descriptor for each grid cells

  15. Step 1: feature extraction • Scale-invariant image regions + SIFT (see lecture 2) • Dense descriptors • Color-based descriptors • Shape-based descriptors

  16. Bag-of-features for image classification SVM Extract regions Compute descriptors Find clusters and frequencies Compute distance matrix Classification Step 1 Step 3 Step 2

  17. Step 2: Quantization

  18. Step 2:Quantization Clustering

  19. Step 2: Quantization Visual vocabulary Clustering

  20. Examples for visual words

  21. Step 2: Quantization • Cluster descriptors • K-mean • Gaussian mixture model • Assign each visual word to a cluster • Hard or soft assignment • Build frequency histogram

  22. K-means clustering • We want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers Algorithm: • Randomly initialize K cluster centers • Iterate until convergence: • Assign each data point to the nearest center • Recompute each cluster center as the mean of all points assigned to it

  23. K-means clustering • Local minimum, solution dependent on initialization • Initialization important, run several times • Select best solution, min cost

  24. From clustering to vector quantization • Clustering is a common method for learning a visual vocabulary or codebook • Unsupervised learning process • Each cluster center produced by k-means becomes a codevector • Provided the training set is sufficiently representative, the codebook will be “universal” • The codebook is used for quantizing features • A vector quantizer takes a feature vector and maps it to the index of the nearest codevector in a codebook • Codebook = visual vocabulary • Codevector = visual word

  25. Visual vocabularies: Issues • How to choose vocabulary size? • Too small: visual words not representative of all patches • Too large: quantization artifacts, overfitting • Computational efficiency • Vocabulary trees (Nister & Stewenius, 2006) • Soft quantization: Gaussian mixture instead of k-means

  26. Hard or soft assignment • K-means  hard assignment • Assign to the closest cluster center • Count number of descriptors assigned to a center • Gaussian mixture model  soft assignment • Estimate distance to all centers • Sum over number of descriptors • Frequency histogram

  27. ….. Image representation frequency codewords

  28. Bag-of-features for image classification SVM Extract regions Compute descriptors Find clusters and frequencies Compute distance matrix Classification Step 1 Step 3 Step 2

  29. Step 3: Classification • Learn a decision rule (classifier) assigning bag-of-features representations of images to different classes Decisionboundary Zebra Non-zebra

  30. Classification • Assign input vector to one of two or more classes • Any decision rule divides input space into decision regions separated by decision boundaries

  31. Nearest Neighbor Classifier • Assign label of nearest training data point to each test data point from Duda et al. Voronoi partitioning of feature space for 2-category 2-D and 3-D data Source: D. Lowe

  32. K-Nearest Neighbors • For a new point, find the k closest points from training data • Labels of the k points “vote” to classify • Works well provided there is lots of data and the distance function is good k = 5 Source: D. Lowe

  33. Linear classifiers • Find linear function (hyperplane) to separate positive and negative examples Which hyperplaneis best? SVM (Support vector machine)

  34. Functions for comparing histograms • L1 distance • χ2 distance • Quadratic distance (cross-bin)

  35. Kernels for bags of features • Histogram intersection kernel: • Generalized Gaussian kernel: • D can be Euclidean distance, χ2distance,Earth Mover’s Distance, etc.

  36. Chi-square kernel • Multi-channel chi-square kernel • Channel c is a combination of detector, descriptor • is the chi-square distance between histograms • is the mean value of the distances between all training sample • Extension: learning of the weights, for example with MKL

  37. Pyramid match kernel • Weighted sum of histogram intersections at multiple resolutions (linear in the number of features instead of cubic) optimal partial matching between sets of features

  38. Pyramid Match Histogram intersection

  39. matches at this level matches at previous level Difference in histogram intersections across levels counts number ofnew pairs matched Pyramid Match Histogram intersection

  40. histogram pyramids number of newly matched pairs at level i measure of difficulty of a match at level i Pyramid match kernel • Weights inversely proportional to bin size • Normalize kernel values to avoid favoring large sets

  41. Example pyramid match Level 0

  42. Example pyramid match Level 1

  43. Example pyramid match Level 2

  44. Example pyramid match pyramid match optimal match

  45. Summary: Pyramid match kernel optimal partial matching between sets of features difficulty of a match at level i number of new matches at level i

  46. Spatial pyramid matching • Add spatial information to the bag-of-features • Perform matching in 2D image space [Lazebnik, Schmid & Ponce, CVPR 2006]

  47. Related work • Similar approaches: • Subblock description [Szummer & Picard, 1997] • SIFT [Lowe, 1999] • GIST [Torralba et al., 2003] SIFT Gist Szummer & Picard (1997) Lowe (1999, 2004) Torralba et al. (2003)

  48. level 0 Spatial pyramid representation Locally orderless representation at several levels of spatial resolution

  49. level 1 Spatial pyramid representation Locally orderless representation at several levels of spatial resolution level 0

  50. level 2 Spatial pyramid representation • Locally orderless representation at several levels of spatial resolution level 1 level 0

More Related