1 / 49

Efficient Image Search and Retrieval using Compact Binary Codes

Efficient Image Search and Retrieval using Compact Binary Codes. Rob Fergus (NYU) Jon Barron (NYU/UC Berkeley) Antonio Torralba (MIT) Yair Weiss (Hebrew U.). CVPR 2008. Large scale image search. Internet contains many billions of images. How can we search them, based on visual content?.

lisandra-graham
Download Presentation

Efficient Image Search and Retrieval using Compact Binary Codes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Efficient Image Search and Retrieval using Compact Binary Codes Rob Fergus (NYU) Jon Barron (NYU/UC Berkeley) Antonio Torralba (MIT) Yair Weiss (Hebrew U.) CVPR 2008

  2. Large scale image search Internet contains many billions of images How can we search them, based on visual content? • The Challenge: • Need way of measuring similarity between images • Needs to scale to Internet

  3. Current Image Search Engines Essentially text-based

  4. Existing approaches to Content-Based Image Retrieval • Focus of scaling rather than understanding image • Variety of simple/hand-designed cues: • Color and/or Texture histograms, Shape, PCA, etc. • Various distance metrics • Earth Movers Distance (Rubner et al. ‘98) • Most recognition approaches slow (~1sec/image)

  5. Our Approach • Learn the metric from training data • Use compact binary codes for speed DO BOTH TOGETHER

  6. Large scale image/video search • Representation must fit in memory (disk too slow) • Facebook has ~10 billion images (1010) • PC has ~10 Gbytes of memory (1011 bits)  Budget of 101 bits/image • YouTube has ~ a trillion video frames (1012) • Big cluster of PCs has ~10 Tbytes (1014 bits) •  Budget of 102 bits/frame

  7. Some file sizes • Typical YouTube clip (compressed) is ~ 108 bits • 1 Megapixel JPEG image is ~ 107 bits • 32x32 color image is ~ 104 bits • - Smallest useful image size 101 - 102 bits is not much  Need a really compact image representation

  8. Binary codes for images • Want images with similar contentto have similar binary codes • Use Hamming distance between codes • Number of bit flips • E.g.: • Semantic Hashing [Salakhutdinov & Hinton, 2007] • Text documents Ham_Dist(10001010,10001110)=1 Ham_Dist(10001010,11101110)=3

  9. Semantic Hashing [Salakhutdinov & Hinton, 2007] for text documents Query Image Semantic HashFunction Address Space Binary code Images in database Query address Semantically similar images Quite differentto a (conventional)randomizing hash

  10. Semantic Hashing • Each image code is a memory address • Find neighbors by exploring Hamming ball around query address Address Space • Lookup time is independentof # of data points • Depends on radius of ball & length of code: Images in database Query address • Code length Choose • Radius

  11. Code requirements • Similar images  Similar Codes • Very compact (<102 bits/image) • Fast to compute • Does NOT have to reconstruct image Three approaches: • Locality Sensitive Hashing (LSH) • Boosting • Restricted Boltzmann Machines (RBM’s)

  12. Input Image representation: Gist vectors • Pixels not a convenient representation • Use Gist descriptor instead (Oliva & Torralba, 2001) • 512 dimensions/image (real-valued  16,384 bits) • L2 distance btw. Gist vectors not bad substitute for human perceptual distance NO COLOR INFORMATION Oliva & Torralba, IJCV 2001

  13. 1. Locality Sensitive Hashing • Gionis, A. & Indyk, P. & Motwani, R. (1999) • Take random projections of data • Quantize each projection with few bits 101 0 Gist descriptor 1 0 No learning involved 1 1 0

  14. 2. Boosting • Modified form of BoostSSC [Shaknarovich, Viola & Darrell, 2003] • Positive examples are pairs of similar images • Negative examples are pairs of unrelated images 0 Learn threshold & dimension for each bit (weak classifier) 0 1 1 0 1

  15. 3. Restricted Boltzmann Machine (RBM) • Type of Deep Belief Network • Hinton & Salakhutdinov, Science 2006 Hidden units Units are binary & stochastic SingleRBMlayer Symmetric weights W Visible units • Attempts to reconstruct input at visible layer from activation of hidden layer

  16. 3. Restricted Boltzmann Machine (RBM) • p(Activation of hidden unit ) = Sigmoid ) • Symmetric situation for visible units SingleRBMlayer • Learn weights (& biases) in unsupervised manner (via sampling-based approach)

  17. Multi-Layer RBM: non-linear dimensionality reduction Output binary code (N dimensions) N Layer 3 w3 256 256 Layer 2 w2 512 512 Layer 1 w1 512 Linear units at first layer Input Gist vector (512 dimensions)

  18. Training RBM models 1st Phase: Pre-training Unsupervised Can use unlabeled data (unlimited quantity) Learn parameters greedily per layer Gets them to right ballpark 2nd Phase: Fine-tuning Supervised Requires labeled data (limited quantity) Back propagate gradients of chosen error function Moves parameters to local minimum

  19. Greedy pre-training (Unsupervised) 512 Layer 1 w1 512 Input Gist vector (512 real dimensions)

  20. Greedy pre-training (Unsupervised) 256 Layer 2 w2 512 Activations of hidden units from layer 1 (512 binary dimensions)

  21. Greedy pre-training (Unsupervised) N Layer 3 w3 256 Activations of hidden units from layer 2 (256 binary dimensions)

  22. Fine-tuning: back-propagation of Neighborhood Components Analysis objective Output binary code (N dimensions) N Layer 3 w3 w3 + ∆w3 256 256 Layer 2 w2 w2 + ∆ w2 512 512 Layer 1 w1 + ∆ w1 w1 512 Input Gist vector (512 real dimensions)

  23. Neighborhood Components Analysis • Goldberger, Roweis, Salakhutdinov & Hinton, NIPS 2004 • Tries to preserve neighborhood structure of input space • Assumes this structure is given (will explain later) Toy example with 2 classes & N=2 units at top of network: Points in output space (coordinate is activation probability of unit)

  24. Neighborhood Components Analysis • Adjust network parameters (weights and biases) to move: • Points of SAME class closer • Points of DIFFERENT class away

  25. Neighborhood Components Analysis • Adjust network parameters (weights and biases) to move: • Points of SAME class closer • Points of DIFFERENT class away Points close in input space (Gist) will be close in output code space

  26. Deliberately add noise Simple Binarization Strategy Set threshold - e.g. use median 0 1 0 1

  27. Overall Query Scheme <10μs Binary code Image 1 RBM Retrieved images <1ms Semantic Hash Query Image Gist descriptor Compute Gist ~1ms (in Matlab)

  28. Retrieval Experiments

  29. Test set 1: LabelMe • 22,000 images (20,000 train | 2,000 test) • Ground truth segmentations for all • Can define ground truth distance btw. images using these segmentations

  30. Defining ground truth • Boosting and NCA back-propagation require ground truth distance between images • Define this using labeled images from LabelMe

  31. Defining ground truth • Pyramid Match (Lazebnik et al. 2006, Grauman & Darrell 2005)

  32. Defining ground truth • Pyramid Match (Lazebnik et al. 2006, Grauman & Darrell 2005) Tree Tree Tree Building Building Building Car Car Car Road Road Road Sky Sky Sky Tree Tree Tree Building Building Building Car Car Car Car Car Car Car Car Car Road Road Road Varying spatial resolution to capture approximate spatial correspondance

  33. Examples of LabelMe retrieval • 12 closest neighbors under different distance metrics

  34. LabelMe Retrieval % of 50 true neighbors in retrieval set 0 2,000 10,000 20,0000 Size of retrieval set

  35. LabelMe Retrieval % of 50 true neighbors in retrieval set % of 50 true neighbors in first 500 retrieved 0 2,000 10,000 20,0000 Number of bits Size of retrieval set

  36. Test set 2: Web images • 12.9 million images • Collected from Internet • No labels, so use Euclidean distance between Gist vectors as ground truth distance

  37. Web images retrieval % of 50 true neighbors in retrieval set Size of retrieval set

  38. Web images retrieval % of 50 true neighbors in retrieval set % of 50 true neighbors in retrieval set Size of retrieval set Size of retrieval set

  39. Examples of Web retrieval • 12 neighbors using different distance metrics

  40. Retrieval Timings

  41. Scaling it up • Google very interested in it • Jon Barron summer internship at Google NYC • NCA has N2 cost • Use DrLIM (Hadsell, Chopra, LeCun 2006) • Train on Google proprietary labels

  42. Further Directions • Spectral Hashing • Brute Force Object Recognition

  43. Spectral Hashing (NIPS ’08) • Assume points are embedded in Euclidean space • How to binarize so Hamming distance approximates Euclidean distance? • Under certain (reasonable) assumptions, analytic form exists • No learning, super-simple • Come to Machine Learning seminaron Dec 2nd ……

  44. 2-D Toy example: 3 bits 7 bits 15 bits Distance from query point Red – 0 bits Green – 1 bit Black – >2 bits Blue – 2 bits Query Point

  45. 2-D Toy Example Comparison

  46. Further Directions • Spectral Hashing • Brute Force Object Recognition

  47. 80 Million Tiny Images (PAMI ‘08) # images Implemented by 105 106 108

  48. LabelMe Recognition examples

  49. Summary • Explored various approaches to learning binary codes for hashing-based retrieval • Very quick with performance comparable to complex descriptors • Remaining issues: • How to learn metric (so that it scales) • How to produce binary codes • How to use for recognition

More Related