1 / 71

Image and video descriptors

Image and video descriptors. Advanced Topics in Computer Vision Spring 2010 Weizmann Institute of Science. Oded Shahar and Gil Levi . Outline. Overview Image Descriptors Histograms of Oriented Gradients Descriptors Shape Descriptors Color Descriptors Video Descriptors.

morag
Download Presentation

Image and video descriptors

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image and video descriptors Advanced Topics in Computer Vision Spring 2010 Weizmann Institute of Science Oded Shahar and Gil Levi

  2. Outline • Overview • Image Descriptors • Histograms of Oriented Gradients Descriptors • Shape Descriptors • Color Descriptors • Video Descriptors

  3. Overview - Motivation • The problem we are trying to solve is image similarity. • Given two images (or image regions) – are they similar or not ?

  4. Overview - Motivation • Solution: • Image Descriptors. • An image descriptors “describes” a region in an image. • To compare two such regions we will compare their descriptors.

  5. Overview - Descriptor To compare two images, we will compare their descriptors Descriptor Function Similar? Similar?

  6. Overview - Similarity • But what is similar to you ? • Depends on the application !

  7. Overview • Image (or region) similarity is used in many CV applications, for example: • Object recognition • Scene classification • Image registration • Image retrieval • Robot localization • Template matching • Building panorama • And many more…

  8. Overview • Example – 3D reconstruction from stereo images. 12 30 75 15 3 80 23 102 110 80 39 103 19 200 208 195 150 196 • Comparing the pixels as they are, will not work!

  9. Overview • Descriptors provide a means for comparing images or image regions. • Descriptors allow certain differences between the regions – scale, rotation, illumination changes, noise, shape, etc.

  10. Overview - Motivation Descriptor Function Similar ? Similar ? • Again, can’t take the pixels alone…

  11. Overview Comonly used as follows • Extract features from the image as small regions 2. Describe each region using a feature descriptor 3. Use the descriptors in application (comparison, training a classifier, etc.)

  12. Overview • Main problems • Features Detection – • Where to compute the descriptors? will cover briefly • Feature Description (Descriptors) • How to compute descriptors? today • Feature Comparison • How to compare two descriptors? will cover briefly

  13. Overview - Features Detection Detection Methods Where to compute the descriptors? • Grid • Key-Points • Global

  14. Key-Points as Detector Output Overview - Features Detection • Can be • Points • Regions (of different orientation, scale and affine trans.) • Squares • Ellipses • Circles • Etc..

  15. Overview – Descriptor Comparison Given two region description, how to compare them? • Usually descriptor come with it’s own distance function • Many descriptors use L2 distance

  16. Overview – Descriptor Invariance • Different descriptors measure different similarity • Descriptors can have invariance for visual effects • Illumination • Noise • Colors • Texture Similar ? • Different applications require different invariance therefore require different descriptors

  17. Outline • Overview • Image Descriptors • Histograms of Oriented Gradients Descriptors • Shape Descriptors • Color Descriptors • Video Descriptors

  18. Descriptor To compare two images, we will compare their descriptors Descriptor Function Similar? Similar?

  19. Descriptors Types of descriptors • Intensity based • Histogram • Gradient based • Color Based • Frequency • Shape • Combination of the above

  20. Descriptors Why not use patches? • Very large representation. • Not invariant to small deformations in the descriptor location. • Not invariant to changes in illumination.

  21. Descriptors Intensity Histogram 0 255 - Not invariant to light intensity change - Does not capture geometric information

  22. Descriptors Histogram of image gradients • Normalize for light intensity invariance • Does not capture geometric information

  23. Descriptors Solution: • Divide the area • For each section compute it’s own histogram SIFT - David Lowe 1999

  24. Descriptors - SIFT How to compute SIFT descriptor Input: an image and a location to compute the descriptor 16 x 16 Step 1: Warp the image to the correct orientation and scale, and than extract the feature as 16x16 pixels

  25. Descriptors - SIFT Step 2: Compute the gradient for each pixel (direction and magnitude) 16 x 16 Step 3: Divide the pixels into 16, 4x4 squares

  26. Descriptors - SIFT Step 4: For each square, compute gradient direction histogram over 8 directions. The result: 128 dimensions feature vector.

  27. Descriptors - SIFT • Warp the feature into 16x16 square. • Divide into 16, 4x4 squares. • For each square, compute an histogram of the gradient directions. => Feature vector (128)

  28. Descriptors - SIFT • Weighted by magnitude and Gaussian window ( σ is half the window size) • Use L2 distance to compare features Can use other distance functions • X^2 (chi square) • Earth mover’s distance • Normalize the feature to unit vector

  29. Descriptors - SIFT • Invariance to illumination • Gradient are invariant to Light intensity shift (i.e. add a scalar to all the pixels) Invariance to shift and rotation • Histograms does not contains any geometric information • Using 16 histograms allows to preserve geometric information. • Normalization to unit length add invariance to light intensity change (i.e. multiply all the pixels by a scalar)

  30. Descriptors - GLOH • Similar to SIFT • Divide the feature into log-polar bins instead of dividing the feature into square. • 17 log-polar location bins • 16 orientation bins • We get 17x16=272 dimensions. Analyze the 17x16=272 Dimensions Apply PCA analysis, keep 128 components C. S. Krystian Mikolajczyk. A performance evaluation of local descriptors. TPAMI 2005

  31. SURF • Use integral images to detect and describe SIFT like features • SURF describes image faster than SIFT by 3 times • SURF is not as well as SIFT on invariance to illumination change and viewpoint change

  32. Descriptors Histograms of Oriented Gradients Descriptors SIFT David Lowe 1999 GLOH Mikolajczyk K., Schmid C 2005 SURF Bay H., Ess A., Tuytelaars T., Van Gool L 2008

  33. Outline Overview Image Descriptors Histograms of Oriented Gradients Descriptors Shape Descriptors Color Descriptors Video Descriptors

  34. Descriptors

  35. Descriptors - Shape Context =? Assume we have a good edge detector Take a patch of edges? Not invariant to small deformations in the shape

  36. Descriptors - Shape Context • Quantize the edges surface using a log-polar binning • In each bin, sum the number of edge points

  37. Descriptors - Shape Context

  38. Descriptors - Shape Context

  39. Complex Notion of Similarity

  40. The Local Self-Similarity Descriptor Input image Correlation surface Image descriptor

  41. 3 2 1 1 2 3 2 1 2 3 3 1 The Local Self-Similarity Descriptor

  42. MAX Color Texture Edges The Local Self-Similarity Descriptor Input image Correlation surface Image descriptor • Properties & Benefits: • A unified treatment of repetitive patterns,color,texture,edges • Captures the shape of a local region • Invariant to appearance • Accounts for small local affine & non-rigid deformations

  43. Template image:

  44. Descriptors Shape Descriptors Allows measuring of shape similarity Shape Context Belongie S., Malik J., Puzicha J. Shape Matching and Object Recognition Using Shape Contexts. PAMI, 2002. Local Self-Similarity Shechtman E., Irani M. Matching Local Self-Similarities across Images and Videos. CVPR, 2007. Geometric Blur rg A. C., Malik J. Geometric Blur for Template Matching. CVPR, 2001. Outperform the commonly used SIFT in object classification task Horster E., Greif T., Lienhart R., Slaney M. Comparing local feature descriptors in pLSA-based image models. 

  45. Outline Overview Image Descriptors Histograms of Oriented Gradients Descriptors Shape Descriptors Color Descriptors Video Descriptors

  46. Color Descriptors

  47. Color Descriptors Color spaces • RGB • HSV • Opponent

  48. Color Descriptors Opponent color space • intensity information is represented by channel O3 • color informationis represented by channel O1 and O2 • O1 and O2 are invariant to offset

  49. Color Descriptors • RGB color histogram • Opponent O1, O2 • Color moments • Use all generalized color moments up to the second degree and the first order. • Gives information on the distribution of the colors.

  50. Color Descriptors RGB-SIFT descriptors are computed for every RGB channel independently Normalize each channel separately Invariant to light color change • rg-SIFT - SIFT descriptors over to r and g channels of the normalized-RGB space (2x128 dimensions per descriptor) • OpponentSIFT - describes all the channels in the opponent color space • C-SIFT - Use O1/O3 and O2/O3 of the opponent color space (2x128 dimensions per descriptor) • Scale-invariant with respect to light intensity. • Due to the definition of the color space, the offset does not cancel out when taking the derivative • G. J. Burghouts and J. M. Geusebroek • Performance evaluation of local color invariants 2009

More Related