Michael J. Swain and Dana H. Ballard C o l o r Indexing

1. Michael J. Swain and Dana H. BallardColor Indexing Presented by Hoang Nguyen CS 598KN February 13, 2005

2. Motivation • Visual Skills for Robots • What: Identify an object with a known location • Where: Locate a known object • What & Where: Locate & Identify ???

3. Motivation (cont.) • The Role of Color in Vision • Geometrical features vs. Color features • Color correlates with the class identity of an object • Color constancy: Perception of color over varying light conditions Chair ! What type of fish?

4. Objective • Algorithms that… • Can identify an object with a known location • Can locate a known object in a crowded scene • Are able to operate in realtime • Are able to operate in varying light conditions • Can tolerate a “certain level of change” in view Color ?

5. Outline • Background • Identification Problem • Location Problem • Comments/Critiques

6. Background • Color Histogram • Defined in color space (RGB, opponent colors axes..) • Obtained by discretizing the image colors and counting the number of times each discrete color occurs in the image array • A bin is a group of similar colors • Properties: • Invariant to translation and rotation. • Changes slowly under change of angle of view, change in scale and occlusion. • Depends on lighting condition

7. Identification problem • Problem Formulation: • Given an image I and a set of model images. • Find the model M most similar to I. Image I Models ?

8. Histogram intersection algorithm • I and M are histogram of image and model • The background of the model M is segmented • Each has n bins • Histogram Intersection = = Number of pixels from the model that have corresponding pixels of the same color in theimage • Histogram Match Value:

9. Histogram Intersection (cont.) Model Histogram of Model M Min(I, M) Histogram of Image I Image Histogram Intersection = 3 H(I,M) = 3 / (4 + 3 + 4 + 1) = 3/12

10. then Histogram Intersection (cont.) • If image histogram can be “scaled” to be the same size as the model histogram • The fraction of the multidimensional space defined by the bins of the histogram occupied by a single model is at most (proved in the paper) => Capable of differentiating among a large number of different objects. Histogram space 1 -  is the minimum Histogram Intersection match value

11. Experiments • 32 objects are matched with 66 models under varying conditions • Use opponent color axes RG = R – G BY = 2 * B – R – G WB = R + G + B (intensity) • Under • Normal condition • Varying in view • Varying in image resolutions • Occlusion • Varying in bin resolutions • Varying in light intensity

12. Results (32 objects are matched)

13. Image Result – Normal condition  2 / 5 < 1 / 2 !!!

14. Indexing into a Large Database • Incremental Intersection • Can be interrupted at any time to get the current result • Offline phase • Assign to each bin in each model histogram a key which is the fraction of the total number of pixels in the histogram that fall in that bin. • Group the bins by index (color) • Sort each group by keys • Online phase • Sort the image histogram bins by size • Start from the largest to B largest, match the image bin to all the model bins with the same index whose key is larger.

15. Location Problem • Problem Formula • Problem: Given target image M, locate it in crowded image I • Algorithm • Histogram Backprojection • Example

16. Histogram Backprojection – Step 1 Target Image = • Mj, Ij is the bin jth of the model and the image • R is the ratio histogram Crowded Image

17. Histogram Backprojection – Step 2 • R is the ratio histogram • C(x,y) is the color at pixel (x,y) • h ( c ) returns the bin of color c • B is the backprojected image Crowded Image R Backprojected Image B

18. Histogram Backprojection – Step 3 convolution • B is the backprojected image • D( r ) is a linear filter with radius r

19. Histogram Backprojection – Step 4

20. Result • Quite efficient • Insensitive to changing the image and histogram resolution • Sensitive to failures of color constancy

21. Comments • A Bible for Color techniques in Computer Vision. • Very simple and efficient. • Under certain assumptions, the accuracy is high. • Suitable for realtime application.

22. Critiques • All images are taken under fixed color light source and only small changes in light/surface pose. • Caution: (from [2]) • Pixel-based normalization • R’ = R / (R + G + B), • G’ = G / (R + G + B) • B’ = B / (R + G + B) • Channel-based normalization • R = R / (Ri), G = G / (Gi), B = B / (Bi) • Experiment (city-block metrics)

23. Critiques (cont.) • Background of model images are segmented. • Images being matched to the database have fairly large regions of constant color => Does not work for texture images. • Application (color + other features) • Face detection • Skin detection • Finding naked people (D. Forsyth)

24. Questions ???

25. References • Michael J. Swain and Dana H. Ballard, Color Indexing. • Graham D. Finlayson et al., Using Colour for Image Indexing. • M. Fleck, D. Forsyth, C. Bregler, Finding naked people. • http://rsb.info.nih.gov/ij/, ImageJ

26. Large database (cont.) • The complexity of sort histogram bins dominates • Match effectiveness climbs quickly