Heather Dunlop 16-721: Advanced Perception January 25, 2006

1. Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cuesby David R. Martin, Charless C. Fowlkes, Jitendra Malik Heather Dunlop 16-721: Advanced Perception January 25, 2006

2. Human Canny Martin, 2002 What is a Boundary?

3. Dataset “You will be presented a photographic image. Divide the image into some number of segments, where the segments represent ‘things’ or ‘parts of things’ in the scene. The number of segments is up to you, as it depends on the image. Something between 2 and 30 is likely to be appropriate. It is important that all of the segments have approximately equal importance.”

4. Dataset • Database of over 1000 images and 5-10 segmentations for each Martin, 2002

5. Boundaries Non-boundaries Boundaries Intensity Brightness Color Texture Martin, 2002

6. Image Boundary Strength Optimized Cues Cue Combination Brightness Model Color Texture Benchmark Human Segmentations Method • Goal: learn the probability of a boundary, Pb(x,y,θ) Martin, 2002

7. Image Features • CIE L*a*b* color space (luminance, red-green, yellow-blue) • Oriented Energy: • fe: Gaussian second derivative • fo: Its Hilbert transform • Brightness • L* distribution • Color • a* and b* distributions (joint or marginal) • Texture

8. Texture • Convolve with a filter bank: • Gaussian second derivative • Its Hilbert transform • Difference of Gaussians • Filter responses give a measure of texture

9. Other Filter Banks Leung-Malik filter set: Schmid filter set: Maximum Response 8 filter set:

10. Textons • Convolve image with filter bank • Cluster filter responses to form textons Adapted from Martin, 2002 and Varma, Zisserman, 2005

11. Texton Distribution • Assign each pixel to nearest texton • Form distribution of textons Adapted from Martin, 2002 and Varma, Zisserman, 2005

12. r (x,y)  Gradient-based Features • Brightness (BG), color (CG), texture (TG) gradients • Half-disc regions described by histograms • Compare distributions with χ2 statistic

13. Texture Gradient • Texton distribution in two half circles Martin, 2002

14. Localization • Tightly localize boundaries • Reduce noise • Coalesce double detections • Improve OE and TG features OE OE localized TG TG localized Martin, Fowlkes, Malik, 2004

15. Optimization • Texture parameters: • type of filter bank • scale of filters • number of textons • universal or image-specific textons • Other possible distance/histogram comparison metrics • Number of bins for histograms • Scale parameter for all cues

16. Evaluation Methodology • Posterior probability of boundary: Pb(x,y,θ) • Evaluation measure: precision recall curve • F-measure: Martin, 2002

17. Cue Combination • Which cues should be used? • OE is redundant when other cues are present • BG+CG+TG produces best results Martin, 2002

18. Classifiers • Until now, only logistic regression was used • Other possible classifiers: • Density estimation • Classification trees • Hierarchical mixtures of experts • Support vector machines Martin, 2002

19. Result Comparison • Alternative methods: • Matlab’s Canny edge detector with and without hysteresis • Spatially-averaged second moment matrix (2MM) Martin, 2002

20. Image Canny 2MM BG+CG+TG Human Results Martin, 2002

21. Image Canny 2MM BG+CG+TG Human Results Martin, 2002

22. Image Canny 2MM BG+CG+TG Human Results Martin, 2002

23. Conclusions • Large data set used for testing • Texture gradients are a powerful cue • Simple linear model sufficient for cue combination • Outperforms existing methods • An approach that is useful for higher-level algorithms • Code is available online: http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/