CS 4487/9587 Algorithms for Image Analysis

1. CS 4487/9587 Algorithms for Image Analysis Segmentation (2D) Acknowledgements: Alexei Efros, Steven Seitz

2. CS 4487/9587 Algorithms for Image AnalysisSegmentation (2D) • Blobs • Need for blobs • Extracting blobs • Thresholding, region growing, mean-shift • Minimum spanning trees • Object extraction • Intelligent scissors (also known as live-wire) • Snakes • Region+boundary methods • Local, greedy, and global processing Extra Reading: Sonka at.al. Ch. 5 Gonzalez and Woods, Ch. 10

3. HW assignment 1 • 2D image segmentation • Live-wire (based on Dijkstra algorithm for shortest paths on a graph) • Implement and experiment on many images • Compare with local/greedy methods • Some useful code libraries and sample images will be posted within a couple of days

4. Goal: Extract “Blobs” • What are “blobs”? • Regions of an image that are somehow coherent • Why? • Object extraction, object removal, compositing, etc. • …but are “blobs” objects? • No, not in general

5. Blob’s coherence • Simplest way to define blob coherence is as similarity in brightness or color: The house, grass, and sky make different blobs The tools become blobs

6. Why is this useful? AIBO RoboSoccer (VelosoLab)

7. Ideal Segmentation

8. Result of Segmentation

9. Thresholding • Basic segmentation operation: mask(x,y) = 1 if im(x,y) > T mask(x,y) = 0 if im(x,y) < T • T is threshold • User-defined • Or automatic • Same as histogram partitioning:

10. Sometimes works well… What are potential Problems?

11. …but more often not Adaptive thresholding

12. Region growing • Is this same as global threshold? • Start with initial set of pixels K (initial seed(s)) • Add to pixels p in K their neighbors q if |Ip-Iq| < T • Repeat until nothing changes

13. Region growing

14. What can go wrongwith region growing ? Region growth may “leak” through a single week spot in the boundary

15. Region growing See region leaks into sky due to a weak boundary between them

16. B C A Due to Pedro Felzenszwalb and Daniel HuttenlocherMotivating example This image has three perceptually distinct regions Q: Where would image thresholding fail? A: Region A would be divided in two sets and region C will be split into a large number of arbitrary small subsets Q: Where would region growing fail? A: Either A and B are merged or region C is split into many small subsets Also, B and C are merged Difference along border between A and B is less then differences within C

17. Color-Based Blob Segmentation • Automatic Histogram Partitioning • Given image with N colors, choose K • Each of the K colors defines a region • not necessarily contiguous • Performed by computing color histogram, looking for modes

18. Finding Modes in a Histogram • How Many Modes Are There? • Easy to see, hard to compute

19. mode o x x x Mean Shift [Comaniciu & Meer] • Initialize random seed, and fixed window • Calculate center of gravity ‘x’ of the window (the“mean”) • Translate the search window to the mean • Repeat Step 2 until convergence Iterative Mode Search

20. Mean-Shift

21. Mean-shift results More Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html

22. Issues: • Although often useful, all these approaches work only some of the time, and are considered rather “hacky”. • Can’t even handle our tiger: Problem is that blobs != objects!

23. Extracting objects • How could this be done?

24. Extracting objects • Many approaches proposed • color cues • region cues • contour cues • We will consider a few of these • Today: • Intelligent Scissors (contour-based) • E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995

25. Intelligent Scissors

26. Intelligent Scissors • Approach answers a basic question • Q: how to find a path from seed to mouse that follows object boundary as closely as possible? • A: define a path that stays as close as possible to edges

27. Intelligent Scissors • Basic Idea • Define edge score for each pixel • edge pixels have low cost • Find lowest cost path from seed to mouse mouse seed • Questions • How to define costs? • How to find the path?

28. Path Search (basic idea) • Graph Search Algorithm • Computes minimum cost path from seed to allotherpixels • 3 sets of nodes • Free • Active • Done Note: diagonal “paths” are scaled by a factor . … Why?

29. image gradient scores 2 2 2 2 2 2 2 2 2 2 2 L L 2 Path’s cost along the top boundary Path’s cost along the top-left boundary After diagonal links are adjusted by Segmentation should be Invariant to Image Rotation After object rotation

30. Instead of nodes, image gradient scores can be assigned directly to graph edges • Treat the image as a graph q p • Graph • node for every pixel p • link between every adjacent pair of pixels e=(p,q) • cost w(e) for each link • Note: each link has a cost • this is a little different than the figure before where each pixel (graph node) had a cost

31. Defining edge costs • Treat the image as a graph q • Want to hug image edges: how to define cost of a link? p • the link should follow the intensity edge • want intensity to change rapidly to the link • the cost of edge should be small when the difference of intensity to the link is large T T

32. Defining edge costs • First, estimate image derivative in the direction orthogonal to the edge • Use finite differences (or cross-correlation filters) q 1 1 p -1 -1

33. Defining edge costs • Second, use some penalty function g( ) assigning low penalty to large directional derivatives and large penalty to small derivatives q p

34. Defining edge costs • Finally, cost of en edge e should be local contrast score adjusted by the edge length Why? q p

35. Defining edge costs • When computing the shortest path we approximate a contour C minimizing continuous geometrically meaningful functional Cost of one edge Cost of a PATH Integral of contrast penalty along the contour

36. 9 4 5 1 3 3 3 2 (see Cormen et.al. “Introduction to Algorithms”, p.595)Dijkstra’s shortest path algorithm • 3 sets of nodes • Free • Active • Done link cost 0 • init node costs to , set p = seed point, cost(p) = 0 • expand p as follows: • for each of p’s neighbors q that are not expanded • set cost(q) = min( cost(p) + cpq, cost(q) ) ALGORITHM

37. 9 4 5 1 3 3 3 2 Dijkstra’s shortest path algorithm 4 9 5 • 3 sets of nodes • Free • Active • Done 1 1 1 0 3 3 2 3 • init node costs to , set p = seed point, cost(p) = 0 • expand p as follows: • for each of p’s neighbors q that are not expanded • set cost(q) = min( cost(p) + cpq, cost(q) ) • if q’s cost changed, make q point back to p • put q on the ACTIVE list (if not already there) ALGORITHM

38. 9 4 5 1 3 3 3 2 Dijkstra’s shortest path algorithm 4 9 5 • 3 sets of nodes • Free • Active • Done 2 3 5 2 1 1 0 3 3 3 4 3 2 3 • init node costs to , set p = seed point, cost(p) = 0 • expand p as follows: • for each of p’s neighbors q that are not expanded • set cost(q) = min( cost(p) + cpq, cost(q) ) • if q’s cost changed, make q point back to p • put q on the ACTIVE list (if not already there) • set r = node with minimum cost on the ACTIVE list • repeat Step 2 for p = r ALGORITHM

39. 9 4 5 1 3 3 3 2 Dijkstra’s shortest path algorithm 4 3 6 5 • 3 sets of nodes • Free • Active • Done 2 3 5 3 2 1 1 0 3 3 3 4 4 3 2 3 • init node costs to , set p = seed point, cost(p) = 0 • expand p as follows: • for each of p’s neighbors q that are not expanded • set cost(q) = min( cost(p) + cpq, cost(q) ) • if q’s cost changed, make q point back to p • put q on the ACTIVE list (if not already there) • set r = node with minimum cost on the ACTIVE list • repeat Step 2 for p = r ALGORITHM

40. 9 4 5 1 3 3 3 2 Dijkstra’s shortest path algorithm 4 3 6 5 • 3 sets of nodes • Free • Active • Done 2 3 5 3 2 1 1 0 3 3 3 4 4 3 2 2 3 • init node costs to , set p = seed point, cost(p) = 0 • expand p as follows: • for each of p’s neighbors q that are not expanded • set cost(q) = min( cost(p) + cpq, cost(q) ) • if q’s cost changed, make q point back to p • put q on the ACTIVE list (if not already there) • set r = node with minimum cost on the ACTIVE list • repeat Step 2 for p = r ALGORITHM

41. B A - processed nodes (distance to A is known) - active nodes (front) - active node with the smallest distance value Path Search (basic idea) Dijkstra algorithm

42. Dijkstra’s shortest path algorithm • Properties • It computes the minimum cost path from the seed to every node in the graph. This set of minimum paths is represented as a tree • Running time, with N pixels: • O(N2) time if you use an active list • O(N log N) if you use an active priority queue (heap) • takes < second for a typical (640x480) image • Once this tree is computed once, we can extract the optimal path from any point to the seed in O(N) time. • it runs in real time as the mouse moves • What happens when the user specifies a new seed?

43. Livewire extensions • Directed graphs • Restricted search space • Restricted domain (e.g. near a priori model) • Restricted backward search • Different edge weight functions • Image-Edge strength • Image-Edge Curvature • Proximity to known approximate model/boundary • Multi-resolution processing

44. Results http://www.cs.washington.edu/education/courses/455/03wi/projects/project1/artifacts/index.html