Image Segmentation Jianbo Shi Robotics Institute Carnegie Mellon University

1. Image SegmentationJianbo ShiRobotics InstituteCarnegie Mellon University Cuts, Random Walks, and Phase-Space Embedding Joint work with: Malik,Malia,Yu

2. Taxonomy of Vision Problems • Reconstruction: • estimateparameters of external 3D world. • Visual Control: • visually guided locomotion and manipulation. • Segmentation: • partition I(x,y,t) into subsets of separate objects. • Recognition: • classes: face vs. non-face, • activities: gesture, expression.

4. We see Objects

5. Outline • Problem formulation • Normalized Cut criterion & algorithm • The Markov random walks view of Normalized Cut • Combining pair-wise attraction & repulsion • Conclusions

6. Edge-based image segmentation • Edge detection by gradient operators • Linking by dynamic programming, voting, relaxation, … Montanari 71, Parent&Zucker 89, Guy&Medioni 96, Shaashua&Ullman 88 Williams&Jacobs 95, Geiger&Kumaran 96, Heitger&von der Heydt 93 - Natural for encoding curvilinear grouping - Hard decisions often made prematurely

7. Region-based image segmentation • Region growing, split-and-merge, etc... - Regions provide closure for free, however, approaches are ad-hoc. • Global criterion for image segmentation • Markov Random Fields e.g. Geman&Geman 84 • Variational approaches e.g. Mumford&Shah 89 • Expectation-Maximization e.g. Ayer&Sawhney 95, Weiss 97 - Global method, but computational complexity precludes exact MAP estimation - Problems due to local minima

8. Bottom line: It is hard, nothing worked well, use edge detection, or just avoid it.

9. Global good, local bad. • Global decision good, local bad • Formulate as hierarchical graph partitioning • Efficient computation • Draw on ideas from spectral graph theory to define an eigenvalue problem which can be solved for finding segmentation. • Develop suitable encoding of visual cues in terms of graph weights. Shi&Malik,97

10. Image segmentation by pairwise similarities • Image = { pixels } • Segmentation = partition of image into segments • Similarity between pixels i and j Sij = Sji 0 Sij • Objective: “similar pixels should be in the same segment, dissimilar pixels should be in different segments”

11. i A Segmentation as weighted graph partitioning Pixels iI = vertices of graph G Edges ij = pixel pairs with Sij> 0 Similarity matrix S = [ Sij ] is generalized adjacency matrix di = Sj Sij degree of i vol A =Si Adi volume of A I i Sij j

12. Cuts in a graph • (edge) cut = set of edges whose removal makes a graph disconnected • weight of a cut cut( A, B ) =Si A,j BSij • the normalized cut NCut( A,B ) = cut( A,B )( + ) 1 . vol A 1 . vol B

13. Normalized Cut and Normalized Association • Minimizing similarity between the groups, and maximizing similarity within the groups can be achieved simultaneously.

14. The Normalized Cut (NCut) criterion • Criterion min NCut( A,A ) Small cut between subsets of ~ balanced grouping NP-Hard!

15. Some definitions

16. Normalized Cut As Generalized Eigenvalue problem • Rewriting Normalized Cut in matrix form:

17. More math…

18. y2i i A y2i i A Normalized Cut As Generalized Eigenvalue problem • after simplification, we get

19. Interpretation as a Dynamical System

20. Interpretation as a Dynamical System

21. Brightness Image Segmentation

22. brightness image segmentation

23. Results on color segmentation

24. Malik,Belongie,Shi,Leung,99

26. Motion Segmentation with Normalized Cuts • Networks of spatial-temporal connections:

27. Motion Segmentation with Normalized Cuts • Motion “proto-volume” in space-time • Group correspondence

28. Results

29. Results Shi&Malik,98

30. Results

31. Results

32. Stereoscopic data

33. Conclusion I • Global is good, local is bad • Formulated Ncut grouping criterion • Efficient Ncut algorithm using generalized eigen-system • Local pair-wise allows easy encoding and combining of Gestalt grouping cues

34. FAQs • Why is this segmentation criterion better? • 30% • Is there a good way of picking W(i,j) • 40% • How do we integrate prior information? how is higher level information encoded? - 20%

35. Learning Segmentation • Learning Segmentation with Random Walk Maila & Shi, NIPS ‘00

36. Goals of this work • Better understand why spectral segmentation works • random walks view for NCut algorithm • complete characterization of the “ideal” case • ideal case is more realistic/general than previously thought • Learning feature combination/object shape model • Max cross-entropy method for learning Malia&Shi,00

37. The random walks view • Construct the matrix P = D-1S D = S = • P is stochastic matrix Sj Pij = 1 • P is transition matrix of Markov chain with state space I p = [ d1 d2 . . . dn ]Tis stationary distribution S11 S12 S1n S21 S22 S2n . . . Sn1 Sn2 Snn d1 d2 . . . dn 1 . vol I

38. Reinterpreting the NCut criterion NCut( A, A ) = PAA + PAA PAB = Pr[ A --> B | A ] under P, p • NCut looks for sets that “trap” the random walk • Related to Cheeger constant, conductivity in Markov chains

39. Reinterpreting the NCut algorithm (D-W)y = mDy m1=0 m2 . . . mn y1 y2 . . . Yn Px = lx l1=1 l2 . . . ln x1 x2 . . . xn mk = 1 - lk yk = xk The NCut algorithm segments based on the second largest eigenvector of P

40. So far... • We showed that the NCut criterion & its approximation the NCut algorithm have simple interpretations in the Markov chain framework • criterion - finds “almost ergodic” sets • algorithm - uses x2 to segment Now: • Will use Markov chain view to show when the NCut algorithm is exact, i.e. when P hasK piecewise constant eigenvectors

41. Piecewise constant eigenvectors: Examples • Block diagonal P (and S) Eigenvectors S Eigenvalues • Equal rows in each block Eigenvectors P Eigenvalues

42. Piecewise constant eigenvectors: general case • Theorem[Meila&Shi] Let P = D-1S with D non-singular and let D be a partition of I. Then P has K piecewise constant eigenvectors w.r.t D iff P is block stochastic w.r.t D and P non-singular. Eigenvectors P Eigenvalues

43. Block stochastic matrices D = ( A1, A2, . . . AK ) partition of I • Pis block stochastic w.r.t D Sj As Pij = Sj As Pi’jfori,i’ in same segment As’ • Intuitively: Markov chain can be aggregated so that random walk over D is Markov • P= transition matrix of Markov chain over D

44. Learning Image fijq normalize Modell Sij Pij Pij* Segmentation 0, j A 00, j A 1 . |A| Learning image segmentation • Targets Pij* = for i in segment A • Model Sij= exp( Sqlqfqij )

45. J lq The objective function J = - Si I Sj I Pij* log Pij J = KL( P* || P ) where p0= and Pi j* =p0Pij* the flow i j • Max entropy formulation maxPij H( j | i ) s.t. <fijq>p0Pij = <fijq>p0Pij* for all q Convex problem with convex constraints at most one optimum • The gradient = <fijq> p0Pij* -<fijq> p0Pij 1 . |I| `1234567890Qwertyuiop[] Asdfghjkl;’ zxcvbnm,./ -=\ 1 . |I|

46. Experiments - the features • IC - intervening contour fijIC = max Edge( k ) k (i,j) Edge( k ) = output of edge filter for pixel k Discourages transitions across edges • CL - colinearity/cocircularity fijCL =+ Encourages transitions along flow lines • Random features i k j Edgeness 2-cos(2ai)-cos(2aj) 1 - cos(al) 2-cos(2ai + aj) 1 - cos(a0) ai j i aj orientation

47. Training examples lCL lIC

48. Test examples Original Image Edge Map Segmentation

49. Conclusion II • Showed that the NCut segmentation method has a probabilistic interpretation • Introduced • a principled method for combining features in image segmentation • supervised learning and synthetic data to find the optimal combination of features Graph Cuts Generalized Eigen-system Markov Random Walks