Segmentation from Examples

1. Segmentation from Examples By: A’laa Kryeem Lecturer: Hagit Hel-Or

2. What is Segmentation from Examples? • Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain • Effective when making a semantic segmentation

3. Why to use Examples • The example defines the granularity of the desired output • Give us the ability to characterize meaningful parts in the image • Using example allow us to use non-parametric model

4. The example defines the granularity of the desired output: Desired Segmentation Training image Test Image Induced Segmentation

5. Why to use Examples • The example defines the granularity of the desired output • Give us the ability to characterize meaningful parts in the image

6. Give us the ability to characterize meaningful parts in the image

7. Semantic Segmentation from an example • We want to segment an image into semantically meaningful parts • Required in various applications

8. Semantic Segmentation from an example • We want to segment an image into semantically meaningful parts • Required in various applications • Problems: • Meaningful parts are often too complex • Semantic interpretation is highly subjective, depending on both the application, and the user

9. Meaningful parts are often too complex

10. Semantic Segmentation from an example • We want to segment an image into semantically meaningful parts • Required in various applications • Problems: • Meaningful parts are often too complex • Semantic interpretation is highly subjective, depending on both the application, and the user

11. Example of image different segmentation

12. Semantic Segmentation from an example • So, How to achieve semantic segmentation • Getting segmented training image(s) as input

13. Training set

14. Semantic Segmentation from an example • So, How to achieve semantic segmentation • Getting segmented training image(s) as input • Using non-parametric representation

15. non-parametric model Each semantic part is represented by a set of square patches

16. Semantic Segmentation from an example • So, How to achieve semantic segmentation • Getting segmented training image(s) as input • Using non-parametric representation • Over-segmenting the Test image into small fragments

17. Over segmented image

18. Semantic Segmentation from an example • So, How to achieve semantic segmentation • Getting segmented training image(s) as input • Using non-parametric representation • Over-segmenting the Test image into small fragments • Compute costs for fragment-label pairs

19. (fragment,label) cost example ?

20. Semantic Segmentation from an example • So, How to achieve semantic segmentation • Getting segmented training image(s) as input • Using non-parametric representation • Over-segmenting the Test image into small fragments • Compute costs for fragment-label pairs • Graph-cuts multi-label optimization

21. Why do we need graph-cuts • Graph-cuts optimization is used to label each fragment in a globally optimal manner

22. Training set Classification scores Patch sets Classification Graph-Cuts optimization Fragments Test image Fragmentation Result

23. Over segmenting • Fragment: small arbitrarily-shaped and simply-connected pixel clusters • We assume that small homogeneous regions always belong to the same semantic part

24. Over segmenting • Fragment: small arbitrarily-shaped and simply connected pixel clusters • We assume that small homogeneous regions always belong to the same semantic part • Advantages: • Enforces a locally coherent labeling • Reduces the computational complexity

25. Graph-cuts multi-label optimization • For each fragment we have k cost values, we need to determine which one is the optimal • Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)

26. Algorithm for semantic segmentation • Pixel labeling costs • Fragmentation • Fragment labeling costs • Graph-cuts optimization

27. Algorithm for semantic segmentation • Pixel labeling costs • Fragmentation • Fragment labeling costs • Graph-cuts optimization

28. Pixel labeling costs • Given Itrainand Ltrain • Representing each label(segment) in Ltrain by a set of square patches • We get k sets {Sl}l=1,…,k one for each label

29. Pixel labeling costs (cont.) • Next, we define φ (p, l) for each (pixel,label) pair • The cost of assigning label l to pixel pItest p:pixel at Itest l : label in Ltrain ssd(P,P’) is the sum of squared distances between P,P’ M:mxmx3 P:mXm neighborhood centered at p P’:mxmpatch = min P’Sl

31. Algorithm for semantic segmentation • Pixel labeling costs • Fragmentation • Fragment labeling costs • Graph-cuts optimization

32. Fragmentation • We partition Itest into small,color-homogeneous regions using mean shift segmentation

33. Fragmentation • We partition Itest into small,color-homogeneous regions using mean shift segmentation • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)

34. Fragmentation • We partition Itest into small,color-homogeneous regions using mean shift segmentation • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions) • Fragment boundaries align with edges in the image

35. Fragmentation (cont.) Random colorization Detailed close-up

36. Algorithm for semantic segmentation • Pixel labeling costs • Fragmentation • Fragment labeling costs • Graph-cuts optimization

37. Fragment labeling costs • Voting scheme in order to compute labeling costs of each fragment • For each fragment fItest we pick a few representative pixels: Rep(f)={pi f } i=I,…,Rf Rf is proportional to |f|

38. Fragment labeling costs (cont.) • We talked about the cost of assigning a pixel into a label Rep(f)}

39. Fragment labeling vs. pixel labeling • Fragment labeling reducing complexity • We have n=Assume |Rep(f)|= ,then we need to compute costs only for pixels

40. Fragment labeling vs. pixel labeling Training seg. Training image Input image • Enforces a locally coherent labeling Fragment labeling Pixel labeling

41. Algorithm for semantic segmentation • Pixel labeling costs • Fragmentation • Fragment labeling costs • Graph-cuts optimization

42. Graph-cuts optimization After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i fragment labeling costs. Costs range in the interval [0,1]

43. Graph-cuts optimization • Now for each pixel pItestwe have a labeling cost • We need to find Ltestthe globally optimal labeling • Requirements: • Minimizes the total labeling cost • Consistent with presence (or absence) of edges

44. Graph-cuts optimization (cont.) • For each pair of neighboring pixels we define: Ψ(p,q,L(p),L(q))= L(p),L(q) : labels assigned to p,q : difference between pixels p,q (RGB euclidian distance)

45. Graph-cuts optimization (cont.) • In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels : : cost of assigning fragment f to label L(f), weighted by the size of each fragment. : neighboring fragments in Itest. = :controls trade-off between regions and boundaries

46. Intuition: • Big value: For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too. This mean we prefer continues regions and not edges.

47. Intuition: • s value: For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value. Favors boundaries,holding out non-continues regions .

48. Graph-cuts optimization (cont.) • Finally Ltest is determined by solving Ltest=minLE(L) Labeling after Graph-Cuts Optimization Fragment labeling

49. Multi-label graph-cut Colored nodes:labels Squares : fragments For each (fragment,label) pair we have an edge. Edge weigh according to φ. Edges between two squares weighed according to Ψ.

50. Multi-label graph-cut Induced graph Each fragment connected to a single label.