Optimisation in Image & Video Processing Graph Cuts & Interactive Segmentation

1. Optimisation in Image & Video ProcessingGraph Cuts & Interactive Segmentation Dr. David Corrigan

2. Graph Cuts. Why? • Can be used to find the global minimum solution for certain energy minimisation problems • Binary labelling problems with submodular prior potential/energy/cost functions • Multilabel problems with convex priors • Allows you to focus on the model rather than the optimisation. • Can find local minimum for other classes of functions • Multilabel problems with concave priors • Less sensitive to initialisation and local minima than ICM.

3. Key References • 3 lectures on Graph Cuts by Richard Hartley • http://videolectures.net/ssll09_hartley_covi/ • All on syllabus except for higher order cliques • Andrew Blake PushmeetKohli, and Carsten Rother, eds. Markov random fields for vision and image processing. The MIT Press, 2011. (Chapters 2, 3 and 7) • Boykov, Y., and Jolly, M.-P.“ Interactive graph cuts for optimal boundary & region segmentation of objects in ND images." Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on. Vol. 1. IEEE, 2001. • Rother, C., Kolmogorov, K., and Blake, A. "Grabcut: Interactive foreground extraction using iterated graph cuts." ACM Transactions on Graphics (TOG). Vol. 23. No. 3. ACM, 2004.

4. What is a Graph Cut? 0 0 n-edge\link t-edge\link 1 1 A Graph Cut partitions the Nodes into 2 subsets and , and the label for each node is Each Grey Node represents a pixel, and the source and sink represent the 0 and 1 label respectively

5. Graph Cuts • Each edge on the graph is assigned a cost or weight. How are weights assigned to each edge? • Consider the energy function for an MRF where Likelihood energy – unary term prior energy – pairwise term

6. How to assign Costs to Links? 0 Note: Boykov & Jolly use slightly different convention but this is equivalent with the tables in their paper For now assume All Weights > 0 We will relax this constraint later 1

7. How to assign weights • Consider an example of a graph with 2 pixels:

8. Finding the Optimal Cut • Key idea: Energy Function for a segmentation is the cost of the corresponding cut. • Min. Energy Min. Cut Cost. • For each node either or must be part of cut but not both • For pixel image, • valid segmentations and cuts

9. Finding the Optimal Cut Red Cut is the Min Cut So both and are segmented with label = 1.

10. Re-Parameterisation of the Graph • We can add energies to the graph with out changing the minimum cut

11. Re-Parameterisation of the Graph

12. Optimisation Algorithms • Max Flow/Min Cut • Ford & Fulkerson 1962 • Dinic 1970 • Boykov Kolmogorov 2004 • Water pipe analogy • Idea is to push flow from source to sink until there is no spare capacity in the graph. • Push-Relabel • Goldberg & Tarjan 1988 • Can be multi-threaded • Eg. CUDA Cuts

13. Max Flow/Min Cut

14. Max Flow/Min Cut

15. Max Flow/Min Cut No more paths remaining. DONE ,

16. When can we use Graph Cuts • We can use this principle to relax our conditions on the edge weights • All Weights > 0 • Through re-parameterisation any sub-modular can be minimised

17. Motion Detection Example • This is known as a Potts Model • In general spatial smoothness priors are sub-modular and so can be optimised using Graph Cuts • Algorithms that penalise spatial smoothness cannot be.

18. Interactive Segmentation

19. Boykov & Jolly ICCV 2001 • Black and White Image Segmentation = +

20. Boykov & Jolly ICCV 2001 • The Prior implements a spatial smoothness CRF where smoothness is reduced near image edges.

21. User Strokes • Define the sets • – the subset of nodes that have been labelled as object/foreground 1 • - the subset of nodes labelled as 1. • Likelihoods based on histograms of foreground and background • Strokes Define Hard Constraints • In practice a sufficiently large value is used instead of

22. Interactive Editing • Uses graph re-parameterisation to allow user to quickly touch up segmentation by adding more strokes. Both graphs have the same min cut Residual Graph after max flow algorithm Graph before max flow algorithm

23. Interactive Editing • Lets say we want to force node to have Graphs will have the same mincut. Run Max Flow on the right graph

24. Interactive Editing No more paths remaining. DONE It is much faster to rerun optimisation on the re-parameterised residual graph rather than rerun it from scratch ,

25. Interactive Editing

26. GrabCut. Rother et al. SIGGRAPH ‘04 • New Interaction Modality

27. GrabCut • Full Colour Segmentation Algorithm • Prior same as for Boykov & Jolly Paper • GMMs for Foreground and Background Likelihoods. • Rectangle specifies • Data for training foreground and background likelihoods • Hard constraints for background. • Segmentation inside the window, everywhere outside is background • No hard constraints for Foreground

28. GrabCut • GMMs optimised using Expectation Maximisation • Segmentation optimised using Graph Cuts • Iteration until convergence • Interactive editing with single graph cut optimisation • Optional alpha matting after convergence to get transparency along the boundary. Optimisation of Likelihoods Optimisation of Segmentation

29. Multi-Label Segmentation Global Optimisation of Energy Functions with Convex Priors

30. Global Optimisation of Graphs with Convex Priors • Consider the Stereo Matching / Disparity Estimation Problem Right Left Depth

31. Stereo Matching • We can use a likelihood similar to the energy function in Block Matching. • Consider at pixel at a position where Also possible to use sum squared difference over a block

32. Convex Priors • We want do define • If it is convex we can solve exactly using Graph Cuts. • is convex for • L1 norm, L2 norm. • could potentially enforce a Conditional Random Field. • However the prior is not convex so the global minimum cannot be found using this technique.

33. Stereo Matching using Graph Cuts • Lets use • There are multiple nodes per pixel. • node where is the number of different label values.

34. Stereo Matching using Graph Cuts • From a binary ST-cut it is possible to deduce a labelling. • The value of the label for a pixel is corresponds to the label whose likelihood is part of the ST-cut • So in the cut shown here and . • The cost of cutting all the n-links is given by • In this case 3 n-links are cut and the total cost is • Different convex priors will have different graph structures • Difficult if not L1 norm

35. Multi-Label Segmentation Approximate Optimisation of Energy Functions with Concave Priors

36. Convex v Concave Priors • Convex Priors tend to over-smoothen segmentation results

37. Discontinuity Preserving Priors • Potts Model • Truncated Linear Model

38. -Expansion Move Algorithm Segmentation using Truncated Linear Model Segmentation using Potts Model

39. What is an -Expansion? • Consider segmentation problem with multiple label values and some current segmentation result. • For any label , an -Expansion allows pixels with other labels to either stay the same or switch to the label • This can be computed using a binary graph cuts segmentation Alpha Expansion

40. -Expansion Move Algorithm • Start with an Initial Solution • For each label visited in some order • Compute Optimal -Expansion move. • Decline the move if there is no energy decrease. • Stop when no move can decrease the energy. • Like ICM it is an iterative algorithm and is prone to finding local minima • However it is much less prone to local minima than ICM and less sensitive to the choice of inititalisation.

41. Setting up Graph for an -Expansion Move • Preliminaries • The likelihood is known. • The prior is known. • An initial segmentation exists. • The updated segmentation will be • The goal is to estimate a binary segmentation of a label field where

42. Setting up Graph for an -Expansion Move • Need to define likelihood and prior for the binary segmentation This will not be 0 in general as the 2 labels can be different

43. Setting up Graph for an -Expansion Move Previous labels

44. Setting up Graph for an -Expansion Move

45. What type of priors can be used? • Hence • Hence • The prior for the binary segmentation must be submodular • Hence • ie. For then is concave

46. initial solution -expansion -expansion -expansion -expansion -expansion -expansion -expansion Stereo Matching Example

47. Other Applications of Graph Cuts

48. Graph-Cut Textures (Kwatra et al. SIGGRAPH ‘03) Image 1 Image 2 http://www.cc.gatech.edu/cpl/projects/graphcuttextures/

49. Graph-Cut Textures • Cut where the two patches agree with each other the best • There is no likelihood • Hard Constraints specified at some points (usually the border of the overlap regions) Overlap Area

50. Torn Frame Segmentation (Corrigan ICIP ‘06) Tear Edge Removal Tear Displacement Removal