1 / 35

Robust Higher Order Potentials For Enforcing Label Consistency

Robust Higher Order Potentials For Enforcing Label Consistency. Pushmeet Kohli Microsoft Research Cambridge Lubor Ladicky Philip Torr Oxford Brookes University, Oxford CVPR 2008. Image labelling Problems. Assign a label to each image pixel. Object Segmentation. Image Denoising.

toshi
Download Presentation

Robust Higher Order Potentials For Enforcing Label Consistency

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor Ladicky Philip Torr Oxford Brookes University, Oxford CVPR 2008

  2. Image labelling Problems Assign a label to each image pixel Object Segmentation Image Denoising Geometry Estimation Sky Building Tree Grass

  3. Object Segmentation using CRFs (Shotton et al. ECCV 2006) CRF Energy Unary potentials based on Colour, Location and Texture features Encourages label consistency in adjacent pixels

  4. Limitations of Pairwise CRFs • Encourages short boundaries (Shrinkage bias) • Can only enforce label consistency in adjacent pixels • Inability to incorporate region based features Image Unary Potential MAP-CRF Solution

  5. Label Consistency in Image Regions • Pixels constituting some regions belong to • Same plane (Orientation) (Hoiem, Efros, & Herbert, ICCV’05) • Same object (Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06) Image (MSRC) Segmentation (Mean shift)

  6. Image labelling using segments Unsupervised Segmentation Object Labelling Image • Geometric Context • [Hoiem et al, ICCV05] • Object Segmentation • [He et al. ECCV06, Yang et al. CVPR07, Rabinovichet al. ICCV07, Batraet al. CVPR08] • Interactive Video Segmentation • [Wang, SIGGRAPH 2005 ] Not robust to Inconsistent Segments!

  7. Our Higher Order CRF Model Encourages label consistency in regions Multiple Segmentations c

  8. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c

  9. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: 0

  10. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] c Cost: f (|c|)

  11. Label Consistency in Segments • Encourages consistency within super-pixels • Takes the form of a PN Potts model [Kohli et al. CVPR 2007] Does not distinguish between Good/Bad Segments ! c Cost: f (|c|)

  12. Quality based Label Consistency Label inconsistency cost depends on segment quality • How to measure quality G(c)? • [Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others] • Colour and Texture Similarity • Contour Energy Measure quality from variance in feature responses Higher order generalization of contrast-sensitive pairwise potential

  13. Quality based Label Consistency Segment Quality (darker is better) Mean shift segmentation MSRC image

  14. Robust Consistency Potentials gmax PN Potts 0 Too Rigid! 0 1 Inconsistent Pixels gmax 0 Robust 0 1 T Inconsistent Pixels

  15. Robust Consistency Potentials Maximum Inconsistency Cost Number of Inconsistent Pixels Slope gmax 0 Robust 0 1 T Inconsistent Pixels

  16. Minimizing Higher order Energy Functions • Message passing is computationally expensive • High runtime and space complexity - O(LN) • L = Number of Labels, N = Size of Clique • Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007] • 2x2 clique potentials for Image Denoising • Take minutes per iteration (Hours to converge)

  17. Minimizing Higher order Energy Functions • Graph Cut based move making algorithm [Kohli et al. CVPR 2007] • Can handle very high order energy functions • Extremely efficient: computation time in the order of seconds • Only applicable to some classes of functions (PN Potts) • Cannot handle robust consistency potential • This paper • Can minimize a much larger class of higher order energy functions • Same time complexity as [Kohli et al. CVPR 2007]

  18. Move making algorithms Expansion and Swap move algorithms [Boykov Veksler and Zabih, PAMI 2001] • Makes a series of changes to the solution (moves) • Each move results in a solution with smaller energy Current Solution How to minimize move functions? Move to new solution Generate pseudo-boolean move function Minimize move function to get optimal move

  19. Minimizing Move Functions using Graph Cuts Most pairwise CRF models used in Computer Vision lead to submodular move functions Second order Pseudo-boolean Function Minimization (submodular) st-mincut (Positive weights) Optimal moves can be found extremely efficiently using graphs cuts

  20. Minimizing Higher Order Energy Functions: Our results • Result 1 • We show that a large class of higher order potentials lead to higher order submodular move functions • Can be minimized in polynomial time • Submodular Function Minimization • Minimizing general submodular functions is computationally expensive • Complexity O(n6) • Cannot handle large problems! Details in Technical Report

  21. Minimizing Higher Order Energy Functions: Our results • Minimizing Higher order functions using Graph cuts • Higher order functions can be transformed to second order functions by adding auxillary variables • Exponential number of auxillary variables needed in general • Result 2 • Our higher order functions can be transformed to second-order functions using ≤2auxillary variables per potential. • Can be minimized extremely efficiently • Complexity << O(n6) Details in Technical Report

  22. Overview of our Method Higher Order Energy Unary Potentials [Shotton et al. ECCV 2006] + Energy Minimization Contrast Sensitive Pairwise Potentials + Segmentation Solution Higher Order Potentials (Multiple Segmentations)

  23. Experimental results Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]

  24. Qualitative Results Image (MSRC-21) PairwiseCRF Higher order CRF Ground Truth Grass Sheep

  25. Qualitative Results (Contd..) Image (MSRC-21) PairwiseCRF Higher order CRF Ground Truth Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV 2006

  26. Quantitative Results: Problems Rough ground truth segmentations Fine structures have small influence on overall pixel accuracy

  27. Generating Accurate Segmentations • Generated accurate segmentation of 27 images • 30 minutes per image Image (MSRC-21) Original Segmentation New Segmentation

  28. Relationship between Qualitative and Quantitative Results Pairwise CRF Higher order CRF Ground Truth Image (MSRC-21) Overall Pixel Accuracy 95.8% 98.7% Small changes in pixel accuracy can lead to large improvements in segmentation results.

  29. Quantitative Accuracy • Measure accuracy in labelling boundary pixels. • Accuracy evaluated in boundary bands of variable width Trimap (8-pixels) Trimap (16-pixels) Hand-labelled Segmentation Image (MSRC-21)

  30. Quantitative Accuracy • Measure accuracy in labelling boundary pixels. • Accuracy evaluated in boundary bands of variable width

  31. Conclusions • Method to enforce label consistency in image regions • Generalization of the commonly used Pairwise CRF model • Allows integration of pixel and region level features for image labelling problems

  32. Thanks

  33. Running Time Results Time (sec) Number of Segmentations Inconsistency Cost

  34. Qualitative Results (Contd..) Image (MSRC-21) PairwiseCRF Higher order CRF Ground Truth

  35. Transformation to second order functions c Auxiliary variables

More Related