1 / 52

Graph Cut based Inference with Co-occurrence Statistics

Graph Cut based Inference with Co-occurrence Statistics. Ľ ubor Ladick ý , Chris Russell, Pushmeet Kohli, Philip Torr. Image labelling Problems. Assign a label to each image pixel. Image Denoising. Object Segmentation. Geometry Estimation. Sky. Building. Tree. Grass.

lada
Download Presentation

Graph Cut based Inference with Co-occurrence Statistics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Graph Cut based Inference with Co-occurrence Statistics Ľubor Ladický, Chris Russell, Pushmeet Kohli, Philip Torr

  2. Image labelling Problems Assign a label to each image pixel Image Denoising Object Segmentation Geometry Estimation Sky Building Tree Grass

  3. Pairwise CRF models Standard CRF Energy Data term Smoothness term

  4. Pairwise CRF models Standard CRF Energy Data term Smoothness term Restricted expressive power

  5. Structures in CRF • Taskar et al. 02 –associative potentials • Kohli et al. 08 – segment consistency • Woodford et al. 08 – planarity constraint • Vicente et al. 08 – connectivity constraint • Nowozin & Lampert 09 –connectivity constraint • Roth & Black 09 – field of experts • Ladický et al. 09 – consistency over several scales • Woodford et al. 09 –marginal probability • Delong et al. 10 – label occurrence costs

  6. Pairwise CRF models Standard CRF Energy for Object Segmentation Local context Cannot encode global consistency of labels!!

  7. Detection Suppression If we have 1000 categories (detectors), and each detector produces 1 fp every 10 images, we will have 100 false alarms per image… pretty much garbage… [Torralba et al. 10, Leibe & Schiele 09, Barinova et al. 10] Image from Torralba et al. 10

  8. Encoding Co-occurrence [Heitz et al. '08] [Rabinovich et al. ‘07] Co-occurrence is a powerful cue • Thing – Thing • Stuff - Stuff • Stuff - Thing [ Images from Rabinovich et al. 07 ]

  9. Encoding Co-occurrence [Heitz et al. '08] [Rabinovich et al. ‘07] Co-occurrence is a powerful cue • Thing – Thing • Stuff - Stuff • Stuff - Thing Proposed solutions : 1. Csurka et al. 08 - Hard decision for label estimation 2. Torralba et al. 03 - GIST based unary potential 3. Rabinovich et al. 07 - Full-connected CRF [ Images from Rabinovich et al. 07 ]

  10. So... What properties should these global co-occurence potentials have ?

  11. Desired properties 1. No hard decisions

  12. Desired properties 1. No hard decisions Incorporation in probabilistic framework Unlikely possibilities are not completely ruled out

  13. Desired properties 1. No hard decisions 2. Invariance to region size

  14. Desired properties 1. No hard decisions 2. Invariance to region size Cost for occurrence of {people, house, road etc .. } invariant to image area

  15. Desired properties 1. No hard decisions 2. Invariance to region size The only possible solution : L(x)={ , , } Global context Local context Cost defined over the assigned labels L(x)

  16. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred  L(x)={ aeroplane, tree, flower, building, boat, grass, sky }  L(x)={ building, tree, grass, sky }

  17. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency

  18. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency a) Memory requirements as O(n) with the image size and number or labels b) Inference tractable

  19. Previous work • Torralba et al.(2003) – Gist-based unary potentials • Rabinovich et al.(2007) - complete pairwise graphs • Csurka et al.(2008) - hard estimation of labels present

  20. Related work • Zhu & Yuille 1996 – MDL prior • Bleyer et al. 2010 – Surface Stereo MDL prior • Hoiem et al. 2007 – 3D Layout CRF MDL Prior • Delong et al. 2010 – label occurence cost C(x) = K |L(x)| C(x) = ΣLKLδL(x)

  21. Related work • Zhu & Yuille 1996 – MDL prior • Bleyer et al. 2010 – Surface Stereo MDL prior • Hoiem et al. 2007 – 3D Layout CRF MDL Prior • Delong et al. 2010 – label occurence cost C(x) = K |L(x)| C(x) = ΣLKLδL(x) All special cases of our model

  22. Inference Pairwise CRF Energy

  23. Inference IP formulation (Schlesinger 73)

  24. Inference Pairwise CRF Energy with co-occurence

  25. Inference IP formulation with co-occurence

  26. Inference IP formulation with co-occurence Pairwise CRF cost Pairwise CRF constaints

  27. Inference IP formulation with co-occurence Co-occurence cost

  28. Inference IP formulation with co-occurence Inclusion constraints

  29. Inference IP formulation with co-occurence Exclusion constraints

  30. Inference LP relaxation Relaxed constraints

  31. Inference LP relaxation Very Slow! 80 x 50 subsampled image takes 20 minutes

  32. Inference: Our Contribution Pairwise representation • One auxiliary variable Z 2 L • Infinite pairwise costs if xi Z [see technical report] *Solvable using standard methods: BP, TRW etc.

  33. Inference: Our Contribution Pairwise representation • One auxiliary variable Z 2 L • Infinite pairwise costs if xi Z [see technical report] *Solvable using standard methods: BP, TRW etc. Relatively faster but still computationally expensive!

  34. Inference using Moves Graph Cut based move making algorithms [Boykov et al. 01] • Series of locally optimal moves • Each move reduces energy • Optimal move by minimizing submodular function Current Solution Search Neighbourhood N Move Space (t) : 2N Number of Variables L Number of Labels Space of Solutions (x) : LN α-expansion transformation function

  35. Inference using Moves Graph Cut based move making algorithms [Boykov, Veksler, Zabih. 01] α-expansion transformation function

  36. Inference using Moves Co-occurence representation Label indicator functions

  37. Inference using Moves Cost of current label set Move Energy

  38. Inference using Moves Move Energy Decomposition to α-dependent and α-independent part α-independent α-dependent

  39. Inference using Moves Move Energy Decomposition to α-dependent and α-independent part Either α or all labels in the image after the move

  40. Inference using Moves Move Energy submodular non-submodular

  41. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) • E'(t) = E(t) for current solution • E'(t) E(t) for any other labelling

  42. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) • E'(t) = E(t) for current solution • E'(t) E(t) for any other labelling Occurrence - tight

  43. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) • E'(t) = E(t) for current solution • E'(t) E(t) for any other labelling Co-occurrence overestimation

  44. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) • E'(t) = E(t) for current solution • E'(t) E(t) for any other labelling General case [See the paper]

  45. Inference Move Energy non-submodular Non-submodular energy overestimated by E'(t) • E'(t) = E(t) for current solution • E'(t) E(t) for any other labelling Quadratic representation

  46. Application: Object Segmentation Label based Costs Standard MRF model for Object Segmentation Cost defined over the assigned labels L(x)

  47. Training of label based potentials Label set costs Approximated by 2nd order representation Indicator variables for occurrence of each label

  48. Experiments • Methods • Segment CRF • Segment CRF + Co-occurrence Potential • Associative HCRF [Ladický et al. ‘09] • Associative HCRF + Co-occurrence Potential • Datasets MSRC-21 • Number of Images: 591 • Number of Classes: 21 • Training Set: 50% • Test Set: 50% PASCAL VOC 2009 • Number of Images: 1499 • Number of Classes: 21 • Training Set: 50% • Test Set: 50%

  49. MSRC - Qualitative

  50. VOC 2010-Qualitative

More Related