Category Independent Region Proposals

1. Category Independent Region Proposals Ian Endres and Derek Hoiem University of Illinois at Urbana-Champaign

2. Finding Objects

3. Scanning Window Horse Dog Cat Car Train … 10,000+ windows

4. Category Independent Search ~100 regions

5. Finding Unfamiliar Objects

6. Finding Objects Objectives: • Minimize number of proposed regions • Maintain high recall of all objects • Provide detailed spatial support (i.e. segmentation)

7. Challenges • Objects extremely diverse • Variety of shapes, sizes • Many different appearances • Within object variation • Multiple materials and textures • Strong interior boundaries • Many objects in an image

8. Overview Generate Proposals: Maximize recall Rank Proposals: Small diverse set of object regions 1 2 3 4 ...

9. Generating Proposals 1. Select Seed 2. Compute affinities for seed 5. Change parameters Repeat 3. Construct binary CRF 4. Compute proposal + Unary term: Affinities Pairwise term: Occlusion Boundaries

10. Generating Seeds • Compute occlusion boundaries (Hoiem et al. ICCV ‘07) • Generate hierarchal segmentation • Incrementally merge regions of oversegmentation • Use regions with sufficient size and boundary strength • Avoids redundant or uninformative seeds

11. Region Affinity • Learned from pairs of regions belonging to an object • Computed between the seed and each region of the hierarchy • Features: color and texture similarity, boundary crossings, layout agreement

12. Color/Texture Similarity • Color, texture histograms for each region • Compute histogram intersection distance between two regions

13. Boundary Crossing • Draw line between region centers of mass • Compute strength of occlusion boundaries crossed

14. Layout Agreement • Predict object extent from each region • Compute strength of agreement between two regions

15. CRF Segmentation • Binary segmentation • Graph composition: • Nodes: Superpixels • Edges: Adjacent superpixels +

16. CRF Segmentation • Graph Potentials • Unary Potential: affinity values for each superpixel • Edge Potential: occlusion boundary strength • Parameters (25 combinations) • Node/Edge weight tradeoff • Node bias + Unary potential: Affinities Edge potential: Occlusion Boundaries

17. Ranking Proposals Generated Ranking Appearance scores 1. wT X1 wT X2 Sort scores 2. wT X3 3. wT X4 4.

18. Lacks Diversity • But in an image with many objects, one object may dominate 1 … 20 2 … 50 … 3 100 … 150 4

19. Encouraging Diversity • Suppress regions with high overlap with previous proposals … 1 20 2 … 3 50 4 … … 100 10

20. Ranking as Structured Prediction • Find the max scoring ordering of proposals • Greedily add proposals with best overall score Appearance score Overlap penalty Gives higher weight to higher ranked proposals Overall score

21. Learning to Rank(Max-margin Structured Learning) • Score of ground truth ordering (R(n)) should be greater than all other orderings (R): • Loss ( ) encourages good orderings: • Higher quality proposals should have higher rank • Each object should have a highly ranked proposal

22. Experimental Setup • Train on 200 BSDS images • Test 1: 100 BSDS images • Test 2: 512 Images from Pascal 2008 Seg. Val.

23. Evaluation • Region overlap • Recall at 50% region overlap • Typically more strict that 50% bounding box overlap • Measures detection quality and segment quality Ai Aj

24. Qualitative Results BSDS (Rank, % overlap) Pascal

25. Vs. Standard Segmentation Ours: 80% 180 proposals Standard: 80% 70,000 proposals (merge 2 adjacent regions) Standard: 53% 3000 proposals Ours: 53% 18 proposals

26. Recalling Pascal Categories

27. Future work • Object Discovery • Incorporate into detection systems • Label regions directly • Voting from proposed regions • Refine proposals with domain knowledge • i.e. wheel or head models