On Evaluating Methods for Recovering Image Curve Fragments

1. On Evaluating Methods for Recovering Image Curve Fragments YuliangGuo and Benjamin Kimia 33yuliangguo@gmail.comkimia@lems.brown.edu School of Engineering Brown University Presentation at POCV2012 06/16/2012

2. Background Edges, Curves, Segmentations have been used as intermediary information in a lot of applications While edges are useful in some applications, curve fragments are more appropriate in other applications

3. Object Recognition (Zhu and Shi, CVPR2010) Curve fragments are more descriptive than edges revealing object identity

4. 3D Object Detection and Pose Estimation (Payet and Todorovic, ICCV2011) Curve Fragments are more descriptive to estimate 3D Pose

5. Multi-view Reconstruction (Fabbri, Kimia. CVPR2010) Matching edges has more ambiguity then matching curve fragments

6. Tracking, Motion-based segmentation (Jain, Kimia, Mundy. CVIU2007) Matching curve fragments has less ambiguity then matching edges

7. Current Existing Curve Fragment Extraction Algorithms • Organized set of edges are more informative and discriminative because of the represented geometry

8. Problem: How to Evaluate the Quality of a Set of Curve Fragments Original Image Curve Fragments Set from Algo1 Curve Fragments Set from Algo2

9. What is Missing? Methodology for evaluating curve fragments rather than edges Ground Truth Dataset : Grouping of edges should be available

10. Challenge: Cost of Deformation between Two Sets of Curve Fragments Random combination of curve fragments need to be explored in searching for the optimal match Cases of Continuous Deformation are Easy Cases of Discontinuous Deformation are Hard

11. Ideas from Edit Distance • Levenshtein’sEdit Distance is applied in string matching. • String Edits: Add, Delete, Replace • Helovs Hello • Jellovs Hello • Edits in Curve Fragments Matching

12. Transformation between Curve Fragment Sets using Edits Delete Merge Split Deform Deform Insert Transform from Red to Green Merge Delete Deform Deform Merge Delete Transform from both side to the Middle state

13. Approach: Find the optimal Path Between Two Sets of Curve Fragments Shortest Path Each Node in this figure represents a full set of curve fragments

14. Necessary Components The cost of deformation: DE +FG +HI • Search for the optimized path: Edit Distance • Neighborhood Structure: Transitions/Edits • Cost: Length of deformation during discontinuous operations • Assumption: Curves arise from various groups of edges in a scene have no instance-type variations • Apply Chamfer’s distance in matching edges. Matching edges within a threshold does not lead to penalize • Only penalize length difference between curve fragment sets

15. More Efficient Search Decompose the search space into a set of independent Corresponding Islands of curve fragments

16. Algorithm: Generating Corresponding Islands Red and Green Curve Fragmentsare from two sets but close to overlap each other Number of Curve Fragments in an island

17. Example of Corresponding Islands Curve Fragments Set 1 Curve Fragments Set 2 Corresponding Islands

18. Algorithm: Search Optimal Match Using Edit Distance within Each Corresponding Islands Miss in Algo 1 Optimal Match Edit Distance Matrix Problem Translate: iteratively search the Optimal Matched combinations from set1 and set2

19. Optimized Matching Between Two Sets of Curve Fragments: Edit Distance Optimized Matching Curve Fragments Missing Curve Fragments Extra Curve Fragments

20. Evaluation: Ground Truth Datasets Evaluation of contour detectors (left) and segmentation algorithms (right) on BSDS Contour Detection and Hierarchical Image Segmentation , Arbelaez, Maire, Fowlkes, Malik, PAMI10 • The BSDS & Evaluation Strategy is widely used in evaluating segmentation & edge detection algorithm • Prior to this, algorithms are compared on ad-hoc basis • However, it has serious drawback applying it to the evaluation of curve fragments

21. 1. Lack of Grouping Information Binary Boundary Map • P. Felzenszwalb and D. Mcallester. A min-cover approach for finding salient curves. In CVPRW’ 06 • Q. Zhu, G. Song and J. Shi. Untangling Cycles for Contour Grouping. In ICCV’ 07 Possible Groupings of Curve Fragments

22. 2. Open Contours not Represented in BSDS • BSDS instructs subjects to mark regions which only generate closed contours • Not able to represent open contours • Disallows internal contours in the dataset

23. 3. Semantic Contours Only Original Image BSDS GT • BSDS methodology instructs subjects to divide image into meaningful segments, representing things or parts of things. This Rules out: • Reflection contours like zebra patterns, • Shading contours (Zhu & Shi, CPVR’ 06). • More Importantly, it subjects contour detection to results of object recognition, such that we can not evaluate contour fragments as a bottom up process

24. Construction of Curve Fragment Ground Truth Dataset (CFGD) Divide into sub-images, present them to objects in random order to minimize the effect of context and recognition

25. Comparing BSDS to CFGD Original Image BSDS ground truth CFGD image

26. Generation of Precision-Recall Curves Total Length of Matched Curve Fragments in Ground Truth Total Length of Curve Fragments in Ground Truth Total Length of Matched Curve Fragments from Algorithm Total Length of Curve Fragments From Algorithm Use edit distance to separate matched, miss and extra curve fragments Recall = Precision = The latent PR curve parameter is length or total contour contrast, which ever is more appropriate to the specific algorithm

27. Evaluation Results on CFGD

28. Conclusion • We develop a novel methodology to compare two sets of curve fragments, considering discontinuous deformation as well as continuous deformation • We collected a new annotation of Curve Fragment Ground-Truth Dataset (CFGD) which is more suitable for evaluating curve extraction algorithms • Downloadable from http://vision.lems.brown.edu/datasets/cfgd

29. Thank You

30. Appendix A: Evaluation of edge detection on BSDS vs. CFGD Third-Order Edge Detection Evaluation on BSDS Evaluation on CFGD

31. Appendix B: Evaluate Retrained Pb and Third-Order Edge Detection on CFGD Fine Scale Annotations Coarse Scale Annotations

32. Quotes from Prior Works • “Although our results on this dataset are reasonably good, we do not feel that the metric used in this experiment is an appropriate measure of performance for our algorithm. The metric depends only on which pixels are marked as edges. Therefore it does not check that the edges were grouped into meaningful curves. Construction of an appropriate empirical metric for multiple curve detection remains an open problem.” • P. Felzenszwalb and D. Mcallester. A min-cover approach for finding salient curves. In CVPRW’ 06 • “Many of the false positives are shading edges, which are not labeled by humans. How- ever, once they are grouped, they could be easy to prune in later recognition process. These are the advantages not reflected by the metric in the Berkeley benchmark, which counts matched pixels independently.” • Q. Zhu, G. Song and J. Shi. Untangling Cycles for Contour Grouping. In ICCV’ 07