Shape-based Object Recognition

1. Shape-based Object Recognition Thomas B. Sebastian Brown University

2. Summary • Shape recognition is important but challenging task • Three main contributions to computer vision: • Robust shape recognition algorithm for matching shock graphs • Relies on edit distance between shock graphs • Gives excellent recognition rates • Useful in indexing large shape databases • Generic curve matching algorithm with several applications • Effective segmentation method, which is useful in several medical applications

3. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

4. Introduction • Recognition is a challenging problem because it needs to integrate multiple cues like color, texture and shape • Shape is an important cue • However, shape is least studied, and is difficult to extract, characterize and represent

5. Example 1: View-point variations Robustness to transformations Recognition algorithms have to overcome several visual transformation challenges

6. Robustness to transformations (cont.) Example 2: Partial occlusion

7. Robustness to transformations (cont.) Example 3: Articulation

8. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

9. Real Example Shock graph representation of shapes • Shocks (or medial axis or skeleton) are locus of centers of maximal circles that are bitangent to shape boundary Shape boundary Shocks

10. Distance between shapes • Distance between shapes is important for recognition • Shape space is collection of all shapes • Shape is a point • Shape deformation sequence is a path • Cost of optimal deformation sequence is distance from A to B

11. Deformation paths • Number of deformation paths between two shapes is infinite • Finding the optimal deformation path is intractable • Three measures are employed in our edit-distance algorithm to make the problem tractable • Avoidance of complexity-increasing deformation paths • Partitioning of shape space using shock graph topology • Discretization of deformation paths

12. Avoidance of complexity-increasing paths • Complexity-increasing shape deformation paths are not optimal • Represent a deformation path by a pair of simplifying deformation paths from A, B to a simpler shape C

13. Changes in shock graph topology • Shock graph topology is unaltered for most shape deformations • At transition shapes small changes lead to abrupt changes in the shock graph topology

14. Partitioning of shape space • Shape cell: Collection of shapes with same shock graph topology • Transition shapes form the boundary between shape cells Shape cell 2 Shape cell 1

15. Shape deformation bundle:Collection of deformation paths passing through same set of shape cells • Represented by set of transition shapes it passes through Discretization of deformation paths

16. Deformation sequence as graph edits • Shock graph edit operation transforms a shape to adjacent transition shape

17. Edit-distance algorithm • Optimal deformation sequence is found using adynamic-programmingbasedpolynomial-time edit-distance algorithm [SODA 99, 01] • Proposed for comparing character strings [Wagner, Fischer] • Extended to matching rooted trees [Tai, Zhang, Shasha] • Employed in traditional edit-distance operations (i) delete, (ii) re-label

18. Splice: deletes a shock branch and merges the remaining two • Contract:deletes a shock branch between degree-three nodes • Merge:combines two branches at a degree-two node Edit-distance for shock graphs • Four edit operations are needed for shock graphs • Deform:changes the attributes of a shock branch

19. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

20. Assigning costs to edit operations • Edit costs must be • Consistent with each other • Consistent with our perceptual metrics of similarity • Our approach is twofold • Define the cost of deform edit • Define cost of other edits as limits of deform cost

21. Deform cost as joint-curve matching • Deform cost relies on extending a metric developed to compare curves [MMBIA00, IWVF01] • Each shock edge corresponds to two boundary segments • Matching shock edges can be viewed as joint-curve matching

22. Curve matching • Goal is to find • Optimal alignment (pairing of points in C, C’) • Distance (cost to deform C to C’)

23. Cost of deforming curves • Assumption: Cost of matching curves is equal to the sum of costs of matching infinitesimal sub-segments • Cost of infinitesimal sub-segments is measured in terms of length and curvaturedifferences

24. Representation of alignment • Alignment is a pairing of points of C, C’ • Treat curves C and C’ as the x and y-axes • Alignment is then a curve in 2D space, called alignment curve

25. Template consisting of nine incoming curves (at different slopes) is used to update the cost • Optimal deformation path is found by back-tracking Finding optimal alignment curve • Use dynamic programming to find optimal alignment curve • Finding optimal alignment curve corresponds to a shortest-path problem

26. Deform cost between shock edges • Deform cost between shock edges consists of • Length differences of the boundary segments • Curvature differences of boundary segments • Difference in width of shape • Difference in relative orientation of boundary segments

27. Edit-distance between shapes • Edit-distance is defined as the sum of the cost of edits in optimal edit sequence

28. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

29. Matching results* Edit-distance algorithm gives intuitive results * Same colors indicate matching edges; gray-colored edges are pruned

30. Boundary noise • Shock graphs are sensitive to boundary noise In optimal edit sequence “noisy” branches are pruned

31. Articulation • Shock graphs represents object parts and part hierarchy Edit-distance is robust in presence of part-based changes

32. Viewpoint variation Smooth change • Deform edit handles smooth changes • Splice and contract edits handle abrupt changes Abrupt change

33. Partial occlusion Edit-distance is robust to partial occlusion

34. Indexing into shape databases* Edit-distance algorithm allows 100% shape recognition between different shape categories * Results duplicated in two databases: 99 shapes and 216 shapes

35. Indexing Results It also allows nearly 100% recognition between shapes in the same shape category

36. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

37. Solutions • Develop a coarse-scale version of match • Use exemplars to represent shape categories Challenges • Shock graph matching is computationally expensive • Each match takes 2-3 minutes • Querying 1000 shape database takes >2000 minutes (35 hours)

38. Compute edit costs using nodes in shock graph • Represent each shock link by its end points • Significant speed-up in computation time • Coarse-scale matching 2-3 seconds • Shock graph edit distance 2-3 minutes Coarse-scale shock graph matching

39. Coarse-scale matching results Top matches contain most of shapes from correct category

40. Coarse vs. fine matching results • Two-stage indexing strategy • Stage 1: Coarse-scale matching to eliminate distant shapes • Stage 2: Fine-scale matching to refine the top matches Coarse Fine Coarse - Top 158 matches have 54 fish shapes Fine - Top 61 matches have 54 fish shapes

41. Exemplars for shape categories • Comparing query with all shapes is unnecessary • If query is “distant” from a fish, it will be distant from all fish • In context of all shapes fish will be tightly clustered Each category is represented by a few exemplars

42. Exemplar-based indexing results • Indexing results are excellent using a large database (1032 shapes, 200 exemplars) • Correct category is selected in top 1, 2 and 5 matches with 78%, 91%, and 99% success rate respectively

43. Number of exemplars • Using fewer exemplars suggest a hierarchical representation • 2-3 primary exemplars rule out 75% of categories • Additional 2-3 auxiliary exemplars rule out another 50% of categories

44. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching

45. Motivation • Traditional deformable models have convergence problems

46. Skeletally coupled deformable model (SCDM) • The underlying principle of skeletally coupled deformable model is use of skeleton between growing seeds to modulate their growth • Growth of seeds is governed by forces at skeletal point and competing boundary

47. Convergence of SCDM • SCDM solves the convergence problem of traditional deformable models

48. Carpal Bone Segmentation Results Results are clinically meaningful

49. 3D Model of Carpal Bones 3D model is useful in studying carpal kinematics, creating computational atlases etc.

50. Contents • Introduction • Shape recognition using shock graphs • Edit-distance algorithm • Edit-costs using curve matching • Results • Indexing into large shape databases • Segmentation using skeletally coupled deformable model • Additional applications of curve matching