Recovering Human Body Configurations using Pairwise Constraints between Parts

1. Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra Malik 1

2. Finding People Challenges: Pose, Clothing, Lighting, Clutter, … 2

3. Previous Work • Related Domains • Tracking People • Detecting Pedestrians • ... … • Localizing Human Figures • Exemplar-based: [Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], … • Part-based: [Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], … • … … 3

4. Beyond “Trees” ? • A hard problem! More information is needed. • Important cues that are NOT in the tree model: • Symmetry of clothing/color • “V-shape” formed by the upper legs • Distance/smooth connection between arms and legs • …… 4

5. Our Approach • Preprocessing with Constrained Delaunay Triangulation • Detecting Candidate Parts from Bottom-up • Learning Pairwise Constraints between Parts • Assembling Parts by Integer Quadratic Programming (IQP) 5

6. Constrained Delaunay Triangulation • Detect edges with Pb (Probability of Boundary) • Trace contours with Canny’s hysteresis • Recursively split contours into piecewise straight lines • Complete the partial graph with Constrained Delaunay Triangulation 6

7. Detecting Parts using Parallelism N C1 (L1,1) C2 (L2,2) T • Candidate parts as parallel line segments (Ebenbreite) • (Scale-invariant) Features for parallelism: |Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2| • Logistic Classifier 7

8. C1 C2 Pairwise Constraints between Parts • Scale (width) consistency • Use anthropometric data as groundtruth • Symmetry of appearance (color) • Orientation consistency • Connectivity • Short distance between adjacent parts • “Smooth” connection between non-adjacent parts • short “gaps” on shortest path (on CDT graph) • small maximum angle on the shortest path • few T-junctions/turns on the shortest path 8

9. Learning Pairwise Constraints 15 hand-labeled images from a skating sequence Empirical distributions of some pairwise features For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian) 9

10. Assembling Parts as Assignment (Lj1,Ci1=(Lj1)) (Lj2,Ci2=(Lj2)) Candidates {Ci} Parts {Lj} assignment  Cost for a partial assignment {(Lj1,Ci1), (Lj2,Ci2)}: 10

11. Assignment by IQP Q(x)=xTHx • Suppose there are m parts and n candidates, the optimal assignment  minimizes a quadratic function where x is a mn1 indicator vector and H is of size mnmn. • This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions. • We choose an approximation scheme which solves mn linear programs followed by gradient descent. • The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems. • We have m=9 and n~150; the total time is less than a minute. 11

12. 12

13. 13

14. 14

15. Conclusion • To find people under general conditions, we need to go beyond the traditional tree-based model; • Most important constraints for the human body are between pairs of body parts; • Pairwise constraints may be learned from a small set of training examples; • Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints. 15

16. 16