Indexing with Unknown Illumination and Pose

1. Indexing with Unknown Illumination and Pose Ira Kemelmacher Ronen Basri Weizmann Institute of Science

2. … Hippo Dino Camel The task – Shape Indexing 1. Recognize 2. Recover pose+lighting

3. Why is it hard? • Unknown pose • Unknown lighting • Clutter • Occlusion

4. Assumptions • Weak perspective projection • 3D rigid transformation • Lambertian model

5. Previous… • Identification using alignment Fischler and Bolles, Huttenlocher and Ullman • “3D to 2D invariants do not exist” Burns etal., Moses etal., Clemens etal. • Indexing faster than alignment Jacobs, Wolfson etal.

6. Previous Indexing Methods • Ignored intensity information • Need many point or line features • Restricted to polyhedral objects

7. Our algorithm… • Handles both pose and lighting • Uses intensities to filter out incorrect matches • Still relies on point features but only very few are needed • General objects

8. p2 p3 (α3 ,β3) P2 p4 P3 (α4 ,β4) p0 P4 p1 P0 P1 Indexing with pose - Affine model (Jacobs ‘96) pi = APi + t 8 DOF -> 5 points (n1,n2,m)

9. p2 p3 (α3 ,β3) P2 p4 P3 (α4 ,β4) p0 P4 p1 (n1,n2,m) P0 P1 Representation in two 2D tables α -space Offline – preprocessing Online – matching (α3,α4) α4 = α3m + n1 β -space β4 = β3m + n2 (β3, β4)

10. p2 p3 (α3 ,β3) P2 p4 P3 (α4 ,β4) p0 P4 p1 (n1,n2,m) P0 P1 Modifications – still two 2D spaces Offline – preprocessing Online – matching α –dual-space (θ,r1) r1= α3cos θ+ α4 sinθ β –dual-space (θ,r2) r2= β3cos θ+ β4 sinθ

11. p2 p3 (α3 ,β3) P2 p4 P3 (α4 ,β4) p0 P4 p1 (n1,n2,m) P0 P1 One 3D space Offline – preprocessing Online – matching (θ,r1,r2)

12. False Matches True match

13. How to eliminate the false matches • Enforce rigidity using inverse Gramian Test- Weinshall, ‘93 • Consistency with lighting  NEXT

14. Harmonic Images – Linear Basis for Lighting(Basri and Jacobs ‘01, Ramamoorthi and Hanrahan ‘01) r

15. Representation by harmonics Unknown light I = b * H Intensities of image point set Harmonics of model point set

16. The consistency measure I = b * H For corresponding image and model sets this is minimal Should we apply it on feature points?

17. “Smooth points” Smooth points Feature points

18. Voting • Sets of points that pass the lighting test vote for their respective model • All models receive scores: • Score = fraction of image sets for which the model appears min • Once model is selected its corresponding subsets used to determine its pose and lighting

19. Experiments • Real 3d objects acquired using laser scanner • Feature points collected automatically using Harris corner detector

20. dino hippo camel shark pinokio bear elephant face Results

21. dino hippo camel shark pinokio bear elephant face Results

22. Results

23. dino hippo camel shark pinokio bear elephant face Results – Indoor scene

24. dino camel shark pinokio bear elephant face hippo Results – Outdoor scene

25. dino camel shark pinokio bear elephant face hippo Results – Night Scene

26. dino camel shark pinokio bear elephant face hippo Results – Night Scene 2

27. Filtering out matches

28. How much does lighting help? Voting based on Affine model

29. How much does lighting help? Affine + Rigidity test

30. How much does lighting help? Affine + Rigidity + Lighting

31. Conclusion • Identify 3d objects in 2d scenes • Unknown pose, light • Clutter, occlusions • General, real objects • Fast, efficient • Combination of intensity cues and geometry

32. Thank you!