1 / 46

Omnidirectional Stereo Vision

CSc80000 Section 2 Spring 2005. Omnidirectional Stereo Vision. Lecture 8. Zhigang Zhu Computer Science Department The City College, CUNY zhu@cs.ccny.cuny.edu http://www-cs.engr.ccny.cuny.edu/~zhu/. Acknowledgements. Collaborators at UMass Edward Riseman Allen Hanson Deepak Karuppiah

jonescrystal
Download Presentation

Omnidirectional Stereo Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSc80000 Section 2 Spring 2005 Omnidirectional Stereo Vision Lecture 8 Zhigang Zhu Computer Science Department The City College, CUNY zhu@cs.ccny.cuny.edu http://www-cs.engr.ccny.cuny.edu/~zhu/

  2. Acknowledgements • Collaborators at UMass • Edward Riseman • Allen Hanson • Deepak Karuppiah • Howard Schultz • … … • Supported by • NSF Environmental Monitoring • DARPA/ITO Mobile Autonomous Robot S/W • China NSF Scene Modeling • Paper (with references) http://www-cs.engr.ccny.cuny.edu/~zhu/zOmniStereo01.pdf @Z. Zhu CCNY

  3. The Class of Omnistereo(omnidirectional stereo vision) • Omnidirectional Vision : How to look • Viewer-centered: outward looking • Object-centered: inward looking • Omnistereo Vision: How many viewpoints • Binocular/N-Ocular: a few (2 or more) fixed • Circular Projection: many inside a small area • Dynamic Omnistereo: a few, but configurable • Object-centered: many, in a large space @Z. Zhu CCNY

  4. Important Issues of Omnistereo • What this lecture is about • Omnistereo Imaging principle for sensor designs • Epipolar geometry for correspondence • Depth error characterization in both direction and distance • Other important issues not in this talk • Sensor designs • Calibration methods • Correspondence algorithms @Z. Zhu CCNY

  5. Omni Imaging & Representation • Omnidirectional (panoramic) Imaging • Catadioptric Camera (single effective viewpoint) • ParaVision by RemoteReality, PAL, and many… • Image Mosaicing • Rotating camera, translating camera, arbitrary motion • Omnidirectional Representation • Cylindrical Representation • Spherical Representation @Z. Zhu CCNY

  6. Panoramic Camera Panoramic Annular Lens (PAL) By Pal Greguss @Z. Zhu CCNY

  7. 360o 0o Panoramic Mosaics from a Rotating Camera (ICMCS99) @Z. Zhu CCNY

  8. 1st frame Cylindrical Panorama • connecting frame • conic mosaic • head-tail stitching • panorama @Z. Zhu CCNY

  9. P (X, Y, Z) Z D Y v f O X Cylindrical Projection Image projection (f, v) of a 3D point P (X,Y,Z) Distance Cylindrical image Vertical axis @Z. Zhu CCNY

  10. Binocular / N-Ocular Omnistereo A few fixed viewpoints • Three configurations • Horizontally-aligned binocular (H-Bi) omnistereo • Vertically-aligned binocular (V-Bi) omnistereo • N-ocular omnistereo – trinocular case • Issues • Distance error in directions of 360 degrees • Distance error versus distance • Epipolar geometry @Z. Zhu CCNY

  11. P (X, Y, Z) f D Z Y v2 v1 f1 f2 O1 X O2 B H-Bi Omnistereo: depth error From Image pair { (f1, v1), (f2, v2) } to a 3D point P (X,Y,Z) Triangulation - Fixed baseline B - Horizontal disparity (vergent angle) Depth Error • Depth accuracy is non-isotropic; max vergent only when f2 =90 • Not make full use of the 360 viewing • Depth error proportional to Depth2 / Baseline @Z. Zhu CCNY

  12. P (X,Y,Z) v1 v2 O1 O2 epipoles B D H-Bi Omnistereo: singularity case Zero Vergent angle when f1=f2=0 or 180 degree Distance Ratio Method - Visible Epipoles: the images of the camera centers in the others could be visible! - Vertical disparity and vertical epipolar lines @Z. Zhu CCNY

  13. triangulation singularity depth-blind spots v1 f1 180 360 0 H-Bi Omnistereo: Epipolar geometry Given point (f2, v2), search for (f1, v1) -The epipolar curves are sine curves in the non-singularity cases and - The epipolar lines are along the v direction in the singularity cases @Z. Zhu CCNY

  14. Z Y X O1 v1 Bv P v2 O2 V-Bi Omnistereo From Image pair { (f1, v1), (f2, v2) } to a 3D point P (X,Y,Z) - Vertical baseline Bv - Vertical disparity v - Same as perspective stereo • Depth accuracy isotropic in all directions • - Depth error proportional to square of distance • Epipolar lines are simply vertical lines • - But NO stereo viewing without 3D reconstruction @Z. Zhu CCNY

  15. R13 R23 O3 O1 O2 R12 R12 R23 R13 N-Ocular Omnistereo Why more viewpoints ? Every point of the 360 FOV from the center of the sensor-triangle can be covered by at least two pairs of rays from different cameras with good triangulations • depth accuracy is still not isotropic, but is more uniform in directions • - one pair of stereo match can be verified using the second pair • - However no gain in epipolar geometry @Z. Zhu CCNY

  16. Circular Projection Omnistereo Many viewpoints on a viewing circle • Omnivergent Stereo (Shum et al ICCV99) • every point in the scene is imaged from two cameras that are vergent on that point with maximum vergence angle; and • stereo recovery yields isotropic depth resolution in all directions. • Solution: Circular Projection/ Concentric Mosiacs • A single off-center rotating camera (Peleg CVPR 99, Shum ICCV99) • Full optical design (Peleg PAMI 2000) • My catadioptric omnistereo rig @Z. Zhu CCNY

  17. Case 1: an omni sensor Z O viewing circle Case 2: two 1D sensors Circular Projection: principle Many viewpoints on a viewing circle A virtual camera moving in a viewing circle captures two set of rays on a plane tangent to the viewing circle: the left-eye in clockwise direction, and the right-eye in counterclockwise direction @Z. Zhu CCNY

  18. f P left-eye ray right-eye ray f1 O1 D B r f2 O viewing circle O2 Circular Projection: geometry Max vergent angles for left and right rays “baseline” “disparity” P: 3D space point r: radius of the viewing circle f1,f2: viewing directions of left and right rays f: vergent angle (angular disparity) B: baseline length (< 2r); D: distance (OP) @Z. Zhu CCNY

  19. Circular Projection: properties • Depth estimation is isotropic • Same depth error in all directions • Make full use of the 360 viewing • Depth error proportional to depth2/baseline • Same as H-Bi Omnistereo • limited baseline (B < 2r) • Horizontal Epipolar lines • Superior than H-Bi Omnistereo when a single viewing circle for left and right omni-images • Extension to Concentric Mosaics with viewing circles of different radii? @Z. Zhu CCNY

  20. Case 2: an omni sensor Z Z O O viewing circle viewing circle Case 1: two 1D sensors Cameras: Single? Multiple? Standard? Special? Circular Projection: Implementation • Requirements: Two sets of rays 180o apart • Methods • 1: Two Rectilinear Cameras • 2: An Omnidirectional camera • Question: Can we do it with a single rectilinear camera? @Z. Zhu CCNY

  21. 2b left-eye ray right-eye ray O path of optical center V V image plane d rotation axis O viewing circle R Single camera approach Circular Projection: Implementation (I) • Rotate a rectilinear camera off its optical center • Take two columns with angular distance 2b << 180o • Viewing circle smaller than circular path of the optical center • Stretching your arm out, camera viewer may be too far from your eyes @Z. Zhu CCNY

  22. OL OR >2b right-eye ray left-eye ray 2g O path of two “virtual” cameras R mirror pair b d rotation axis (optical center) O Rv image plane viewing circle Catadioptric approach Circular Projection: Implementation (2) • Rotate a pair of mirror with a camera around its optical center • Look outward at the scene through two slit windows • Larger viewing circle since mirrors enlarge the viewing angle • Camera viewer right in front of your eyes @Z. Zhu CCNY

  23. Target Image 1 Image 2 Baseline Camera 1 Camera 2 Dynamic Omnistereoa few viewpoints moving freely (OmniVision2000) • Requirements: • Optimal configuration for any given point in the world • Change the vergent angle and the baseline freely • Issues: • Dynamic Calibration • View Planning @Z. Zhu CCNY

  24. rotation shift Dynamic Ominstereo: depth error • Question 1: Vergent angle • Max vergent angle (f2 = 90o) • Question 2: Baseline • The larger the better? • The error in estimating the baseline @Z. Zhu CCNY

  25. PAL 2 PAL 1 O2 cylinder body a B O1 Rc Dynamic Ominstereo: mutual calibration • Sensors as calibration targets • Make use of the visible epipoles • Known target geometry • Cylindrical body of the moving platform @Z. Zhu CCNY

  26. Mutual calibration and human tracking: an example Pano 1: Image of the 2nd robot Images of a person Pano 2: Image of the 1st robot Results: B = 180 cm, D1 = 359 cm, D2 = 208 cm @Z. Zhu CCNY

  27. Dynamic Ominstereo: Optimal view • Baseline error proportional to B2 • Larger baseline, even larger error • Overall distance error is min if • “Best” baseline and max vergent angle • Distance error with optimal configuration proportional to D1.5 @Z. Zhu CCNY

  28. T(1) O2(2) O2(1) T(2) rotation O1 shift Dynamic Ominstereo: Optimal view application • Track a single target by two robots • One stationary, one moving • Omnistereo head with reconfigurable vergent and baseline @Z. Zhu CCNY

  29. rotation shift Dynamic Ominstereo: error simulation • Student project in the spring of 2003 • Java Applet • http://www-cs.engr.ccny.cuny.edu/~zhu/omnistereo/simulation/ @Z. Zhu CCNY

  30. Comparisons • Four Cases • Fixed viewpoint omnistereo • One fixed, one circular projection • Both circular projection • Dynamic omnistereo • Java Interactive Simulations • http://www-cs.engr.ccny.cuny.edu/~zhu/omnistereo/errormaps/ @Z. Zhu CCNY

  31. Java Interactive Simulations • http://www-cs.engr.ccny.cuny.edu/~zhu/omnistereo/errormaps/ @Z. Zhu CCNY

  32. plane in-ward rotation object Earth translation outward rotation Modeling a building Object-Centered OmniStereo • Looking inward rather than Looking outward • Modeling objects rather than scenes • Many viewpoints over a large space Modeling the Earth @Z. Zhu CCNY

  33. “path” of optical center right-eye ray O 2b R d left-eye ray image plane rotation axis virtual viewing circle object Omni modeling of an object • Inward-Looking Rotation • Many viewpoints over a large circle • Circular projection: viewing circle within the object • Can rotate the (small) object (e.g. human) instead moving the camera @Z. Zhu CCNY

  34. plane Earth Omni modeling of the earth • Modeling the earth • Airplane flying along great circles • Taking the leading and trailing edge of each frame • Data amount: • 1017 pixels if 10 cm2/pixel • 1015 pixels if 1 m2/pixel • 1012 = 1 Tera = 1000 Giga • Modeling a small area • Rotation can be approximated as translation • Parallel-perspective stereo mosaics • Virtual flying through @Z. Zhu CCNY

  35. Sensor Image Plane “Right” Mosaic “Left” Mosaic Parallel-perspective stereo mosaics • Ideal model: Sensor motion is 1D translation, Nadir view • Two “virtual” Pushbroom cameras • Real Applications • Airborne camera (Umass, Sarnoff..) • Ground Vehicles (Tsinghua, Osaka) @Z. Zhu CCNY

  36. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  37. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  38. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  39. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  40. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  41. Re-Organizing the images…. Stereo pair with large FOVs and adaptive baselines @Z. Zhu CCNY

  42. GPS/IMU Height H from Laser Profiler P(X,Y,Z) Two views from different perspective stereo Recovering Depth from Mosaics • Parallel-perspective stereo mosaics • Depth accuracy independent of depth (in theory) Adaptive baseline displacement disparity Fixed ! @Z. Zhu CCNY

  43. Stereo mosaics of Amazon rain forest Left Mosaic • 166-frame telephoto video sequence -> 7056*944 mosaics Right Mosaic Depth Map @Z. Zhu CCNY

  44. Stereo viewing • Red: Right view; Blue/Green: Left view @Z. Zhu CCNY

  45. Accuracy of 3D from stereo mosaics(ICCV01, VideoReg01) • Adaptive baselines and fixed disparity -uniform depth resolution in theory and accuracy proportional to depth in practice • 3D recovery accuracy of parallel-perspective stereo mosaics is comparable to that of a perspective stereo with an optimal baseline @Z. Zhu CCNY

  46. Conclusions @Z. Zhu CCNY

More Related