Comparison of stereovision odometry approaches

1. Comparison of stereovision odometry approaches Niko Suenderhauf Chemnitz University of Technology Germany Kurt Konolidge SRI International USA Thomas Lemaire, Simon Lacroix Robotics and AI group LAAS/CNRS, Toulouse (at Laas 10/04  03/05) (at Laas (08/04  03/05)

2. “Reach that goal”, “Map this area”… Missions are defined in terms of localization Environment models are required Spatial consistency ensured by localization Safe execution of the planned trajectories Robust control ensured by localization On the importance of localization “If you are not localized, you are lost !”

3. Outline • Principle of stereovision odometry • “Dead reckoning” approach • More global approaches • Conclusions

4. 1. Stereovision 5. Motion estimation 3. Pixels tracking 2. Pixels selection 4.Stereovision Principle of stereovision odometry

5. Principle of stereovision odometry • A lot of contributions now in the robotics literature • [Mallet-Lacroix-2000] • [Olson-Matthies-2000] - cf Matthies in the 80’s • [Corke-2004] • … • Related approaches • “scan matching” approaches - without image feature associations (e.g. [Zhang-1992]) • [Kim-ICRA-2005] - without stereo correspondences • Three functionalities involved • Stereovision (sparse or dense) • Image feature association • Pose estimation

6. Feature tracking or feature matching ? • Feature tracking • Close images (high spatial rate) • Aiding sensor (to focus the search) • No feature selection necessarily required • Feature matching • Feature extraction / selection (or  ?) • A bit more time • Work for almost any motion - no estimate necessary

7. Feature matching • Features : Harris precise detector [Schmid-ICCV-1998] • Interest point matching [Jung-ICCV-2001]

8. Feature matching • Features : Harris precise detector [Schmid-ICCV-1998] • Interest point matching [Jung-ICCV-2001] Detected points Matched points An other example

9. Feature matching 1.5 scale change 3.0 scale change • Features : Harris precise detector [Schmid-ICCV-1998] • Interest point matching [Jung-ICCV-2001]

10. Std dev. on disparities (here with ZNCC along epipolar) • Relation between correlation curve and d : Error model : f ( c ) s = d Error models On stereovision : empical analysis (cf [Matthies-1992])

11. Correlation surface Gaussian distribution Error models On interest point matching : “gaussian fitting” model Correlation surface locally computed around the matches (ZNCC score) Validity of such a model ? Don’t we miss a proportional factor ?

12. Outline • Principle of stereovision odometry • Feature matching • Error models • “Dead reckoning” approach • More global approaches • Conclusions

13. Fairly good precision (up to 1% on 100m trajectories) Dead reckoning approach • Relative t+1 / t poses computed with “constrained” least square minimisation (e.g. [Haralick-1989]) • Simple iterative outlier rejection algorithm (no RANSAC required)

15. Dead reckoning error • Propagating the uncertainty of 3D matching points set to optimal motion estimate [Haralick-1994] • - 3D matching points set • - Optimal motion estimate • - Cost function • Covariance of the random perturbation u : propagation using Taylor series expansion of the Jacobian of the cost function • around

16. Outline • Principle of stereovision odometry • Feature matching • Error models • “Dead reckoning” approach • More global approaches • Conclusions

17. : 3D points : image coordinates Bundle adjustment approach • Classic way to solve the “structure from motion” problem in computer vision • Non linear-minimization provides a MLE (up to a scale parameter) • Can also optimize camera parameters (11 d.o.f. in Pi) • n points, m poses : 3n + 6m parameters… • Better have good initial estimates ! • Sparse bundle adjustment [Hartley-2004]

18. Sparse bundle adjustment with stereo • “Simply” add second image pixels/poses in the function to minimize • Naïve outlier rejection procedure costly (better use RANSAC ?) • Various possibilities : • Used in a “dead reckoning” way • Used on a fixed size of images (or within a given distance) : “sliding window approach” • Global optimization : “Full SBA”

19. Full SBA « local » SBA’s SBA with stereo : indoor data set

20. D-CP-GPS Full SBA « local» SBA’s SBA with stereo : outdoor data set

21. SBA with stereo : conclusions • Full SBA simply not tractable (batch, required 4.5 min CPU time on the outdoor data set) • Sliding window SBA seems better than dead-reckoning approach • Nb of images of 3-4 seems enough

22. General SLAM operations “Stereo-based” SLAM operations • Landmark detection • Relative observations (measures) • Of the landmark positions • Of the robot motions • Observation associations • Refinement of the landmark and robot positions • Vision : interest points • Stereovision • Visual motion estimation / INS / Odo • Interest points matching • Extended Kalman filter EKF-SLAM approach • « Local memory » SLAM : forget landmarks that disappear • Can be run in « real time » • Can incorporate any aiding sensor • Various « forget strategies » can be defined

23. By the way, between images 31 and 32 : Local EKF-SLAM approach : data set • Along a 60m loop trajectory : • 100 stereo pairs • Looking inwards

24. Local EKF-SLAM approach : results

25. Local EKF-SLAM approach : results

26. (Full EKF-SLAM approach : results) landmark uncertainty ellipses (x5)

27. (Full EKF-SLAM approach : results)

28. (((( Full EKF-SLAM approach : results ))))

29. Outline • Principle of stereovision odometry • Feature matching • Error models • “Dead reckoning” approach • More global approaches • SBA-based approach • EKF-SLAM approach • Conclusions ?

30. Panoramic cameras !!! (not even stereo ? cf “visual-SLAM” recent results, “view-based” localisation… Conclusions • A vast number of “parameters” to check/assess • Algorithmic parameters : • Kind of matching algorithm (stereo and motion matches) • Feature definition and selection • Estimation • Dead reckoning • SBA approaches • SLAM approaches • System parameters : • Image size • Focal length • Stereovision baseline and height • Bench orientation (forward, sidewards, downwards)

31. Journal of Field Robotics Editor-In-Chief Sanjiv Singh, Carnegie Mellon Editorial Board Robert Ambrose, NASA JSC Greg Baiden, Laurentian Univ. Martin Buehler, Boston Dynamics Raja Chatila, LAAS Peter Corke, CSIRO Eric Feron, MIT Ernie Hall, Univ. of Cincinnati Alonzo Kelly, CMU Larry Matthies, NASA JPL Eduardo Nebot, Univ. of Sydney Simon LaCroix, LAAS Annibal Ollero, Univ. of Seville Vincent Rigaud, IFREMER Jonathan Roberts, CSIRO David Wettergreen, CMU Ron Arkin, Georgia Tech, Alberto Broggi, Univ. of Parma Aarne Halme, HUT Peter Lawrence, Univ. of British Columbia David Nister, Univ. of Kentucky John Reid, John Deere Mirek Skibinewski, Purdue James Trevelyan, Univ of Western Australia Tony Stentz, CMU Brian Wilcox, NASA JPL Kazuya Yoshida, Tohoku Univ. The Journal of Field Robotics seeks to promote rapid dissemination of important research results in robotics for unstructured and dynamic environments.  Articles describing robotics research with applications to the environment, construction, forestry, agriculture, ,mining, subsea, intelligent highways, search and rescue, military, and space (orbital and planetary) are encouraged. Articles in sensing, sensors, mechanical design, computing architectures, communication, planning, learning, and control, applied to field applications are encouraged. The first issue is expected to be available in January 2006. Further Details: http://www.ri.cmu.edu/~jfr