1 / 31

Laboratorio di M acchine I ntelligenti per il riconoscimento di V ideo, I mmagini e A udio

DIIIE - University of Salerno INSA - Lyon. Stereo Vision for Obstacle Detection: a Graph-Based Approach. P. Foggia, Jean-Michel Jolion, A. Limongiello , M. Vento 6th IAPR – TC-15 Workshop on Graph-based Representations in Pattern Recognition (GbR 2007) Alicante June 11-13, 2007.

turi
Download Presentation

Laboratorio di M acchine I ntelligenti per il riconoscimento di V ideo, I mmagini e A udio

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DIIIE - University of Salerno INSA - Lyon Stereo Vision for Obstacle Detection: a Graph-Based Approach P. Foggia, Jean-Michel Jolion,A.Limongiello, M. Vento 6th IAPR – TC-15 Workshop onGraph-based Representationsin Pattern Recognition(GbR 2007) Alicante June 11-13, 2007 Laboratorio diMacchineIntelligenti per il riconoscimento di Video,Immagini eAudio

  2. Obstacle Detection Automatic analysis of a dynamic video streaming acquired from a Mobile Platform for obstacle detection in unstructured environment Why an hard task: • mechanical vibrations of the cameras • light changing • no information about the environment • no information about the obstacles • low execution time Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  3. Obstacle Detection Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  4. Obstacle Detection Obstacle Detection is possible after a good representation of the environment A good representation must be related to our goal • To have a 3D representation of the scene we consider the Stereo Vision paradigm [3,4] • We can obtain information on the deepness of the pixels starting from two different views of the same scene Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  5. Outline • Related works: • A comparison • Open problems • Our approach: • The rationale • The Algorithm • Results • Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  6. Related Works Related Works Traditionally, the researches have applied 3D reconstruction approaches in the Autonomous Navigation framework, based on a punctual matching between the image pair For Obstacle detection it is not very important to have a good reconstruction of the surfaces, but it is important to identify adequately the space occupied by each object The Rationale The Algorithm • A good taxonomy has been done by Scharstein and Szeliski (IJCV 2002) [3] and by Cha Zhang (2002) [4] • Selected Algorithms: • Sum of Squared Differences (SSD) • Dynamic Programming (DP) • Graph Cut (GC) • Dense Features (DF) Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  7. Related Works Related Works • Feature-based approaches provide a correspondence between feature-points (as corners, edges, etc.) • Recently, they have been disregarded because they produce a sparse depth map only for the feature points, that is not so much useful in real applications • They are normally fast and enough stabile in real contexts The Rationale The Algorithm • Area-based techniques provide a correspondence between each point of the image pair, so they produce a dense depth map not right necessary in AMR • They are generally time consuming • They suppose strong geometrical constraints (i.e. horizontal epipolar line) Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  8. Related Works Related Works • Local approaches provide the solution for a pixel without considering the solution on the rest of the image • They are faster than the Global approaches • They have problems in case of repetitive and uniform patterns The Rationale The Algorithm • Global approaches provide the solution for the whole image trying to optimize the global solution • They are slower than the Local approaches (no Real Time) • They are more robust in case of repetitive and uniform patterns and in case of little local perturbations Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  9. Related Works Related Works: a comparison The Rationale The Algorithm Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  10. ???? Related Works Related Works: open problems • Punctual matching between the image pair is unsuitable in some realistic framework (texture-less regions) The Rationale The Algorithm • The motion of the robot produces mechanical vibrations of the cameras with a consequent loss of epipolar line constraint Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  11. Related Works Related Works: open problems • It is typically time consuming: matching is performed for each pixel of the image and good results are possible defining time consuming optimization functions The Rationale • Left and right image could have different acquiring conditions (lighting, focus, digitalization noise etc.) The Algorithm Results ? Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  12. ? OK ? ? Our approach: The Rationale Related Works • In some cases we can not have enough information to find the correspondence looking just at a single pixel. • For example, pixels inside homogeneous areas, or pixels suffering from perspective or photometric distortions, digitalization errors, vibration of the cameras. The Rationale Our idea is to face the stereo matching problem as a matching between homologous regions(instead of pixels) The Algorithm Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  13. Our approach: The Rationale Related Works • We determinate the disparity value for the whole region, so we define an approximation of disparity property: the horizontal displacement between the regions • We start from the projection of a region (and not of a point) on the stereo pair The Rationale The Algorithm Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  14. Our approach: The Algorithm Related Works The algorithm is based on a graph representation of the stereo pair and a stereo registration of regions using graph matching The Rationale The Algorithm Results Conclusions ? Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  15. Our approach: The Algorithm Related Works • The left and right images are segmented and each area identifies a node of a graph Left Image Right Image The Rationale • A bipartite graph matching between the two graphs is computed in order to match each area of the left image with only one area of the right image Segmentation Segmentation The Algorithm Graph Representation Graph Representation Recursive Weighted Bipartite Graph Matching • This process yields a list of reliably matched areas and a list of so-called don’t care areas. Results don’t care areas matched areas • The Outputs of the algorithm are the disparity map and the performance map Disparity Computation Conclusions disparity map performance map Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  16. Our approach: The Algorithm Related Works Segmentation • The segmentation process is simple and very fast: we are not interested in a fine segmentation (multi-threshold segmentation) The Rationale • We have similar segments between the left and right images because: • the stereo imagesrepresent two different view points of the same scene • we process an adaptive quantization for each image according to its lighting condition The Algorithm Results The segmentation process does not influence the rest of algorithm, because a recursive definition of the matching and a performance function guarantee a recovery of some segmentation problems Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  17. N0R N0L … N1R NnL … NmR Our approach: The Algorithm Related Works Graph Representation • Each 4-connected area from segmented image is a node of an attributed graph: • colMean: the RGB mean value of the blob • size: the number of pixels in a connected area • coord: the coordinates of the box containing the blob • blobMask: a binary mask for the pixels belonging to the blob The Rationale The Algorithm Let GL = {N0L,…,NnL} and GR = {N0R,…,NmR} be the two graphs representing the left and right image respectively Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  18. Our approach: The Algorithm Related Works Recursive Weighted Bipartite Graph Matching The Rationale • Each edge (NiL, NjR) of the complete bipartite graph has a cost, depending of color, dimension and position: • The lower is the cost, the more suitable is that edge • If the cost of an edge is higher than a threshold, the edge is considered unprofitable and is removed from the graph The Algorithm Results • The matching with the lowest cost among the ones with maximal cardinality is selected as the best solution Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  19. Our approach: The Algorithm Related Works Recursive Weighted Bipartite Graph Matching The matching is generally time-consuming For this reason the search area (that is the subset of possible couples of nodes) is bounded by the epipolar and disparity bands These constraints come from stereo vision geometry, but in our case they represent a generalization The Rationale The Algorithm The epipolar band is a generalization for epipolar line, that is the maximum vertical displacement of two corresponding nodes The disparity band is the maximum horizontal displacement of two corresponding nodes Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  20. Our approach: The Algorithm Related Works L R The Rationale epipolar band The Algorithm R Two nodes of the right image that do not belong to the search area (bounded from epipolar and disparity band) Results Conclusions max disparity disparity band Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  21. Our approach: The Algorithm Related Works RecursiveWeighted Bipartite Graph Matching The graph matching process yields a list of reliably matched areas and a list of so-called don’t care areas The matched areas are considered for the disparity computation The Rationale The Algorithm The list of the don’t care areas is processed in order to group adjacent blobs in the left and right image and consequently reduce split and merge artifacts of the segmentation process, a new matching of these nodes is found The recursive definition of this phase assures a reduction of the don’t care areas in few steps, but sometimes this process is not needed because don’t care areas are very small Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  22. Our approach: The Algorithm Related Works Disparity Computation The disparity computation is faced superimposing the corresponding nodes until the maximum covering occurs The Rationale The horizontal displacement, corresponding to the best fitting of the matched nodes, is the disparity value for the node in the reference image The Algorithm Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  23. Our approach: The Algorithm Related Works Output Disparity Map Graphic Perform The Rationale The Algorithm Two post-filters have been applied: Results Enlargement of the contour for each node Hole closing using the mean of the contour Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  24. Our approach: The Results Related Works Advantages: • Texture-less regions: no problems to match uniform regions because of region based approach Graph matching Punctual matching The Rationale Vibration of cameras: more robustness because we compute the disparity matching between regions and use a generalized epipolar line constraint The Algorithm DP (2 sec) OUR (1.1 sec) Results Low Execution Time: no matching for each pixel; it is defined a searching area for graph matching OUR (1.7 sec) SSD (<1 sec) Conclusions Left/Right lack of homogeneity: the cost function in the WBGM is enough independent from local and global perturbation between the two images Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  25. Our approach: The Results Related Works We report some results obtained on a realistic video acquired from our mobile platform (100 frames): camera vibration, light changing, uniform obstacles The Rationale The Algorithm Results The comparison is made between our method and two of the most used methods in the literature: SSD [Kanade et al., PAMI, 99] and SSD Multiscale [Konolige et al., 2005] Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  26. RG RI RD Our approach: The Results Related Works • The performance index are: • recall = subset of regions correctly detected as obstacles (RI) • obstacle regions in the Ground Truth (RG) • precision = subset of regions correctly detected as obstacles (RI) • detected obstacle regions (RD) The Rationale The Algorithm • Relative Distance Error = | detected distance – real distance | . real distance Results Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  27. Our approach: The Results Related Works The Rationale The Algorithm SSD Results Conclusions OUR APPROACH SSD MULTISCALE Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  28. Conclusions Related Works We have presented a stereo matching algorithm providing a fast and robust detection of object positions insteadof a detailedbut slow reconstruction of the 3D scene The Rationale The Algorithm The algorithm has been experimentally validated showing an encouraging performance when compared to the most commonly used matching algorithms, especially on real-world images Results Future works are oriented to test our method in outdoor environment and to develop a temporal coherence of the solution in the video sequence Conclusions Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  29. References • [1] M. Bertozzi, A. Broggi, A. Fascicoli: “Vision-based intelligent vehicles: State of art and perspectives”. Robotics and Autonomous Systems, Vol. 32, pp. 1-16, October 1, 1999. • [2] G. N. DeSouza, A. C. Kak: “Vision for Mobile Robot Navigation: A Survey”. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24-2, February 2002. • [3] D. Scharstein, R. Szeliski: “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms”. International Journal of Computer Vision, Vol. 47-1, pp. 7-42, May 2002. • [4] C. Zhang: “A Survey on Stereo Vision for Mobile Robots”. Dept. of Electrical and Computer Engineering, Carnegie Mellon University. 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA. 2002. • [5] O. Faugeras, B. Hotz, H. Mathieu, T. Viéville, Z. Zhang, P. Fua, E. Théron, L. Moll, et al.: “Real Time Correlation-Based Stereo: Algorithm, Implementations and Applications”. INRIA Technical Report 2013, 1993. Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  30. References • [6] S. Denasi, C. Lanzone, P. Martinese, G. Pettiti, G. Quaglia, L. Viglione, Real-time system for road following and obstacle detection, in: Proceedings of the SPIE on Machine Vision Applications, Architectures, and Systems Integration III, October 1994, pp. 70–79. • [7] M. Lützeler, E.D. Dickmanns, Road recognition with MarVEye, in: Proceedings of the IEEE Intelligent Vehicles Symposium ’98, Stuttgart, Germany, October 1998, pp. 341–346. • [8] H. C. Longuet-Higgins, “A computer algorithm for reconstruction a scene from two projections”, Nature, vol. 293, pp. 133-135, 1981. • [9] M. E. Spetsakis, J. Aloimonos, “Structure from motion using line correspondences”, International Journal Computer Vision, vol. 4, pp.171-183, 1990. • [10] R.Y. Tsai, T.S. Huang, “Uniqueness and estimation of three dimensional motion parameters of rigid objects with curved surfaces”, IEEE Transactions PAMI, vol. 6, pp. 13-27, 1984. Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

  31. References • [11] G. Halevy, D. Weinshall: “Motion of disturbances: Detection and tracking of multi-body nonrigid motion”. Machine Vision Application, Vol. 11-3, pp. 122–137, 1999. • [12] B.K.P. Horn: “Robot Vision”. MIT Press, Cambridge, Massachusetts, 1986. • [13] T. Kanade, and M. Okutomi: “A stereo matching algorithm with an adaptive window: theory and experiment”. IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 16, pp. 920-932. 1999. • [14] Agrawal, M and Konolige, K and Iocchi, L. Real-time detection of independent motion using stereo, in Proceedings IEEE workshop on visual motion, 2005. • [15] D. Marr, T. Poggio: “A computational theory of human stereo vision”. Proc. R. Soc., Vol. 204-B, pp. 301-328, 1979. • [16] http://mars.sgi.com/default1.html • [17] R. Nevatia, K. Babu: “Linear feature extraction and detection”. Computer Graphics Image Processing, Vol. 13, pp. 257-269, 1980. Alessandro Limongiello –“Stereo Vision for Obstacle Detection: a Graph-Based Approach” – GbR 07 – Alicante June 11-13, 2007

More Related