Yong Seok Heo , Kyoung Mu Lee, and Sang Uk Lee

1. Mutual Information-based Stereo Matching Combined with SIFT Descriptor in Log-chromaticity Color Space Yong SeokHeo, Kyoung Mu Lee, and Sang Uk Lee Department of EECS, ASRI, Seoul National University, 151-742, Seoul, Korea IEEE Conference on Computer Vision and Pattern Recognition, 2009.

2. Outline • Introduction • System Overview • Mutual Information as a Stereo Correspondence Measure • Proposed Algorithm • Experiments • Conclusion

3. Introduction

4. Introduction • Radiometric variations (between two input images) • Degrade the performance of stereo matching algorithms. • Mutual Information : • Powerful measure which can find anyglobal relationship of intensities • Erroneous as regards local radiometric variations

5. Introduction • Different camera exposures(global) • Different light configurations (local) Conventional MI Conventional MI

6. Objective • To present a new method: • Based on mutual information combined with SIFT descriptor • Superior to the state-of-the art algorithms (conventional mutual information-based) +

7. System Overview

8. Proposed Alogorithm

9. Mutual Information(as a Stereo Correspondence Measure)

10. Energy Function • Energy Function: • In MAP-MRF framework • f: disparity map

11. Mutual Information • Used as a data cost: • Measures the mutual dependence of the two random variables Disparity Map Left / Right Image

12. Mutual Information • Entropy: • Joint Entropy: • P(i): marginal probability of intensityi • P(iL,iR): joint probability of intensity iL and iR • Entropy • Entropy • Joint Entropy 1 0 1 0 1 0

13. Mutual Information • Suppose you are reporting the result of rolling a fair eight-sided die. What is the entropy? →The probability distribution is f (x) = 1/8, for x =1··8 , Therefore entropy is: = 8(1/8)log 8 = 3 bits

14. Mutual Information • Entropy • Entropy • Joint Entropy H(IL, IR ) H( IL | IR ) H( IR | IL ) MI( IL;IR ) H( IL) H( IR)

15. Mutual Information • Entropy: • Joint Entropy: • P(i): marginal probability of intensityi • P(iL,iR): joint probability of intensity iL and iR • Entropy • Entropy • Joint Entropy 1 0 i2 i1 1 0 1 0

16. Pixel-wise Mutual Information • Previous: Mutual Information of whole images • Difficult to use it as a data cost in an energy minimization framework → Pixel-wise

17. Pixel-wise Mutual Information = P(.) / P(.,.) : marginal / joint probability G(.) / G(.,.) : 1D / 2D Gaussian function

19. Left Image Intensity Right Image Intensity

20. Conventional MI • Cannot handle the local radiometric variations caused by light configuration change • Collect correspondences in the joint probability assuming that there is a global transformation • The shape of the corresponding joint probability is very sparse. • Do not encode spatial information

21. Conventional MI • Different camera exposures (global) • Different light configurations (local) Conventional MI Conventional MI

22. Proposed Algorithm

23. 111

24. Log-chromaticity Color Space • Transform the input color images to log-chromaticity color space [5] • To deal with local as well as global radiometric variations • Used to establish a linear relationship between color values of input images [5] Y. S. Heo, K.M. Lee, and S. U. Lee. Illumination and camera invariant stereo matching. In Proc. of CVPR, 2008

25. 111

26. SIFT Descriptor • Robust and accurately depicts local gradient information • Computed for every pixel in the log-chromaticity color space

27. Energy Function • Data Cost: • Mutual Information: • SIFT descriptor distance: constant () Log-chromaticity intensity

28. 111

29. Joint Probability Using SIFT Descriptor

30. Joint Probability Using SIFT Descriptor • K-channel SIFT-weighted joint probability: • : Euclidean distance • VL,K(P) / VR,K(P) : SIFT descriptors for the pixelP • l : SIFT descriptor size T = 1 If True T = 0 If false i2 i1

31. Joint Probability Using SIFT Descriptor • A joint probability is computed at each channel • Use estimated disparity map from the previous iteration. • is governed by the constraint that corresponding pixels should have similar gradient structures.

32. 111

33. Energy Function • Data Cost: • Smooth Cost: MI SIFT

34. Energy Minimization • The total energy is minimized by the Graph-cutsexpansion algorithm[3]. [3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 23(11):1222–1239, 2001.

35. Energy Minimization

36. Experiments

37. Experimental Results • The std. devσ of the Gaussian function is 10, τ = 30, l = 4*4*8 = 128 • The window size of the SIFT descriptor : 9X9 • λ = 0.1, μ= 1.1, VMAX=5 • The total running time of our method for most images does not exceed 8 minutes. • Aloe image (size : 427 X 370 / disparity range : 0-70) is about 6 minutes on a PC with PENTIUM-4 2.4GHz CPU.

38. MI vs. SIFT L: illum(1)-exp(1) / R: illum(3)-exp(1) MI SIFT MI + SIFT Error Rate 17.6% 11.97 % 9.27 % MI SIFT MI + SIFT Error Rate 26.45% 17.87 % 11.83 %

39. L: illum(1)-exp(1) / R: illum(3)-exp(1)

40. Different Exposure Left Image Right Image Ground Truth Proposed Rank/BT NCC ANCC MI

41. Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

42. Different Exposure Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

43. Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

44. Different Exposure Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

45. Different Light Source Configurations Left Image Right Image Ground Truth Proposed 111 Rank/BT NCC ANCC MI

46. Exposure Exposure Light Configuration Light Configuration

47. Exposure Exposure Light Configuration Light Configuration

48. Conclusion

49. Conclusion • Propose a new stereo matching algorithm based on : • mutual information (MI) combined with • SIFT descriptor • Quite robust and accurate to local as well as global radiometric variations