Depth Image Enhancement Using Local Tangent Plane Approximations

1. Depth Image Enhancement Using Local Tangent Plane Approximations ○ Kiyoshi Matsuo Yoshimitsu Aoki HokuyoAutomatic Co., Ltd. Keio University

2. Contents • Background • Research purpose • Related work • Proposed method (overview and highlight) • Local coordinates based depth image enhancement • Experimental results • Conclusion

3. Research purpose Depth image enhancement for consumer RGB-D cameras * Consumer RGB-D camera > Designed for human interaction applications - low accuracy - low resolution + low cost + portable Ex. Soft Kinetic DS311 Enhancement of both the accuracy and resolution could help many applications. Ex. fine-grained object recognition, precise indoor navigation, and etc…

4. Related work Global optimization based enhancement Markov Random Field 1. Measured depth data is used as optimization data prior. 2. Various pixel based information is used as optimization smoothness prior. (gradient, segmentation, edge saliency, non-local mean, and co-sparseness etc.) 3. Depth enhancement is achieved by an optimization of over the image plane. [ Diebel et al. 2006 ] “An application of Markov random fields to range sensing” [ NIPS 2006 Diebel et al. ] “High quality depth map upsampling for 3d-tof cameras” [ ICCV 2011 Park et al. ] “A Joint intensity and depth co-sparse analysis model for depth map surperresolution” [ ICCV 2013 Kiechle et al. ] “Depth map inpainting under a second-order smoothness prior” [ Image Analysis vol. 7944 Herrera et al. ] etc. Local filter based enhancement Joint bilateral upsampling filter 1. Local measured depth data is summarized by using similarity weights. 2. Similarity weights are defined on the global image plane coordinates. (pixel-distance, color-difference, gradient, and color geodesic distance, etc.) 3. Depth enhancement is achieved by local calculations on the image plane. [ Kopf et al. 2007 ] “Joint bilateral upsampling” [ ACM transactions on Graphics 2007 Kopf et al. ] “Spatial-depth super resolution for range images” [ CVPR 2007 Yang et al. ] “Pixel weighted average strategy for depth sensor data fusion” [ ICIP 2010 Garcia et al. ] “Upsampling range data in dynamic environments” [ CVPR 2010 Dolson et al. ] “Joint geodesic upsampling of depth images” [ CVPR 2013 Liu et al. ] etc.

5. Related work Previous enhancement methods use pixel-coordinates. These methods smooth the surfaces to be parallel to the image plane. Image plane No geometric relationship Measuring object The geometries of surfaces are not take into account, (and are sometimes corrupted by them)

6. Contributions To improve accuracy, we introduce local tangent planes as local coordinates to handle the geometries. Global coordinates Local coordinates * pixel-coordinates of the image * charts defined on each local tangent * depend only on the measuring device (independent of the measuring surfaces) * depend on the local geometries of measuring surfaces Accurate depth image enhancement is achieved. (especially in noisy cases)

7. Proposed method (overview) * Output data Only overview of our method (more details will be shown at our poster.) 2. Surface reconstruction 1. Estimation of local tangent planes Proposed in ICCVW 2013 * Input data A pair of depth and color images A high-resolution accurate depth image

8. How to use local coordinates ? To recover smooth surfaces, we smooth normal components of surfaces on each local coordinates Measured surface Smoothed surface Colored by the height from the tangent plane Height smoothed surface Thesefilters on local coordinates smooth thesurfaces while preserving the local geometries of local tangent planes.

9. Experimental results Source RGB-D image MRF [Diebel and Thrun] PWAS [Garcia et al] Previous ours [Matsuo and Aoki] Our method Captured by SoftKinetic DS311 The interior regions of surfaces were reconstructed more accurately.

10. Conclusion • Local coordinate based depth image enhancement. • Local tangent planes are used as local coordinates. • Normal component smoothing reduces granular noises. • These filters preserve the local geometries of local tangents. • The effectiveness is shown by real sensor data.

11. Thank you for this opportunity to present to you today. And thank you for your attention ! If you have any questions and comments, visit our poster IS 2-4.