1 / 54

Presented By: Itzik Ben Shabat January 2014

Patch Based Synthesis for Single Depth Image Super-Resolution (ECCV 2012) Oisin Mac Aodha , Neill Campbell, Arun Nair and Gabriel J. Brostow. Presented By: Itzik Ben Shabat January 2014. Contents. Problem & Motivation SR General Overview Related Work The Proposed Method Results

ermin
Download Presentation

Presented By: Itzik Ben Shabat January 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Patch Based Synthesis for Single Depth Image Super-Resolution (ECCV 2012)Oisin Mac Aodha, Neill Campbell, Arun Nair and Gabriel J. Brostow Presented By: Itzik Ben Shabat January 2014

  2. Contents • Problem & Motivation • SR General Overview • Related Work • The Proposed Method • Results • Qualitative • Quantitative • Future Work • Paper review

  3. Problem & Motivation • How do we convert a Low Resolution (LR) image to High Resolution (HR) ? • Get a better camera (Sensor)

  4. Problem & Motivation • Cant get a better camera? Super Resolve the image! (SR)

  5. Problem & Motivation • Now do it in RGB-D ! ! ! PMD CamCube - 200X200 MS Kinect - 640x480 PointGreyBumbleBee 2 - 640x480 at 48fps

  6. SR General Overview Common approaches: • Take multiple LR images from different angles and reconstruct the additional information (requires multiple images)

  7. SR General Overview Common approaches: • Use a LR to HR database (requires a database) • Focus on this approach

  8. Related Work • Intensity Images • EbSR [15] - Freeman, W.T., Liu, C.: Example-based super resolution. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011) • Similar: • Filter input • Normalized Patch matching • Solving minimum energy problem (using BP) • Different • Not Designed for RGB-D images • Matching HR and interpolated LR patches

  9. Related Work • EbSR looking closer EbSR Output Ground trouth

  10. Related Work • Intensity Images • ScSr [17] - Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010) • Similar: • Use patches and minimization problem • Different: • Not designed for RGB-D images • Two dictionaries • Database structure (sparse representation) • Solves 2 minimization problems separately – global and local • No noise reduction implementation

  11. Related Work • Depth + Intensity Hybrids • Cross Bilateral [1] - Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR. (2007) • Similar: • Specific for RGB-D • Use bilateral filter • Different: • Requires additional input (destination resolution image) • Doesn’t use patches • Solves a fusion problem

  12. Related Work • Depth + Intensity Hybrids • MRF SR [25]- Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS. (2005) • Similar: • Uses MRF • Different: • Uses multi-resolution MRF • Requires additional input (destination resolution image) • Doesn’t use patches • Solves a fusion problem

  13. The Proposed Method • Challenges • Construct database • Noise • “Flying pixels” at discontinuities • Wrong depths for specular or dark materials • Edges – jarring artifacts (different than rgb image)

  14. The Proposed Method - Overview

  15. The Proposed Method – Database • Constructing the database • Less sources for database construction than rgbimages • Considered synthetic Vs. Real datasets • Database uses 30 scenes of 800x800 (scenes flipped left to right) – 5.3 million patches • Pruning to remove redundant patches (planar surfaces) Synthetic Laser Scan

  16. The Proposed Method - Overview

  17. The Proposed Method - Filtering • Noise Reduction • Assumption – High frequency=noise • A. Bilateral filter on input patches before patch normalization • Edge preserving • Noise reducing • Nonlinear • Weighted average of intensity values from nearby pixels • *Used in Adobe Photoshop “Blur” function • B. Bicubic filter on database HR patches before down-sampling

  18. The Proposed Method - Filtering After Bilateral filtering Input

  19. The Proposed Method - Filtering • Noise Reduction • Pro – Cleaner image for patching • Con – Some data is lost

  20. The Proposed Method - Overview

  21. The Proposed Method – Matching Input Depth Image

  22. The Proposed Method – Matching Input Depth Image N non overlapping low resolution input patches xi For each xi we wish to find its corresponding high resolution yi Patches are normalized

  23. The Proposed Method – Matching High Resolution Database

  24. The Proposed Method – Matching Input Depth Image Output Depth Image High Resolution Database

  25. The Proposed Method – Matching Input Depth Image Output Depth Image High Resolution Database

  26. The Proposed Method – Matching Output Depth Image High Resolution Database

  27. The Proposed Method – Matching Input Depth Image Output Depth Image … High Resolution Database

  28. The Proposed Method – Matching Input Depth Image Output Depth Image … High Resolution Database

  29. The Proposed Method – Matching Input Depth Image Output Depth Image … High Resolution Database

  30. The Proposed Method – Matching Input Depth Image Output Depth Image … High Resolution Database

  31. The Proposed Method – Matching Input Depth Image Output Depth Image … … High Resolution Database

  32. The Proposed Method - Matching • Matching Patches to database • Matching is done between LR patches • Kd tree is used for speeding up the process

  33. The Proposed Method - Overview

  34. The Proposed Method - Reconstruction • Solving minimum energy problem • Solved using TRW-S algorithm (based on belief propagation) • Ed -Unary Potential - Difference between normalized matching LR patches

  35. The Proposed Method - Reconstruction • Es -Pairwise Potential - Difference between un-normalized HR patch overlaps

  36. The Proposed Method - Reconstruction • Es - Pairwise Potential - yi yj

  37. The Proposed Method - Reconstruction • Es - Pairwise Potential - yi yj

  38. The Proposed Method - Reconstruction • Es - Pairwise Potential - yi yj

  39. The Proposed Method - Reconstruction • Es - Pairwise Potential - yi yj

  40. The Proposed Method - Reconstruction )2 )2 + ( Es=( - - yi yj

  41. The Proposed Method - Reconstruction • Normalization – is un-normalized based on the input patch min and max values

  42. The Proposed Method - Overview

  43. The Proposed Method – Filter Results • Noise reduction • C. Post processing Denoising – Outlier detection and correction using threshold Result after denoising Result input

  44. Results - Qualitative • Exp 1: • Used Middleburry stereo dataset • Down-sampled the ground truth (X2,X4) • Reconstructed • Compared RMSE • Exp2: • 3 laser scans • Upsampled by 4 • Ground touth comparison • Exp3: • Use synthetic Vs. real database

  45. Results - Qualitative • Proposed method Vs. Other Methods (Exp. 2) Proposed Method

  46. Results - Qualitative • Proposed method - Real Vs. Synthetic training data (Exp. 3)

  47. Results - Quantitative Method used • Reminder: MRF RS Cross Bilateral ScSREbSR RGB-D image used RGB-D image used Method used Upsampling factor Method used RMSE

  48. Results - Qualitative Results Movie

  49. Conclusions • 1st or 2nd best from intensity based methods • MRF RS, Cross Bilateral are better but require more data • Speed is not realtime compatible • Super resolving moving depth videos • Synthetic data exhibit better results than scanned data for training.

  50. Future work • Extend to exploit temporal context (in video) • Exploit context when querying the database • Develop a sensor specific noise model for better results

More Related