1 / 43

Image Matting and Its Applications

Image Matting and Its Applications. Chen-Yu Tseng Advisor: Sheng- Jyh Wang 2012-10-29. Image Matting. A process to extract foreground objects from an image, along with an alpha matte ( the opacity of the foreground color ). Input Image. Alpha Matte . Extracted Foreground.

rupert
Download Presentation

Image Matting and Its Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Image Matting and Its Applications Chen-Yu Tseng Advisor: Sheng-Jyh Wang 2012-10-29

  2. Image Matting • A process to extract foreground objects from an image, along with an alpha matte (the opacity of the foreground color) Input Image Alpha Matte Extracted Foreground

  3. Two Approaches of Image Matting • Supervised Matting • With User’s Guidance • Unsupervised Matting • Without User’s Guidance Input Image User’s Guidance e.g. Trimap: White  Foreground Black  Background Unknown  Gray

  4. Two Schemes of Supervised Matting Propagation-based Scheme Sampling-based Scheme Infer Alpha Matte with Some Color Samples A Local-based Approach • Infer Alpha Matte with Propagation through a Graphical Model • A Global-based Approach • Unknown • Pixel • Foreground • Color Set Foreground Pixel • Background • Color Set • Unknown Pixel • Background Pixel

  5. Propagation-based scheme - Matting Laplacian Approach • A Graphical Model with Connectivity between Pixels • The Connectivity Is Learned from the Image Structure • Capability for Dealing with Both • Supervised Matting (Inference Problem) • Unsupervised Matting (Decomposition Problem) Foreground Pixel • Unknown Pixel • Background Pixel

  6. Reference of Matting LaplacianApproach • First proposed by Levin et al. for supervised matting (closed-form matting) • A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008. • Extended to unsupervised matting (spectral matting) • A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no. 10, pp. 1699-1712, Oct. 2008. • Extended to learning-based matting • Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In ICCV, pages 889–896, 2009. • Extended to multi-layer matting • D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,” IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.

  7. Matting Laplacian Background Estimating Pair-wise Affinity Supervised Matting Input Image Graphical Model Foreground Node: Image Pixels Edge: Affinity Matting Laplacian Matrix: Recording the Connectivity between Pair of Pixels

  8. Introduction of Graph Laplacian • Vertex Index • : Adjacency Matrix 2 1 : LaplacianMatrix 3 4 5 : Degree Matrix • : Adjacency Matrix A Graph with Five Vertexes

  9. Cutting Cost Function with Graph Laplacian Cost Function for Cutting Criterion Low-cost Assignment High-cost Assignment 2 1 2 1 3 4 3 4 5 5

  10. Construction of Matting Laplacian • Color-model-based Approach (Original) • Estimating Affinity Based on Relative Color Distance • Learning-based Approach (Extended) • Learning Affinity Based on Image Structure

  11. Construction of Matting LaplacianColor-model-based Approach b g r Color Distribution Input Image • A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.

  12. Construction of Matting LaplacianLearning-based Approach • Learning Affinity among Local Pixels Extending to a Local Patch q Assuming all Pixels Sharing the Same Linear Coefficient Linear Alpha-color Model for Single Pixel: : Alpha Vector for Patch q : Feature Matrix : Linear Coefficient : Alpha Value for Pixel i : Feature Vector () : Linear Coefficient

  13. Construction of Matting LaplacianLearning-based Approach Derived Linear Coefficient Rewritten Linear Model

  14. Construction of Matting LaplacianLocal Cost Function Local Linear Model Local Cost Function Patch q : Local Laplacian Matrix for Patch q Input Image

  15. Construction of Matting LaplacianLocal  Global Local Cost Function Patch q : Local Laplacian Matrix for Patch q Global Cost Function Input Image

  16. Supervised Matting (Closed-form Matting) • Foreground Pixel User’s Guidance, Input Image • Unknown Pixel • Background Pixel Cost Function for Supervised Matting Optimal Solution Affinity Cost Data Cost

  17. Experimental Results Input Image Alpha Matte Synthesized Result

  18. Unsupervised Matting (Spectral Matting) • Solving Alpha Matte without User’s Guidance • Procedures • Decomposing Image into Several Matting Components • Combining Matting Components into Alpha Matte

  19. Spectral Clustering s.t.=1 : Eigenvector : Eigenvalue L is symmetric and positive semi-definite. The smallest eigenvalue of L is 0, the corresponding eigenvector is the constant one vector 1. L has n non-negative, real-valued eigenvalues 0= λ1 ≦ λ 2 ≦ . . . ≦ λ n. 2 1 3 4 5 • : Laplacian Matrix A Graph Example

  20. Spectral Clustering & Matting Components • Linear • Transformation • Zero-Eigenvectors • : Laplacian Matrix • Binary Indicating Vectors

  21. Overview of Spectral Matting K-means Clustering & Linear Transformation Input Image Matting Laplacian Matting Components Smallest Eigenvectors

  22. Spectral Clustering & K-means Pixel i Input Image s-dimensional Space K-means Clustering … s-smallest Eigenvectors

  23. K-means Projection into Eigen Space Generating Matting Components … … … Smallest Eigenvectors

  24. Reconstructing Alpha Matte from Matting Components Input Image Matting Components + + = Selected Matting Components Alpha Matte

  25. Reconstructing Alpha Matte by Grouping Matting Components Alpha Matte Generation : Combination Vector Matting cost function Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte

  26. Results by Levin et al.

  27. Summary • Constructing Matting Laplacian • Solving Supervised Matting Problem • Solving Unsupervised Matting Problem

  28. Proposed Approaches • Efficient Cell-based Framework for Reducing Computations • Multi-scale Analysis • Extended Applications (Depth Image Reconstruction) Depth Reconstruction in Shape From Focus (SFF) Depth Reconstruction from Single Image Reconstructed Depth Input Image Reconstructed Depth Input Image

  29. Cell-based Framework Pixel-wise Affinity Conventional Matting Laplacian Pixel-wise Data Distribution Cell-wise Affinity Image Cell-based Matting Laplacian Cell-wise Data Distribution

  30. Multi-scale Affinity Learning Cell-based Approach Image & Computation Patterns Pixel-based Approach

  31. Multi-scale Affinity Learning … … Finest Level Coarsest Level Cell-based Graph

  32. Results of Reconstructed Alpha Matte Input • (a) Grouping Results by Levin et al. • (b)Grouping Results by Levin et al. with Coarse-to-fine Scheme. (c) Ours 2nd Rank 1st Rank

  33. Results (a) Input images (b) Levin’s result (c) Our result

  34. Proposed Approaches • Efficient Cell-based Framework for Reducing Computations • Multi-scale Analysis • Extended Applications (Depth Image Reconstruction) Depth Reconstruction in Shape From Focus (SFF) Depth Reconstruction from Single Image Reconstructed Depth Input Image Reconstructed Depth Input Image

  35. Depth Reconstruction in Shape From Focus (SFF) Optical Direction W2 W1 W2 Optical Direction Multi-focus Image Sequence Focus Value W1

  36. Low-SNR Problem • Spatially Varying Precision • Low-texture  Low-SNR • Leading Noisy Result Low- precision High- precision Input Image Observation

  37. Proposed Maximum-a-posteriori Estimation Local Learning Learning-based Graph Inference Multi-focus Image Sequence Reconstructed Depth

  38. Proposed Maximum-a-posteriori Estimation : Optimal Result : Depth Image : Observation : Input Image Posterior Likelihood Prior Learned from Image Local Observation with Spatial-varying Precision

  39. Likelihood Model Posterior Likelihood Prior Local Observation with Spatial-varying Precision Low- precision High- precision Input Observation Precision Result

  40. Prior Model Posterior Likelihood Prior Learning from Input Image Local Learning Learning-based Graph Multi-focus Image Sequence

  41. Maximum-a-posteriori Estimation for Depth Reconstruction Input Image Observation Reconstructed Depth

  42. Results of Shape from Focus Input Image S. Nayar, 1994 Ours M. Mahmood, 2012 T. Aydin, 2008

  43. Conclusions • Construction of Matting Laplacian • Conventional Approach • Multi-scale Cell-based Approach • Supervised Matting • Spectral Matting • Depth Reconstruction

More Related