Single View Modeling

1. Single View Modeling • Guillaume Dugas-Phocion • Li Zhang • Jean-Sebastien Samson • Steve M. Seitz

2. Problem Extract a 3D model from a single image

3. Approach Free-form surface modeling without strong assumptions • No assumptions about • Shading • Geometry • Creation of a surface model • User-added constraints • Fast and simple user interface • Real-time adjustement of the model

4. Related work Single View Modeling • A. Criminisi, I. Reid, and A. Zisserman“Single View Metrology”, 1999. • J. J. Koenderink, A. J. van Doorn, A. M. L. Kappers and J. T. Todd“Pictorial Relief”, 1998. • S.B. Kang, “Depth Painting for Image-Based Rendering Ap-plications”, 1998. • V. Blanz and T. Vetter, “A Morphable Model For The Synthesis Of 3D Faces”, 1999.

5. Related work (2) Surface Modeling • R. Szeliski, “Fast Surface Interpolation Using Hierarchical Basis Functions”, 1990. • S. Gortler and M. Cohen, “Variational Modeling with Wavelets”, 1994.

6. Organisation Organisation • Constraints & Discontinuities • Hierarchical structure • Quad-Tree • The new wavelet-based transformation • Results and examples • Future work

7. I Constraints & Discontinuities Framework Build a surface model by : • Setting some constraints • Fixed points • Surface normals • Adding a smoothness to the surface • Modifying this smoothness locally • Position and normal discontinuities • Fairing curves Demo

8. Surface representation I Constraints & Discontinuities Height field • Logical solution for a single view • Advantages : easy math, fast convergence • Drawbacks : holes, no layer structure

9. I Constraints & Discontinuities Position constraints Fix the height of a surface point • Linear constraint • 2 possibilities : • The point is a grid point :Expression : f(xi) = h • The point is NOT a grid pointInterpolation to get f(x)

10.  a    P : point where we set the normal constraint I Constraints & Discontinuities Normal constraints Fix the local orientation of the surface f() – f() • Dy f(P) = • Dx f(P) = a f() – f() a

11. I Constraints & Discontinuities Smoothness function Give a smooth-like behavior to the surface • Solution : minimize a smoothness function • Discrete version : σ =  Dxx² f(x,y) + Dyy² f(x,y) + 2 · Dxy² f(x,y)dx dy surface σd = wi · (Dxx² + Dyy² + 2 · Dxy² ) · f(pi) grid points wi = Φi (x,y)dx dy

12. min σd = vT H v min σd = vT H v Smoothness function:2nd order polynomial in height values : Smoothness function:2nd order polynomial in height values : Hv = 0 Hv = 0 Constraints : 1st order polynomial in height values : Constraints : 1st order polynomial in height values : Av = b Av = b Final sparse system : H AT v 0 = A 0  b I Constraints & Discontinuities Solving the problem How to obtain a sparse linear system ?

13. I Constraints & Discontinuities Discontinuities HATv0 = A0b HATv0 = A0b Adding discontinuities = modifying the smoothness of the surface = deleting some terms in the smoothness function σd =  wi · (Dxx² + Dyy² + 2 · Dxy² ) · f(xi)

14. I Constraints & Discontinuities Position discontinuities Adding a position discontinuity = In the H matrix, deleting the terms involved in corresponding edges

15. I Constraints & Discontinuities Normal discontinuities Adding a normal discontinuity = In the H matrix, deleting the terms involved in the corresponding points

16. II Hierarchical structure Hierarchical structure Quad-tree structure • Increase the level of details • Without slowing the system • Recursive structure (wavelets) Demo

17. II Hierarchical structure Hierarchical structure Ways of changing the resolution. User specified User specified Along a curve Along a curve Along a curve High curvature High curvature High curvature

18. II Hierarchical structure Speed up the solver 1D hypothesis, no wavelets. How to accelerate the convergence ? f = f(xi) ·Φi Demo

19. II Hierarchical structure Speed up the solver 1D hypothesis, basic wavelets. v : height fieldc : wavelet coefficientsW : wavelet transformation v = W c WTHW WT ATc0 = A W0’b

20. II Hierarchical structure Speed up the solver Wavelet transformation (from coefficient to height) 1D Convolution 2D Convolution hI = (hA + hB) / 2 + cI

21. II Hierarchical structure Speed up the solver What happens with discontinuities ? 1D Convolution Instead of : hI = (hA + hB) / 2 + cI We get : hI = hB + cI Demo

22. II Hierarchical structure Speed up the solver 2D Convolution • Instead of : hO = (hI + hJ + hK + hL) / 4 + cO • We get :hO = (hI + hL) / 2 + cO

23. III Results Results Demo

24. IV Future work Future work • Handle large images • Handle perspective • Fill the holes • Relight the scene • Constraint detector ?

25. Questions ?

26. Further details

27. Position constraints • Easiest constraint : fixing the height of the ith grid point • Expression : f(xi) = h • Corresponding row of the A matrix : • Corresponding value in the b vector : the user-set height h [ 0 … 0 1 0 … 0 ] i-1 i i+1

28. L K P I J Position constraints (2) • If the point we want to fix is not a grid point : we compute its height with the direct neighbors (interpolation) • f(P) = i · f(I) + j · f(J) + k · f(K) + l · f(L) = h • Corresponding row in the A matrix : • Corresponding value in the b vector : the same h [ 0 … cI … cJ … cK … cL … 0 ] iI … iJ … iK … iL (index)

29. Linear problem • Each term as a 2nd order polynom in height values : Dxx²f(xi,i) = [ ________________________ ]² = ∑ ∑ xk,l , m, n f(xk,l)·f(xm,n) = vT·Hxx i,i·v v : vector of the height map. dim = n² Hxx i,i : specific operator for Dxx²f(xi,i) dim = n f(xi,i) + f(xi,i) - 2·f(xi,i) valid for a regular grid δx² k,l m,n 4

30. 1D wavelet basis functions

31. Speed up the solver 2D regular basis functions