Image Based PTM Synthesis For Realistic Rendering of Low Resolution 3D Models

1. Image Based PTM Synthesis For Realistic Rendering of Low Resolution 3D Models Pradeep Rajiv 200402028 Advisors : Prof M.AnoopNamboodiri, IIIT Hyderabad

2. Outlines of the presentation • The problem of rendering real world objects and its interpretation. • Its challenges and importance in the realm of Computer Graphics & Computer Vision. • A peek at various tools that help in solving this problem and their limitations. • Our solution to this interesting problem!!

3. Where does it all start?

4. All roads lead to Computer Graphics!!

5. Computer Graphics • Representation and synthesis of visual content. • Mathematical & computational foundations of image generation and processing. • 3D Representation of geometric data & rendering of 2D images . • 3D model consists of surface- geometry and surface color/texture information.

6. What’s the problem that we are solving!! Problem Statement : Given the shape model, a small set of views of a large real-world object and its reflectance properties, synthesize a realistic texture model of the object . Goal: Synthesis of a texture model that looks visually pleasing, similar to its real-world counter part, and also dynamically changes its visual appearance interacting with the light conditions around. Render the model so generated in arbitrary light and view conditions.

7. Terminology To Explain • Shape Model. • Digital acquisition, modeling & rendering. • Reflectance properties. • Synthesis of texture models.

8. Shape Models of 3D objects • Mathematical description of the surface geometry. • A set of points pi (xi , yi , zi) over the surface of an objects represent its surface geometry. • The level of surface detail is directly proportional to the number of points. • Representation: Polygonal mesh models and Dense point models

9. Dense Point Models Polygonal Mesh Models

10. Digital acquisition & modeling of an object • Each point (x, y, z) on the surface of an object is assigned color. • Color is usually a RGB vector (ri ,gi ,bi ) or a position ( tx ,ty ) in an Image I. • Color can be functional Fi=( fri ,fgi ,fbi ) I where I is some color space.

11. Texture Mapping • Technique to add surface detail to an object by wrapping or projecting the color information from an image. • An important criteria and extensively employed in computer graphics

12. Rendering of Models

13. Image Based Modeling and Rendering (IBMR) • Ably capture visual and structural information. • Acquisition of details is easy and has universal applicability. • Rely on a set of images to construct a 3D model, texture it and generate novel views. • Directly generate novel views using multi-view geometry.

14. IBMR based View Reconstruction

16. Reflectance Properties of Natural Materials • Visual characteristics arise from varying 1) surface normals 2) reflectance. • Cause shadows, inter-reflections, and specularities, • Change in visual appearance with changing light and viewing conditions. (d) (a) (b) (c)

17. Reflectance Textures • Model reflectance properties of a texture as well as its color information. • Obtained by applying Image-based relighting techniques to model surface reflectance properties. Reflectance Textures Vs Simple Color Textures

18. Image based Texture Synthesis

19. Surface Texture Synthesis • Techniques to synthesize texture directly over the surface of a 3D object. • Praun et al.’s lapped textures (2000). • Turk et al.’s texture synthesis on surfaces (2001). • Wei and Levoy’s texture synthesis over arbitrary manifold surfaces (2001).

22. Problem statement Revisited Given a polygonal mesh model of an large structure, its views under varying light conditions, a sample reflectance texture map of its material, synthesize a reflectance texture model of the object and render it under arbitrary light and view conditions.

23. Motivation • Realistic rendering of real world objects is an important area of computer graphics. • Prominent usage in movies, games and archival of historical artifacts. • Holds the key for Digital Heritage.

24. Challenges • Acquisition, rendering of shape & surface details. • High resolution shape modeling. & limited resolution of capture devices at large scales. • Labor intensive assemblage of a single large model. • Time & storage space. • Modeling of Surface light interactions.

25. IBMR for Large structures Sample Material The Buddha Statue in HussainSagar Zoomed 8X Zoomed 4X 8 times nearer 4 times nearer Up-close View

26. Modeling & Rendering of Large Structures • Detailed Shape and surface acquisition. • Huge shape model and large number of high resolution images. • Point-based modeling and rendering of millions of polygons and billions of points.

27. Digital Michelangelo • Built using laser scan technology and light fields. • Contains 4 billion polygons and 7000 images. • Clean up, alignment , merge, and processing was huge & gigantic cost incurring.

28. Photograph vs Rendering

29. Limitations of existing IBMR techniques • Effected by resolution of digital acquisition. • Acquisition at lower resolutions results in coarser surface detail. • High resolution modeling seldom possible in case of large structures. • Cannot model surface reflectance details.

30. Our Approach to Realistic Modeling of Large Structures

31. Our Approach • De-coupling of shape and surface detail. • Shape capture in coarser polygonal mesh models. • Relegation of surface details to reflectance textures. • Synthesis of reflectance texture over the mesh model. • Units of synthesis selected from the sample conditional upon the set of object views. • Image based Modeling + Texture synthesis

32. Polynomial Texture Maps(PTM) (u,v) lv lu Light Direction Space

33. PTM • Deviced by Malzbendaret al (2002). • Belong to class of Uni-directional Texture Functions • Model the surface luminance at each texel as a bi-quadratic polynomial function. • Store RGB per pixel and luminance coefficients (a0 – a5) per texel. • Chromaticity of a pixel (Rn, Gn, Bn) remains constant.

34. PTM light parametrization (u,v) - texture co-ordinates (a0-a5) - fitted luminance coefficients (lu,lv) – projection of light direction into texture plane.

35. Fitting PTM to Image Data • Set of images {Ik} of the object are obtained under different light conditions {(luk, lvk)}. • The best fit (a0-a5) at each pixel(u,v) is computed using SVD so as to fit the pixel data {Ik(u,v)}. • The above representation is called LRGB PTM.

36. PTM Synthesis • Based on patch based texture synthesis. • The texture is viewed as a realization of a homogenous markovian process. • Textural characteristics of a block W are completely determined by its causal neighborhood NW . • Blocks with similar causal neighborhood are copied from the input and pasted.

37. Pasting blocks with similar causal neighborhood W NW

38. Our METHOD: Image Based PTM SYNTHESIS

39. Image based PTM synthesis • Generates the reflectance model of an object from its mesh model, sample PTM and a sparse set of views. • Extends the patch based PTM synthesis algorithm to also include the image based constraints. • Selection of synthesis blocks is conditional upon the image set.

40. Builds on texture transfer by Efroset al (2001) and 3D texture synthesis by Yacov-Hel-Or et al (2003). • Inputs: A coarser shape model, high resolution sample PTM and a sparse set of views. • Synthesizes a texture that behaves more like its real world counter part in different light conditions.

41. Synthesis for planar surfaces • Synthesizes reflectance model of a planar surface. • Uses a sample PTMin, a set of views {Ik} as constraints and generates reflectance map PTMout. • Takes patches from PTMinas building blocks. • At each step k, a block Bk is selected from PTMinand stitched into PTMoutwith an overlap of width We. • alpha-blending in the overlapping regions.

42. Block selection strategy • Raster scan fashion of synthesis. • At each step k, a candidate block Bk is selected from PTMin and pasted at the next position (x, y). • The selection of Bk is governed by two constraints • Image based constraints • Overlapping constraints

44. Image based Constraints • Set of images {In} captured under light positions (lun, lvn) decide the candidate patches. • For the desired PTM block Bk, f(Bk, (lu, lv)) should be similar to the set of image blocks {b(In, x, y)}. • The blocks {B}ϵPTMin are ranked according to S. • The blocks {B} are ranked according to score S.

45. Block Overlap Constraints • Patch Bkshould agree with its neighbors in PTMout in the overlapping regions. • Blocks selected based on image based constraints are again ranked based on overlapping measure. • L2 norm over the difference of luminance coefficients in the overlapping region is the error measure. • Bwith minimal error measure is the desired block.

46. Results: Rough Plaster surface High resolution sample PTM PTM model Object

47. Up-close view PTM model Image based Modeling

48. Sample PTM Object PTM model

49. Object Our Results Unconstrained Synthesis

50. Image constrained PTM synthesis for real world objects • Synthesis reflectance texture models of objects which are 3D in nature. • Quilting blocks are triangles of various shapes, sizes and orientation. • Synthesizes reflectance texture over the triangles of a mesh model.