Eigen-Texture Method Appearance Compression based on 3D modeling

1. Eigen Texture Method : Appearance compression based methodSurface Light Fields for 3D photographyPresented by Youngihn Kho

2. Eigen-Texture MethodAppearance Compression based on 3D modeling • Authors Ko Nishino, Yochihi Sato, Katsushi Ikuechi (CVPR 99) • http://www.cvl.iis.u-tokyo.ac.jp/~kon/eigen-texture/index.html

3. Appearance based method Inputs : 3D Geometry and a set of color images Outputs : Synthesized images from arbitrary viewing points. Objective : Compressed representation of appearance of model

4. Eigen-Texture methodOverview sampling encoding decoding

5. Sampling • 3D geometry is captured by range camera. • Photos are taken by rotating the object and registered into the geometry. • Each color image is divided into small areas according to their corresponding triangles in the object. • Cell : Normalized triangle patches. • Cell image : Warped color image • Compression is done with sequences of cell images of each cell.

6. 357° θ θ appearance change of one patch 0° Sampling Image coherence makes high compression ratio and interpolation of appearances.

7. Compression key idea • Instead of storing whole sequence of images, we find a small set of new cell images (eigen cells), then represent each cell image as a linear combination of those eigen cells. • Then we store only those eigen cells and the coefficients of each cells.

8. At a glance… Original data Using principal vector Size = 2*N Size = N + 2

9. n n Matrix construction • Single cell image • Sequence of cell images M : # of images N : # of pixels in each cell

10. Eigen method Keigen vectors

11. Eigen ratio • Sort the eigen values and pick k largest eigen values. • We need all eigen values if we want to preserve original, but we need compression.

12. k eigen-cells scores synthesized view = a0× + a1× + a3× linear combination of base images Decoding with eigen vectors

13. Compression ratio • Size of a sequence of cell images : M x 3N • Size of k-eigen cells : k x 3N • “coefficients” of image cells : k x M • Therefore,

14. Cell-adaptive dimensions • Several factors influence the required dimension - Specularity, mesh precision, shadowing… • Since we compress for each sequence of cell images, we can use different dimension for each cells. • This can be done by using fixed threshold eigen ratio.

15. Interpolation • Synthesize from novel view point is done in the eigen-space by interpolating “coefficients” (or scores)

16. Integrating into the real scene • Can render scene under arbitrary illumination condition • Sample color images under several single light conditions • Synthesize the scene under approximate arbitrary illumination condition by linear combination of those base images

18. Shadowing

19. Discussion • Contribution - high compression ratio - interpolation in eigen space - global illumination effects • Drawbacks - expensive pre-computation time - limited positions. - dense mesh?

20. Surface Light Fieldsfor 3D photography • Authors Daniel N. Wood, et al. (SIGGRAPH 2000) • http://www.cs.washington.edu/research/graphics/projects/slf/

21. Surface Light Field • Surface Light Field is a function that assigns a color to each ray originating from every point on the surface. • Conceptually, every points in the surface has it’s corresponding lumisphere..

22. Overview Overview Estimation And Compression Data Acquisition Rendering Editing

23. Overview Overview Estimation And Compression Data Acquisition Rendering Editing

24. Data Acquisition • Range scanning • Reconstructing the geometry • Acquiring the photographs • Registering the photographs to the geometry

25. Scan and reconstruct Use closest points algorithm & volumetric method by Curless and Levoy

26. Acquiring photographs Used Stanford spherical gantry with a calibrated camera

27. Register photographs to the geometry Manually selected correspondences

28. How to represent? • MAPS (Multiresolution Adaptive Parameterization of Surfaces) - Base mesh + wavelets - (Aaron Lee ,et al. SIGGRAPH ‘98) map Base mesh Original mesh

29. Data lumishpere in each points Scanned mesh Base mesh Lumisphere

30. Overview Estimation And Compression Data Acquisition Rendering Editing

31. Estimation and Compression • Estimation problem : How do we find piece-wise linear lumisphere from given data lumisphere? • Three methods - Pointwise- fairing - Function quantization - Principal function analysis

32. Point-wise fairing • Estimate the least-squares best approximating lumisphere for individual surface points. • Error function = distance term + thin plate energy term • Results high quality but suffers large file size. – needs compression technique.

33. Point-wise fairing

34. Point-wise fairing

35. Compression • Don’t want each grid point to have its own lumisphere. • Rather, want a small set of lumispheres that can be used to nicely approximate all the data lumisphere. • Standard techniques : - Vector quantization - Principal function analysis

36. Compression in point-wise fairing • Compression in the point-wise fairing method. - vector quantization or principal component analysis directly on their results. • Not a good idea because, - we’ve already had re-sampling step and many parts of they are fiction! • So directly manipulate data lumispheres.

37. Two pre-processing • Two transformation are applied to make them more compressible. - Median removal - Reflected re-parameterization

38. Median removal

39. Reflected re-parameterization

40. Reflected re-parameterization

41. Effect of reflection Before After

42. Function quantization Input data lumisphere Codebook of lumispheres

43. Function quantization • Lloyd iteration : Start with initial single codeword and a random set of training lumispheres, then repeatedly - Split and perturb codebook and repeatedly apply - Projection : find closest codeword index. - Optimization : for each cluster, find best piece-wise lumisphrer until error difference is less than some threshold. Until desirable size.

44. Lloyd iteration Codeword

45. Lloyd iteration Clone and perturb code words

46. Lloyd iteration Divided by several clusters

47. Lloyd iteration Optimize code words in the cluster

48. Lloyd iteration New clusters

49. Principal function analysis • Generalization of principal component analysis • Again we find a set of code words (prototypes), but instead of assign to each grid point, we approximate with linear combination of prototypes.

50. Principal function analysis Input data lumisphere Prototype lumisphere Subspace of lumispheres