Recovering Photometric Properties of Architectural Scenes from Photographs

1. Recovering Photometric Properties of Architectural Scenes from Photographs Yizhou Yu Jitendra Malik Computer Science Division University of California at Berkeley July 1998

2. Context • IBMR re-renders from novel viewpoints. • Façade, Plenoptic modeling, • Lumigraph, Light field, • Panoramic mosaics • But, unlike traditional rendering, lighting cannot be changed.

3. The Problem • Texture Maps are not Reflectance Maps ! • Need to factorize images into lighting and reflectance maps Illumination Radiance Reflectance

4. Objective • Start from photographs • Recover parametric models for lighting and reflectance • Re-render the scene under novel lighting conditions

5. Some Photographs...

6. Camera Radiance Response Curve • Pixel brightness value is a nonlinear function of radiance. • Debevec & Malik[Siggraph’97] give a method to recover this nonlinear mapping. Intensity Saturation Radiance Radiance

7. Previous Work • BRDF measurement and recovery • [Ward 92],[Dana et al. 97] • [Sato & Ikeuchi 96], [Sato et al. 97] • Rendering outdoor scenes under skylight • [Nishita and Nakamae 86], [Tadamura et al. 93]

8. Basic Approach • Recover geometric model • Measure and recover illumination • Recover reflectance • Predict illumination at novel times of day • Render Illumination Radiance Reflectance

9. Technical Challenges • Nonlinear mapping between input radiance and digital output . • Photographs cannot easily recover full spectral BRDF. • Re-rendering the scene at novel times of day requires predicting lighting conditions.

10. Basic Approach • Measure and recover illumination • Recover reflectance • Predict illumination at novel times of day • Render Illumination Radiance Reflectance

11. Modeling the Illumination • The sun • Its diameter extends 31.8’ seen from the earth. • The sky • A hemispherical area light source. • The surrounding environment • Modeled as a set of oriented Lambertian facets.

12. A Sky Radiance Model----based on [Perez 93] zenith Sky element sun • Recover a set of parameters for each color channel • Take photographs for parts of the sky • Use Levenberg-Marquardt algorithm to fit data Lvz, a, b, c, d, e, f

13. A Recovered Sky Radiance Model R,G,B channels

14. Coarse-grain Environment Radiance Maps • Partition the lower hemisphere into small regions • Take photographs at several times of day • Project pixels into regions and obtain the average radiance • Use photometric stereo to recover a facet model for each region

15. Basic Approach • Measure and recover illumination • Recover reflectance • Predict illumination at novel times of day • Render

16. Recovering Reflectance • Parametric model [Lafortune et al.] • Triangulate the surfaces • Set a grid on each triangle to capture spatial variations • Use one-bounce reflection to approximate self-interreflections

17. Pseudo-BRDF • R, G, B color channels perform integration. Define pseudo-BRDF : • In general, the pseudo-BRDF varies with the spectral distribution of the light source. • Recover two sets of surface pseudo-BRDFs • One ==> spectral distribution of the sun • The other ==> the sky and environment

18. Diffuse Term • For each side, at least two photographs for diffuse albedo recovery. • From the photograph not lit by the sun • From the photograph lit by the sun • Solve for

19. Specular Term • Use an empirical specular reflection model proposed in [Lafortune et al. 97]. • Recover the parameters using least squares and robust statistics.

20. Basic Approach • Measure and recover illumination • Recover reflectance • Predict illumination at novel times of day • Render

21. Simulating Novel Lighting for the Sun and Sky • Interpolation with solar position alignment to obtain novel sky radiance distributions • Use to model solar radiance during sunrise and sunset • This is similar to the absorption term used in scattering theory.

22. A Local Facet Model for the Environment lsun nenv • Recover a distinct model for each environment region • Obtain environment radiance maps. • Set up over-determined systems as in photometric stereo and ignore inter-reflections. • Solve for

23. Recovered Environment Radiance Models Synthetic Real

24. Relative Importance of the Components • On shaded sides, the irradiance from the landscape is larger than that from the sky. • On sunlit sides, the sun dominates the illumination. • The specular component is very small compared to the diffuse component.

25. Video

26. Basic Approach • Measure and recover illumination • Recover reflectance • Predict illumination at novel times of day • Render

27. Comparison with Real Photographs Synthetic Real

28. High Resolution Re-rendering • Low resolution and High resolution • and are given. • since the illumination has small variations in high frequencies.

29. High Resolution Re-rendering Real reference image High resolution synthetic image Low resolution synthetic image

30. Video

31. Summary • An approach to render real architectural scenes under novel lighting conditions • The pseudo-BRDF concept • Methods for modeling lighting at novel times of day • A simple method for high resolution re-rendering

32. Acknowledgments • George Borshukov • Paul Debevec • David Forsyth • Greg Ward Larson • Carlo Sequin • MURI 3DDI California MICRO Program Philips Corporation