Computer Graphics

1. Computer Graphics Lecture Notes #16 Image-Based Modelling, Rendering and Lighting

2. Global Illumination and Image-Based Lighting • Traditional Computer Graphics involves: • Modelling with matter: geometry with reflectance properties. • Image-Based Modelling & Rendering is: • Modelling and rendering with Light, often deriving geometry and materials in the process. • Image-Based Lighting allows: • Combination of real and synthetic graphics with consistent illumination, using images as light sources. Lecture Notes #16

3. Pursuit of Photorealism • For complex models, tools can require lot of effort and rendering is very slow. • Choice: do usual modelling or just take a photo (that would provide photorealism!) • IBMR is about using these photos and transcending their limitations (eg. Light). • Don’t need anything like a complete model, but may need a number of photographs… Lecture Notes #16

4. Pursuit of Photorealism • Need enough photographs to give coverage. • Major result is - rendering is faster. • So Image-Based Modelling and Rendering (IBMR) starts with the answer! • Research inspired by film industry. • Related to techniques in Computer Vision. • New so no taxonomy, just techniques. Lecture Notes #16

5. IBMR Techniques • Panoramas – movement within and between panoramic cylinders, eg. Quicktime VR. • Panoramas can be real or synthetic. • View interpolation –from a few photos, interpolate view from any position (morphing). • * Lighting – adding real or synthetic objects to a photographed scene, illuminated with scene lights. • Modelling – extracting models from photos. Lecture Notes #16

6. “Traditional graphics” Lecture Notes #16

7. Computer Vision Lecture Notes #16

8. Vision meets graphics Lecture Notes #16

9. However ... • Vision falls short Lecture Notes #16

10. And ... … so does graphics Lecture Notes #16

11. Image-Based Rendering (IBR) Lecture Notes #16

12. What is an image? • Collection of radiance values • radiance along a ray • 3D position • 2D direction Lecture Notes #16

13. Plenoptic function • Radiance value for all possible rays = plenoptic function. • All possible images are a subset of this plenoptic function. • Too much stuff! • Goal of IBR is generate a continuous representation of the plenoptic function. Lecture Notes #16

14. Plenoptic function • Radiance is constant along a ray (line) • 2D position • 2D direction Once we know one ‘origin’, we know them all Lecture Notes #16

15. What is an image? • Image = rays going through one point • usually restricted to viewing frustum, but can also be panoramic Lecture Notes #16

16. What is an object? • Image = rays going through one point + image plane • 2D function (position on image plane) Lecture Notes #16

17. What is an object? • Outgoing radiance field of an object • 2D function (position on surface) Lecture Notes #16

18. What is an object? • All light leaving the object Lecture Notes #16

19. What is an object? • All light leaving the object • 4D function (2D position + 2D direction) Lecture Notes #16

20. What is an object? • All possible images of an object Lecture Notes #16

21. What is an object? • All possible images of an object Lecture Notes #16

22. What is an object? • We don’t really need the object Lecture Notes #16

23. What is an object? • We don’t really need the object Lecture Notes #16

24. Lumigraph / Light Field • Object is only defined by its radiance field stuff 4D function (Levoy - Cohen et al 96) Lecture Notes #16

25. Lumigraph - capture (Stanford - Levoy et al.) Lecture Notes #16

26. Layered Depth Images (McMillan) • Problem with one photo is “holes” when view point moved: • this is why a number of photos are generally required • fill the holes using intensities of neighbouring pixels • interpolation … Lecture Notes #16

27. Image Based Lighting • Add models or objects to scenes and allow them to be manipulated in the scenes. • Modelling with light allows added objects to be illuminated consistent with image existent lighting - photorealistic inclusions. • We start with the answer by finding the scene illumination. Lecture Notes #16

28. Image Based Lighting (all pictures P. Debevec 98-99) • Real Scene • Goal: place synthetic objects on table Lecture Notes #16

29. Extracting scene lighting • Capture illumination using illumination sphere Lecture Notes #16

30. Real scene Image Based Lighting Lecture Notes #16

31. Image Based Lighting captured illumination field Lecture Notes #16

32. Image Based Lighting light based model synthetic objects local scene Real scene Lecture Notes #16

33. light based model Image Based Lighting • Use renderer - compute effects of synthetic objects on local scene synthetic objects (brdf known) local scene (brdf estimated) Lecture Notes #16

34. Image Based Lighting • Render into the scene background Lecture Notes #16

35. Image Based Lighting • Render synthetic objects Lecture Notes #16

36. Image Based Lighting • Effect of local scene on real scene Lecture Notes #16

37. Image Based Lighting • Add differences to image Lecture Notes #16

38. Reconstruction from images? • Computer Vision related. • Constructs new views and extracts models. • Epipolar geometry • expresses relationships between points in different images. • Difficult to predict full impact in CG. • State of the Art • 3D reconstruction from uncalibrated images. Lecture Notes #16

39. References • You won’t find this material in the major graphics texts since it is new and not yet mainstream but I recommend the following: • The Computer Image, Watt & Policarpo, Addison-Wesley 1998. • SIGGRAPH courses for the past few years, we have some on CD rom and on-line within EdVEC and not immediately available to Informatics machines – ask me if you have a strong interest and need. Lecture Notes #16