Synthetic aperture confocal imaging

1. 21:00 stopping video after scattering, ~22:00 in actual delivery21:00 stopping video after scattering, ~22:00 in actual delivery

2. Outline technologies large camera arrays large projector arrays camera–projector arrays optical effects synthetic aperture photography synthetic aperture illumination synthetic confocal imaging 0:30, 1:15 for entire zoomin0:30, 1:15 for entire zoomin

3. Stanford Multi-Camera Array[Wilburn 2002] 640 × 480 pixels ×30fps × 128 cameras synchronized timing continuous video streaming flexible physical arrangement 0:15 flexible physical arrangement gives us a lot of mileage0:15 flexible physical arrangement gives us a lot of mileage

4. Ways to use large camera arrays widely spaced light field capture tightly packed high-performance imaging intermediate spacing synthetic aperture photography 0:30 if the cameras are tightly packed relative to the distance to the scene then we can think of the array as a single virtual camera of high performance along one or more imaging dimensions, such as resolution, speed, dynamic range, and so on papers at CVPR what I’ll be talking about today...intermediate spacing where the cameras essentially sample a synthetic aperture0:30 if the cameras are tightly packed relative to the distance to the scene then we can think of the array as a single virtual camera of high performance along one or more imaging dimensions, such as resolution, speed, dynamic range, and so on papers at CVPR what I’ll be talking about today...intermediate spacing where the cameras essentially sample a synthetic aperture

5. Synthetic aperture photography 2:00, 3:45 for SAP2:00, 3:45 for SAP

6. Synthetic aperture photography

7. Synthetic aperture photography Taken to the extreme objects off the focal plane become so blurry that they effectively disappear at least if they are smaller than the aperture Leonardo noticed 500 years ago a needle placed in front of his eye because it was smaller than his pupil did not occlude his vision Taken to the extreme objects off the focal plane become so blurry that they effectively disappear at least if they are smaller than the aperture Leonardo noticed 500 years ago a needle placed in front of his eye because it was smaller than his pupil did not occlude his vision

8. Synthetic aperture photography

9. Synthetic aperture photography

10. Synthetic aperture photography Another way to think about synthetic aperture photography take the images from all the cameras, rectify them to a common plane shift them by a certain amount, and add them together Objects that become aligned by the shifting process will be sharply focused objects in front of that plane... objects in back of that plane... Another way to think about synthetic aperture photography take the images from all the cameras, rectify them to a common plane shift them by a certain amount, and add them together Objects that become aligned by the shifting process will be sharply focused objects in front of that plane... objects in back of that plane...

11. Related work not like synthetic aperture radar (SAR) more like X-ray tomosynthesis [Levoy and Hanrahan, 1996] [Isaksen, McMillan, Gortler, 2000] 0:300:30

12. Example using 45 cameras 0:300:30

13. Synthetic pull-focus

14. Crowd scene 0:150:15

15. Crowd scene

16. Synthetic aperture photographyusing an array of mirrors 11-megapixel camera 22 planar mirrors 0:300:30

17.

18.

19. Synthetic aperture illumation 0:45, 1:15 for SAI if you replace cameras with projectors many of the principles I’ve talked about still apply in particular, an array of projectors, suitably aligned, produces a real image in space with such a shallow depth of field that if you stick your hand astride the image, or a white angled screen in this case, the image is focused only at one place on your hand, it’s blurry in front of and behind that point in this case I’m using only a few projectors, so there’s some aliasing off the focal plane if I had enough projectors, it would be a smooth blur0:45, 1:15 for SAI if you replace cameras with projectors many of the principles I’ve talked about still apply in particular, an array of projectors, suitably aligned, produces a real image in space with such a shallow depth of field that if you stick your hand astride the image, or a white angled screen in this case, the image is focused only at one place on your hand, it’s blurry in front of and behind that point in this case I’m using only a few projectors, so there’s some aliasing off the focal plane if I had enough projectors, it would be a smooth blur

20. Synthetic aperture illumation technologies array of projectors array of microprojectors single projector + array of mirrors applications bright display autostereoscopic display [Matusik 2004] confocal imaging [this paper] 0:30 single projector + array of mirrors as we do in this paper bright display we wear sunglasses in our laboratory when looking at these images autostereoscopic display i.e. a light field display0:30 single projector + array of mirrors as we do in this paper bright display we wear sunglasses in our laboratory when looking at these images autostereoscopic display i.e. a light field display

21. Confocal scanning microscopy 1:00, 5:15 up to and including patterns at this point I need to back up and tell you a bit about confocal microscopy if you introduce a pinhole only one point on the focal plane will be illuminated1:00, 5:15 up to and including patterns at this point I need to back up and tell you a bit about confocal microscopy if you introduce a pinhole only one point on the focal plane will be illuminated

22. Confocal scanning microscopy ...and a matching optical system, hence the word confocal this green dot will be both strongly illuminated and sharply imaged while this red dot will have less light falling on it by the square of distance r, because the light is spread over a disk and it will also be more weakly imaged by the square of distance r, because its image is blurred out over an disk on the pinhole mask, and only a little bit is permitted through so the extent to which the red dot contributes to the final image falls off as the fourth power of r, the distance from the focal plane...and a matching optical system, hence the word confocal this green dot will be both strongly illuminated and sharply imaged while this red dot will have less light falling on it by the square of distance r, because the light is spread over a disk and it will also be more weakly imaged by the square of distance r, because its image is blurred out over an disk on the pinhole mask, and only a little bit is permitted through so the extent to which the red dot contributes to the final image falls off as the fourth power of r, the distance from the focal plane

23. Confocal scanning microscopy of course, you’ve only imaged one point so you need to move the pinholes and scan across the focal planeof course, you’ve only imaged one point so you need to move the pinholes and scan across the focal plane

24. Confocal scanning microscopy

25. 0:15 the object in the lower-right image is actually spherical, but portions of it that are off the focal plane are both blurry and dark, effectively disappearing0:15 the object in the lower-right image is actually spherical, but portions of it that are off the focal plane are both blurry and dark, effectively disappearing

26. Synthetic confocal scanning 1:30 our goal is to approximate this effect at the large scale we can understand the photometry of this setup using a simplified counting argument if we had 5 cameras then the ratio would be 25:1 instead of 5:11:30 our goal is to approximate this effect at the large scale we can understand the photometry of this setup using a simplified counting argument if we had 5 cameras then the ratio would be 25:1 instead of 5:1

27. Synthetic confocal scanning 5:0 or 5:1 if we had 5 cameras as well as 5 projectors, then the ratio would be 25:0 or 25:15:0 or 5:1 if we had 5 cameras as well as 5 projectors, then the ratio would be 25:0 or 25:1

28. Synthetic confocal scanning works with any number of projectors = 2 discrimination degrades if point to left of no discrimination for points to left of slow! poor light efficiency depth of field a microscoper would call it the axial resolution to make the depth of field shallower, spread out the projectors, i.e. a larger synthetic aperturedepth of field a microscoper would call it the axial resolution to make the depth of field shallower, spread out the projectors, i.e. a larger synthetic aperture

29. Synthetic coded-apertureconfocal imaging different from coded aperture imaging in astronomy [Wilson, Confocal Microscopy by Aperture Correlation, 1996] 2:152:15

30. Synthetic coded-apertureconfocal imaging

31. Synthetic coded-apertureconfocal imaging

32. Synthetic coded-apertureconfocal imaging

33. Synthetic coded-apertureconfocal imaging

34. Synthetic coded-apertureconfocal imaging

35. Synthetic coded-apertureconfocal imaging note all the tildas in the formulas this algorithm is statistical in nature for example, if we flip a coin to decide whether to illumninate a particular tile on a particular trial the binomial theorem tells us how much variability we’ll get over a given number of trials the effect of this variability the image of our focal plane will be slightly non-uniform, and objects off the focal plane won’t be entirely dark after the confocal subtraction but for visual purposes our technique works well with a modest number of trials, like 16 far fewer than would be required to scan out the focal plane, as in the usual confocal scanning algorithm something else about this variability...note all the tildas in the formulas this algorithm is statistical in nature for example, if we flip a coin to decide whether to illumninate a particular tile on a particular trial the binomial theorem tells us how much variability we’ll get over a given number of trials the effect of this variability the image of our focal plane will be slightly non-uniform, and objects off the focal plane won’t be entirely dark after the confocal subtraction but for visual purposes our technique works well with a modest number of trials, like 16 far fewer than would be required to scan out the focal plane, as in the usual confocal scanning algorithm something else about this variability...

36. Synthetic coded-apertureconfocal imaging let’s examine the red dot in isolation if due to this variability it is not entirely dark then we at least want the beams affecting its color, shown here with red dashed lines to be uncorrelated with the beams affecting the color of adjacent points, like this orange one so that objects like this off the focal plane don’t develop objectionable spatial patterns for this to happen we need patterns in which the illumination of different tiles are spatially uncorrelatedlet’s examine the red dot in isolation if due to this variability it is not entirely dark then we at least want the beams affecting its color, shown here with red dashed lines to be uncorrelated with the beams affecting the color of adjacent points, like this orange one so that objects like this off the focal plane don’t develop objectionable spatial patterns for this to happen we need patterns in which the illumination of different tiles are spatially uncorrelated

37. Example pattern 0:150:15

38. Patterns with less aliasing

39. Implementation using an array of mirrors 4:00 up to and including scattering media4:00 up to and including scattering media

40. 4:00 up to and including scattering media4:00 up to and including scattering media

41. Confocal imaging in scattering media small tank too short for attenuation lit by internal reflections 0:30, 1:30 for WHOI setup I observed a confocal effect but it was modest theory said the effect should be stronger indeed, it is stronger as I confirmed this summer...0:30, 1:30 for WHOI setup I observed a confocal effect but it was modest theory said the effect should be stronger indeed, it is stronger as I confirmed this summer...

42. Experiments in a large water tank 1:001:00

43. Experiments in a large water tank 4-foot viewing distance to target surfaces blackened to kill reflections titanium dioxide in filtered water transmissometer to measure turbidity titanium dioxide the stuff in white painttitanium dioxide the stuff in white paint

44. Experiments in a large water tank stray light limits performance one projector suffices if no occluders

45. Seeing through turbid water 0:45, 1:45 for WHOI results this is very turbid water the “attenuation length” (a technical term that roughly translates to “how far you can see clearly”) is about 8 inches and I’m trying to see through 4 feet if you contrast enhance these images, you can see the improvement in signal-to-noise ratio0:45, 1:45 for WHOI results this is very turbid water the “attenuation length” (a technical term that roughly translates to “how far you can see clearly”) is about 8 inches and I’m trying to see through 4 feet if you contrast enhance these images, you can see the improvement in signal-to-noise ratio

46. Application tounderwater exploration 0:30 so I think we can expect video projectors being mounted on future underwater vehicles this is the Hercules remotely operated vehicle exploring the wreck of the Titanic two months ago in the North Atlantic the question is can you produce an overhead view like this, of the Titanic in a single shot taken from far away using shaped illumination rather than by mowing the lawn with the underwater vehicle which is difficult, dangerous, time consuming, and produces a mosaic with parallax errors 0:30 so I think we can expect video projectors being mounted on future underwater vehicles this is the Hercules remotely operated vehicle exploring the wreck of the Titanic two months ago in the North Atlantic the question is can you produce an overhead view like this, of the Titanic in a single shot taken from far away using shaped illumination rather than by mowing the lawn with the underwater vehicle which is difficult, dangerous, time consuming, and produces a mosaic with parallax errors

47. Research challenges in SAP and SAI theory aperture shapes and sampling patterns illumination patterns for confocal imaging optical design How to arrange cameras, projectors,lenses, mirrors, and other optical elements? How to compare the performance of different arrangements (in foliage, underwater,...)? 1:45, 2:45 until end Where do we go from here? sampling patterns meaning – arrangements of cameras or projectors There are many optical design problems here1:45, 2:45 until end Where do we go from here? sampling patterns meaning – arrangements of cameras or projectors There are many optical design problems here

48. Challenges (continued) systems design multi-camera or multi-projector chips communication in camera-projector networks calibration in long-range or mobile settings algorithms tracking and stabilization of moving objects compression of dense multi-view imagery shape from light fields if you agree that arrays of cameras and projectors are a good idea, then How can we build them compactly and inexpensively?if you agree that arrays of cameras and projectors are a good idea, then How can we build them compactly and inexpensively?

49. Challenges (continued) applications of confocal imaging remote sensing and surveillance shape measurement scientific imaging applications of shaped illumination shaped searchlights for surveillance shaped headlamps for driving in bad weather selective lighting of characters for stage and screen one of my favorites is in the theater perform confocal imaging of an actor on a stage, in real-time, using imperceptible structured light generated by an ultra-high-speed projector (such as my co-authors are exhibiting in etech) to produce an moving matte for the actor, then use that matte to throw a spotlight only on the actor, without casting a shadow on the stage, and without creating a pool of light on the ground!one of my favorites is in the theater perform confocal imaging of an actor on a stage, in real-time, using imperceptible structured light generated by an ultra-high-speed projector (such as my co-authors are exhibiting in etech) to produce an moving matte for the actor, then use that matte to throw a spotlight only on the actor, without casting a shadow on the stage, and without creating a pool of light on the ground!

50. Computational imagingin other fields medical imaging rebinning tomography airborne sensing multi-perspective panoramas synthetic aperture radar astronomy coded-aperture imaging multi-telescope imaging 0:30 let me end with this thought... the techniques I’ve presented today arose from a study of computational imaging in other scientific disciplines I think we’ve only scratched the surface here for example, the original idea for light field rendering 10 years ago, creating new perspective images by rebinning other images, was inspired by a technique called rebinning in medical imaging many of the ideas in the present paper came out of studying synthetic aperture radar and coded-aperture imaging in astronomy0:30 let me end with this thought... the techniques I’ve presented today arose from a study of computational imaging in other scientific disciplines I think we’ve only scratched the surface here for example, the original idea for light field rendering 10 years ago, creating new perspective images by rebinning other images, was inspired by a technique called rebinning in medical imaging many of the ideas in the present paper came out of studying synthetic aperture radar and coded-aperture imaging in astronomy

51. Computational imagingin other fields geophysics accoustic array imaging borehole tomography biology confocal microscopy deconvolution microscopy physics optical tomography inverse scattering there are many other fields with interesting computational techniques that might have application in computer graphics or computer visionthere are many other fields with interesting computational techniques that might have application in computer graphics or computer vision

52. The team staff Mark Horowitz Marc Levoy Bennett Wilburn students Billy Chen Vaibhav Vaish Katherine Chou Monica Goyal Neel Joshi Hsiao-Heng Kelin Lee Georg Petschnigg Guillaume Poncin Michael Smulski Augusto Roman collaborators Mark Bolas Ian McDowall Guillermo Sapiro funding Intel Sony Interval Research NSF DARPA 0:300:30

53. Related papers The Light Field Video Camera Bennett Wilburn, Michael Smulski, Hsiao-Heng Kelin Lee, and Mark Horowitz Proc. Media Processors 2002, SPIE Electronic Imaging 2002 Using Plane + Parallax for Calibrating Dense Camera Arrays Vaibhav Vaish, Bennett Wilburn, Neel Joshi,, Marc Levoy Proc. CVPR 2004  High Speed Video Using a Dense Camera Array Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, Mark Horowitz Proc. CVPR 2004  Spatiotemporal Sampling and Interpolation for Dense Camera Arrays Bennett Wilburn, Neel Joshi, Katherine Chou, Marc Levoy, Mark Horowitz ACM Transactions on Graphics (conditionally accepted)