Light field microscopy

1. Light field microscopy Marc Levoy, Ren Ng, Andrew Adams Matthew Footer, Mark Horowitz Stanford Computer Graphics Laboratory

2. Executive summary • captures the 4D light field inside a microscope • yields perspective flyarounds and focal stacks from a single snapshot, but at lower spatial resolution • focal stack → deconvolution microscopy → volume data

3. bigscenes small scenes Devices for recording light fields (using geometrical optics) • handheld camera [Buehler 2001] • camera gantry [Stanford 2002] • array of cameras [Wilburn 2005] • plenoptic camera [Ng 2005] • light field microscope (this paper)

4. Light fields at micron scales • wave optics must be considered • diffraction limits the spatial × angular resolution • most objects are no longer opaque • each pixel is a line integral through the object • of attenuation • or emission • can reconstruct 3D structure from these integrals • tomography • 3D deconvolution

5. Conventional versus plenoptic camera

6. uv-plane st-plane 125μ square-sided microlenses Conventional versus plenoptic camera

8. Σ Digital refocusing • refocusing = summing windows extracted from several microlenses Σ

9. Example of digital refocusing

10. Refocusing portraits

11. Macrophotography

12. Digitally moving the observer • moving the observer = moving the window we extract from the microlenses Σ Σ

13. Example of moving the observer

14. Moving backward and forward

15. A light field microscope (LFM) eyepiece intermediate image plane objective specimen

16. A light field microscope (LFM) • 40x / 0.95NA objective ↓ 0.26μ spot on specimen× 40x = 10.4μ on sensor ↓ 2400 spots over 25mm field • 1252-micron microlenses ↓ 200 × 200 microlenses with12 × 12 spots per microlens sensor eyepiece intermediate image plane objective specimen → reduced lateral resolution on specimen= 0.26μ × 12 spots = 3.1μ

17. 2.5mm 160mm 0.2mm A light field microscope (LFM) sensor

18. Example light field micrograph • orange fluorescent crayon • mercury-arc source + blue dichroic filter • 16x / 0.5NA (dry) objective • f/20 microlens array • 65mm f/2.8 macro lens at 1:1 • Canon 20D digital camera 200μ ordinary microscope light field microscope

19. f The geometry of the light fieldin a microscope • microscopes make orthographic views • translating the stage in X or Y provides no parallax on the specimen • out-of-plane features don’t shift position when they come into focus objective lenses are telecentric

20. Panning and focusing panning sequence focal stack

21. Mouse embryo lung(16x / 0.5NA water immersion) 200μ pan focal stack light field

22. (wave optics dominates) (geometrical optics dominates) Axial resolution(a.k.a. depth of field) • wave term + geometrical optics term • ordinary microscope (16x/0.4NA (dry), e = 0) • with microlens array (e = 125μ) • stopped down to one pixel per microlens → number of slices in focal stack= 12

23. (UMIC SUNY/Stonybrook) (Noguchi) (DeltaVision) 3D reconstruction • confocal scanning [Minsky 1957] • shape-from-focus [Nayar 1990] • deconvolution microscopy [Agard 1984] • 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data

24.  {PSF} 3D deconvolution [McNally 1999] • object * PSF → focus stack •  {object} × {PSF} →  {focus stack} •  {focus stack}  {PSF} →  {object} • spectrum contains zeros, due to missing rays • imaging noise is amplified by division by ~zeros • reduce by regularization, e.g. smoothing focus stack of a point in 3-space is the 3D PSF of that imaging system

25. Silkworm mouth(40x / 1.3NA oil immersion) 100μ slice of focal stack slice of volume volume rendering

26. volume rendering all-focus image [Agarwala 2004] Insect legs(16x / 0.4NA dry) 200μ

27. (DeltaVision) (from Kak & Slaney) 3D reconstruction (revisited) • 4D light field → digital refocusing →3D focal stack → deconvolution microscopy →3D volume data • 4D light field → tomographic reconstruction →3D volume data

28. Optical Projection Tomography [Sharpe 2002] Implications of this equivalence • light fields of minimally scattering volumes contain only 3D worth of information, not 4D • the extra dimension serves to reduce noise, but could be re-purposed?

29. Conclusions • captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time Calcium fluorescent imaging of zebrafish larvae optic tectum during changing visual stimula

30. Conclusions • captures 3D structure of microscopic objects in a single snapshot, and at a single instant in time but... • sacrifices spatial resolution to obtain control over viewpoint and focus • 3D reconstruction fails if specimen is too thick or too opaque

31. Nikon 40x 0.95NA (dry) Plan-Apo Future work • extending the field of view by correcting digitally for objective aberrations

32. 200μ Future work • extending the field of view by correcting digitally for objective aberrations • microlenses in the illumination path → an imaging microscope scatterometer angular dependence of reflection from single squid iridophore

33. http://graphics.stanford.edu/projects/lfmicroscope