Exploring Sensors and Range Data in 3D Computer Vision

1. Sensors CSC I6716 Spring 2004 3D Computer Vision and Video Computing Instructor: Zhigang Zhu Part 1 Topic 2: Sensors http://www-cs.engr.ccny.cuny.edu/~zhu/VisionCourse-2004.html

2. Acknowledgements The slides in this lecture were adopted and modified from lectures by Professor Allen Hanson University of Massachusetts at Amherst

3. Sensors • Static monocular reflectance data (monochromic or color) • Films • Video cameras (with tapes) • Digital cameras (with memory) • Motion sequences (camcorders) • Stereo (2 cameras) • Range data (Range finder) • Non-visual sensory data • infrared (IR) • ultraviolet (UV) • microwaves • Many more

4. The Electromagnetic Spectrum C = f l E  f Visible Spectrum 700 nm 400 nm

5. The Human Eye

6. Rods Cones The Eye Retina • The Retina: • rods (low-level light, night vision) • cones (color-vision) • synapses • optic nerve fibers • Sensing and low-level processing layer • 125 millions rods and cones feed into 1 million nerve fibers • Cell arrangement that respond to horizontal and vertical lines

7. Film, Video, Digital Cameras • Black and White (Reflectance data only) • Color (Reflectance data in three bands - red, green, blue)

8. Blue Green Red Color Images Spatial Resolution Spectra Resolution Radiometric Resolution Temporal Resolution ‘Dimensions’ of an Image Spatial (x,y) Depth (no. of components) Number of bits/channel Temporal (t) Pixel

9. Across the EM Spectrum Crab Nebula

10. Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 1019 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

11. Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 1019 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

12. Across the EM Spectrum Cargo inspection using Gamma Rays Mobile Vehicle and Cargo Inspection System (VACIS®) Gamma rays are typically waves of frequencies greater than 1019 Hz Gamma rays can penetrate nearly all materials and are therefore difficult to detect Courtesy:Science Applications International Corporation (SAIC),

13. Across the EM Spectrum • Medical X-Rays

14. Across the EM Spectrum • Chandra X-Ray Satellite

15. Across the EM Spectrum • From X-Ray images to 3D Models: CT Scans

16. Across the EM Spectrum Dandelion - UV • Flower Patterns in Ultraviolet Potentilla

17. Across the EM Spectrum • Messier 101 in Ultraviolet

18. Across the EM Spectrum • Traditional images

19. Across the EM Spectrum • Non-traditional Use of Visible Light: Range

20. Across the EM Spectrum • Scanning Laser Rangefinder

21. Across the EM Spectrum • IR: Near, Medium, Far (~heat)

22. Across the EM Spectrum • IR: Near, Medium, Far (~heat)

23. Across the EM Spectrum • IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

24. Across the EM Spectrum • (Un)Common uses of Microwaves Exploding Water Movie CD Movie

25. Across the EM Spectrum • Microwave Imaging: Synthetic Aperture Radar (SAR) Tibet: Lhasa River San Fernando Valley Red: L-band (24cm) Green: C-band (6 cm) Blue:C/L Athens, Greece Thailand: Phang Hoei Range

26. Across the EM Spectrum • Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR (elevation)

27. Across the EM Spectrum • Low Altitude IFSAR IFSAR elevation, automatic, in minutes Elevation from aerial stereo, manually, several days

28. Across the EM Spectrum • Radio Waves (images of cosmos from radio telescopes)

29. Stereo Geometry • Single Camera (no stereo)

30. Film plane Film plane pl(x,y) pr(x,y) f = focal length f = focal length Optical Center Optical Center Stereo Geometry LEFT CAMERA RIGHT CAMERA P(X,Y,Z) B = Baseline

31. Stereo Geometry LEFT IMAGE RIGHT IMAGE P Pl(xl,yl) Pr(xr,yr) Disparity = xr - xl ≈ depth

32. Stereo Images • A Short Digression Stereoscopes

33. Stereo Images Darjeeling Suspension Bridge

34. Picture of you?

35. Stereo • Stereograms

36. Stereo X-Ray

37. Range Sensors • Light Striping David B. Cox, Robyn Owens and Peter Hartmann Department of Biochemistry University of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/

38. Mosaics • A mosaic is created from several images

39. Mosaics • Stabilized Video

40. Mosaics • Depth from a Video Sequence (single camera) GPS Height H from Laser Profiler P(X,Y,Z)

41. Mosaics • Brazilian forest…..made at UMass CVL

42. Why is Vision Difficult? • Natural Variation in Object Classes: • Color, texture, size, shape, parts, and relations • Variations in the Imaging Process • Lighting (highlights, shadows, brightness, contrast) • Projective distortion, point of view, occlusion • Noise, sensor and optical characteristics • Massive Amounts of Data • 1 minute of 1024x768 color video = 4.2 gigabytes (Uncompressed)

43. The Need for Knowledge Variation Knowledge Motion Context Function Shape Shape Purpose Specific Objects Generic Objects Structure Size

44. The Figure Revealed

45. The Effect of Context

46. The Effect of Context - 2

47. Context, cont. • ….a collection of objects:

48. Context • The objects as hats:

49. Context • And as something else….. • ‘To interpret something is to give it meaning in context.’

50. Vision System Components • …..at the low (image) level, we need • Ways of generating initial descriptions of the image data • Method for extracting features of these descriptions • Ways of representing these descriptions and features • Usually, cannot initially make use of general world knowledge IMAGE (numbers) DESCRIPTION (symbols)