Image Understanding

1. Image Understanding

2. Introduction • Computer vision • Give machines the ability to see • The goal is to duplicate the effect of human visual processing • We live in a 3-D world, but camera sensors can only capture 2-D information. • Computer vision is the “flip” side of computer graphics – but much harder!

3. Introduction • Computer vision is composed of: • Image processing • Image analysis • Image understanding

4. Introduction • Image processing • The goal is to present the image to the system in a useful form • image capture and early processing • remove noise • detect luminance differences • detect edges • enhance image

5. Introduction • Image analysis • The goal is to extract useful information from the processed image • identify boundaries • find connected components • label regions • segment parts of objects • group parts together into whole objects

6. Introduction • Image understanding • The goal is to make sense of the information. Draw qualitative, or semantic, conclusions from the quantitative information. • make a decision about the quantitative information • classify the parts • recognize objects • understand the objects’ usage and the meaning of the scene

7. Introduction • Image understanding uses techniques and methods from: • Physics – models of the visual world • Mathematics - statistics and differential calculus • Spatial pattern recognition • Artificial intelligence • Psychophysics

8. Robot: Count all the Chairs Source: Bülthoff, Max Planck Institute for Biological Cybernetics (MPIK), Tübingen, Germany http://www.ercim.org/publication/Ercim_News/enw53/christensen.html

9. Robot: Which One is a Car?

10. The Importance of Context Are these letters “A” or “H”?

11. The Importance of Context Are these letters “A” or “H”?

12. The Importance of Context Hwonmyawrdsocnaoyueradniihtsentsnece?

13. Low-level Representations • Low-level: little knowledge about the world • The data that is manipulated usually resembles the data that is captured. For example, if the image is captured using a CCD camera (2-D), the representation can be described by an image function whose value is brightness depending on 2 parameters: the x-y coordinates of the location of the brightness value.

14. High-level Representations • High-level: incorporate knowledge about the world external to the image • Image may be mapped to a formalized model of the world (model may change dynamically as new information becomes available) • Data to be processed is dramatically reduced: instead of dealing with pixel values, deal with features such as shape, size, relationships, etc • Usually expressed in symbolic form

15. Low-level Mechanisms • Low-level vision only takes us to the sophistication of a very expensive digital camera

16. High-level Mechanisms • High-level vision and perception requires brain functions that we do not fully understand yet

17. High-level Mechanisms Image from https://plus.google.com/107117483540235115863/posts/MBtyGRBvwkH

18. Bottom-up or Top-down? Bottom-up? Top-Down? Information flow Information flow

19. Top-down Control Visual Completion:

20. Top-down Control Visual Completion:

21. Top-down Control Visual Completion:

22. Top-down Control Visual Completion:

23. Expectation and Learning From Palmer (1999)

24. Occlusion Illusion Which semi-circle appears larger?

25. Occlusion Illusion Which semi-circle appears larger?

26. The Human Visual System • Optical information from the eyes is transmitted to the primary visual cortex in the occipital lobe at the back of the head.

27. The Human Visual System • Light enters the eye through the cornea, aqueous humor, lens, and vitreous humor before striking the light-sensitive receptors of the retina. • After striking the retina, light is converted into electrochemical signals that are carried to the brain via the optic nerve. - 20 mm focal length lens - iris controls amount of light entering eye by changing the size of the pupil

28. The Human Visual System image from www.photo.net/photo/edscott/vis00010.htm

29. The Human Visual System • The distribution of rods and cones across the retina is highly uneven • The fovea contains the highest concentration of cones for high visual acuity From Palmer (1999)

30. How much do we really see? +

31. How much do we really see? + If you can read this you must be cheating

32. Change Blindness • Lack of attention to an object causes failure to perceive it • People find it difficult to detect major changes in a scene if those changes occur in objects that are not the focus of attention • Our impression that our visual capabilities give us a rich, complete, and detailed representation of the world around us is a grand illusion!

33. Center-Surround Organization • The receptive field of a neuron in the retina can be described as having a center-surround organization. When light covers the receptive field uniformly, a random pattern of action potentials results. However, if light activates only the central part of the receptive field and not the surrounding area, an elevated response in terms of the firing rate with respect to the random response will result, and the neuron is said to have an on-center/off-surround organization. For this case, light activating only the inhibitory surround will cause a significant decrease in the firing rate. A neuron exhibiting the opposite pattern of activation is said to have an off-center/on-surround organization.

34. Off-Center/On-Surround On-Center/Off-Surround Stimulus Response Response Center-Surround Organization

35. Center-Surround Organization and Contrast Sensitivity low Contrast high 1 10 100 Spatial frequency (cycles per degree)

36. Center-Surround Organization and Contrast Sensitivity low Contrast high 1 10 100 Spatial frequency (cycles per degree)

37. low Contrast high 1 10 100 Spatial frequency (cycles per degree) Center-Surround Organization and Contrast Sensitivity

38. Lateral Inhibition

39. Lateral Inhibition Input light level 10 5 Output perception

40. 6 6 7 2 3 3 Output perception Lateral Inhibition • A biological neural network in which neurons inhibit spatially neighboring neurons. Architecture of first few layers of retina. Input light level 10 10 10 5 5 5 5 10 Receptors -0.2 -0.2 -0.2 -0.2 -0.2 -0.2 -0.2 +1 +1 +1 +1 +1 +1 Output Cells 10-2-2 = 10-2-2 = 10-2-1 = 5-2-1 = 5-1-1 = 5-1-1 =

41. Lateral Inhibition

42. Lateral Inhibition Not much inhibition - - - + - + - - Lots of inhibition Less lateral inhibition in the fovea as compared to the periphery?

43. Simultaneous Contrast • Two regions that have identical spectra result in different color (lightness) perceptions due to the spectra of the surrounding regions • Background color can visibly affect the perceived color of the target

44. Simultaneous Contrast

45. Simultaneous Contrast

46. Simultaneous Contrast Profile Light intensity 10 5 0 left square right square Horizontal position

47. Simultaneous Contrast Light intensity 5 10 10 5 5 10 10 0 0 5 5 0 0 5 10 right square 5 left square 0 5 10 10 5 5 10 10 0 0 5 5 0 0 5 Excitation (+1) -1 -1 -2 -2 -1 -1 -2 -2 0 0 -1 -1 0 0 Left inhibition (-0.2) -2 -2 -1 -1 -2 -2 0 0 -1 -1 0 0 -1 -1 Right inhibition (-0.2) Output (Sum) 2 7 7 2 2 7 8 -2 -1 4 4 -1 -1 4 left square right square

48. Simultaneous Contrast? According to simultaneous contrast theory, the gray cross on the left should appear lighter than the cross on the right, because it is surrounded by dark squares. Instead, it appears darker. Could it be because we prefer to see a gray square floating over a white (black) background, rather than a cross?

49. Lightness Constancy Indoors - 100 units of light total. White paper reflects 90 units, and black ink reflects 10 units. Outdoors - 10,000 units of light total. White paper reflects 9000 units, and black ink reflects 1000 units. Why does the black ink outside (1000 units reflected) look darker than the white page does indoors (only 90 units reflected)?

50. Color Vision image from www.photo.net/photo/edscott/vis00010.htm The objective description of color is that it is the visible portion of the electro-magnetic spectrum.