Low Level Visual Processing

1. Low Level Visual Processing

2. Information Maximization in the Retina • Hypothesis: ganglion cells try to transmit as much information as possible about the image. • What kind of receptive field maximizes information transfer?

3. Information Maximization in the Retina • In this particular context, information is maximized for a factorial code: • For a factorial code, the mutual information is 0 (there are no redundancies):

4. Information Maximization in the Retina Independence is hard to achieve. Instead, we can look for a code which decorrelates the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.

5. Information Maximization in the Retina We assume that ganglion cells are linear: The goal is to find a receptive field profile, Ds(x), for which the ganglion cells are decorrelated (i.e., a whitening filter).

6. Information Maximization in the Retina • Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean): • The crosscorrelogram is also a convolution

7. Fourier Transform • The Fourier transform of a convolution is equal to the product of the individual spectra: • The spectrum of a Dirac function is flat.

8. Information Maximization in the Retina • To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.

9. Information Maximization in the Retina

10. Information Maximization in the Retina

11. Solution: use a noise filter first, : Information Maximization in the Retina • If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise.

12. Information Maximization in the Retina

13. Information Maximization in the Retina • The shape of the whitening filter depends on the noise level. • For high contrast/low noise: bandpass filter. Center-surround RF. • For low contrast/high noise: low pass filter. Gaussian RF

14. + Information Maximization in the Retina j

15. Information Maximization in the Retina 4 4 3 3 2 2 1 1 0 0 5 0 10 15 0 10 5 (Hz) (H temporal frequency temporal frequency

16. Information Maximization beyond the Retina • The bottleneck argument can only work once… • The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter • Use other constrained beside infomax, such as sparseness.

17. Center Surround Receptive Fields • The center surround receptive fields are decent edge detectors +

18. Center Surround Receptive Fields

19. Feature extraction: Energy Filters

20. 2D Fourier Transform Orientation Frequency

21. 2D Fourier Transform

22. 2D Fourier Transform

23. Motion Energy Filters Time Space

24. Motion Energy Filters • In a space time diagram 1st order motion shows up as diagonal lines. • The slope of the line indicates the velocity • A space-time Fourier transform can therefore recover the speed of motion

25. Motion Energy Filters

26. Motion Energy Filters

28. Motion Energy Filters • 1st order motion Time Space

29. Motion Energy Filters • 2nd order motion

30. Motion Energy Filters • In a space time diagram, 2nd order motion does not show up as a diagonal line… • Methods based on linear filtering of the image followed by a nonlinearity cannot work • You need to apply a nonlinearity to the image first

31. Motion Energy Filters • A Fourier transform returns a set of complex coefficients:

32. Motion Energy Filters • The power spectrum is given by

33. Motion Energy Filters

34. Motion Energy Filters aj cos x2 cj + I bj sin x2

35. Motion Energy Filters aj cos x2 cj + I bj sin x2

36. wt wx

37. Motion Energy Filters • Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle. • Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…

38. Motion Energy Filters • How do you compute a Fourier transform with neurons? Use neurons with spatio-temporal filters looking like oriented sine and cosine functions. • Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…

39. Motion Energy Filters • Solution: use oriented Gabor-like filters or causal version of Gabor-like filters. • To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.

40. Motion Energy Filters x2 + x2

41. wt wx From V1 to MT • V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies

42. From V1 to MT • MT cells are tuned to velocity across a wide range of spatial and temporal frequencies wt wx

43. MT Cells

44. Pooling across Filters • Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.

45. Pooling across Filters + Flicker

46. Energy Filters • For second order motion, apply a nonlinearity to the image and then run a motion energy filter.

47. Motion Processing: Bayesian Approach • The energy filter approach is not the only game in town… • Bayesian integration provides a better account of psychophysical results

48. Energy Filters: Generalization • The same technique can be used to compute orientation, disparity, … etc.

49. Energy Filters: Generalization • The case of stereopsis: constant disparity correspond to oriented line in righ/left RF diagram.

50. Energy Filters: Generalization