Computational Neuroscience Final Project – Depth Vision

1. Omri Perez 2013 Computational NeuroscienceFinal Project – Depth Vision

2. Intro

3. Pictorial Depth Cues Physiological Depth Cues Motion Parallax Stereoscopic Depth Cues Depth Cues

4. Two Physiological Depth Cues: • Accommodation • Convergence Physiological Depth Cues

5. Accommodation Physiological Depth Cues

6. Accommodation • relaxed lens = far away • accommodating lens = near What must the visual system be able to compute unconsciously? Physiological Depth Cues

7. Convergence Physiological Depth Cues

8. Convergence • small angle of convergence = far away • large angle of convergence = near • What two sensory systems is the brain integrating? • What happens to images closer or farther away from fixation point? Physiological Depth Cues

9. Parallax Motion Depth Cues

10. Parallax • Points at different locations in the visual field move at different speeds depending on their distance from fixation • http://www.youtube.com/watch?v=ktdnA6y27Gk&NR=1 Motion Depth Cues

11. Seeing in Stereo Seeing in Stereo

12. Seeing in Stereo It’s very hard to read words if there are multiple images on your retina It’s very hard to read words if there are multiple images on your retina

13. Seeing in Stereo It’s very hard to read words if there are multiple images on your retina It’s very hard to read words if there are multiple images on your retina But how many images are there on your retinae?

14. Your eyes have a different image on each retina • hold pen at arms length and fixate the spot • how many pens do you see? • which pen matches which eye? Binocular Disparity

15. Your eyes have a different image on each retina • now fixate the pen • how many spots do you see? • which spot matches which eye? Binocular Disparity

16. Binocular disparity is the difference between the two images Binocular Disparity

17. Binocular disparity is the difference between the two images • Disparity depends on where the object is relative to the fixation point: • objects closer than fixation project images that “cross” • objects farther than fixation project images that do not “cross” Binocular Disparity

18. Corresponding retinal points Binocular Disparity

19. Corresponding retinal points Binocular Disparity

20. Corresponding retinal points Binocular Disparity

21. Corresponding retinal points Binocular Disparity

22. Points in space that have corresponding retinal points define a plane called the horopter or Panum’sfusional area Binocular Disparity The Horopter

23. Points not on the horopter will be disparate on the retina (they project images onto non-corresponding points) Binocular Disparity

24. Points not on the horopter will be disparate on the retina (they project images onto non-corresponding points) The nature of the disparity depends on where they are relative to the horopter Binocular Disparity

25. points nearer than horopter have crossed disparity points farther than horopter have uncrossed disparity Binocular Disparity

26. Why don’t we see double vision? Binocular Disparity

27. Why don’t we see double vision? Images with a small enough disparity are fused into a single image Binocular Disparity

28. Why don’t we see double vision? Images with a small enough disparity are fused into a single image The region of space that contains images with close enough disparity to be fused is called Panum’s Area Binocular Disparity

29. Panum’s Area extends just in front of and just behind the horopter Binocular Disparity

30. Our brains interpret crossed and uncrossed disparity as depth That process is called stereoscopic depth perception or simply stereopsis Stereopsis

31. Stereopsis requires that the brain can encode the two retinal images independently Stereopsis

32. Primary visual cortex (V1) has bands of neurons that keep input from the two eyes separate Stereopsis

33. The basic processing unit of depth perception The cortical column consists of a complete set of orientation columns over a cycle of 180º and of right and left dominance columns in the visual cortex. A hypercolumn may be about 1 mm wide. Cortical hyper columns in V1

34. To compute the binocular depth of stereo images Our goal Left Right

35. To emulate the receptive fields of V1 neurons we use the Gabor function. Even symmetry Odd symmetry Simulating receptive fields in v1 Sinus .* 2D Gaussian  Gabor

36. We filter by doing a 2D convolution of the filters with the image. The different results are averaged together. Filtering the images with gabor filters Left Right

37. Estimation of binocular disparity 2D cross correlation (xcorr2) Tip: In most cases, peak cross correlation results in the x axis (columns) between the left and right eye should only be positive!

38. Basic algorithm – maximum of cross correlation

39. Random dot stereogram (RDS)

40. 1. maximum of cross correlation 2. First neuron to fire in a 2D LIF array (winner take all). The input is the cross correlation result. 3. Population vector of 2D LIF array after X simulation steps. Same input. Notice the horizontal smearing in 2 and 3 is because of cross over activity when switching patches 3 variations:

41. Load a pair of stereo images. You can use the supplied function image_loader.m which makes sure the image is in grayscale and has a proper dynamic range. Generate the filters. You can use the supplied function generate_filters.m to generate the array of filters. I urge you to try out different sizes (3rd parameter) than the default ones in the function. Filter each of the two images using the filters from the previous stage. You can use the function filter_with_all_filters.m to do this. Now, using the two filtered images, iterate over patches, calculate the cross correlation matrix and determine the current depth using the methods 1-3 described in the previous slide. You can tweak the overlap of the patches to reduce computation time. Note for methods 2 and 3: a. you can use the supplied function LIF_one_step_multiple_neurons.m to simulate the LIF neurons b. For methods 2 and 3 it is wise to normalize the xcorr2 results, e.g. by dividing by the maximum value or some such normalization. Your task

42. Incorporate all these into a function of the form: result = find_depth_with_LIF( Left_im_name,Right_im_name, method_num,patch_size, use_filters) Where result is the matrix representing the estimated depth (pixel shifts), Left and Right_im_name are the names of the stereo images. method_num is the number of the method (1-3, see above). patch_size the ratio of the patch size to the image dimensions. E.g. in an image that is 640x480 a ratio of 1/15 will produce a patch of size ~43x32. Please note that in matlab the indexing convention is rowsxcols (and not x,y) so the image is actually 480x640. use_filters is a flag that determines whether to filter the images (step 3 in previous slide) before computing the depth map (useful for debugging, however should be set to true when generating the final results). • In addition to the supplied stereo image pairs, you should also generate a left and right random dot stereogram image pair using the supplied function RDS_generator.m together with a mask (I supplied you with an example mask, RDS_Pac-Man_mask.png) • You can find other stereo image pairs online, e.g. http://vasc.ri.cmu.edu/idb/html/stereo/index.html • Bonus: You can add a fourth depth estimation method of our choice. This can be something you read somewhere or an original idea. For example you can use one of the methods 1-3 but change the patch scan so it won’t be an orderly right to left then down one row and right to left. Instead it can be a random scan which, among other things, will cause several regions to be left uncalculated but other regions more tightly sampled. Your task - continued

43. Your code together with any images (regular and RDS) you used and the supplied images. A document showing the depth results on the two supplied stereo image pairs and one RDS you generated, for each of the 3 methods . (If you chose to do the bonus then show the results for the bonus method as well). Don’t forget when showing results for the RDS to relate them to the mask used to generate it. The document should contain a concise explanation of what you did, your algorithms and interpretation of the results. What to hand in (e-mail in)

44. The project should be submitted by mail to Omri. Good Luck and a succesful test period (and vacation?) !!! How to hand in