Binocular disparity and stereopsis
Download
1 / 164

Binocular disparity and Stereopsis - PowerPoint PPT Presentation


  • 211 Views
  • Uploaded on

Binocular disparity and Stereopsis. Bruce Cumming. Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health. Put red lens over left eye, blue lens over right eye Stereo anaglyph by Prof. Michael Greenhalgh, Australian National University (with permission).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Binocular disparity and Stereopsis' - geordi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Binocular disparity and stereopsis

Binocular disparity and Stereopsis

Bruce Cumming

Laboratory of Sensorimotor Research,

National Eye Institute,

National Institutes of Health


Put red lens over left eye, blue lens over right eye

Stereo anaglyph by Prof. Michael Greenhalgh, Australian National University (with permission).


Stereopsis
stereopsis

L

R


Correspondence problem
correspondence problem

left eye’s image

right eye’s image


Random dot patterns
random-dot patterns

  • a completely unnatural stimulus

  • image changes every few ms

  • no recognisable objects e.g. faces

  • each dot has dozens of identical potential matches

  • and yet a clear perception of depth!


Neurons and depth perception
Neurons and depth perception

  • A simple model to generate disparity signals.

  • How neurons reflect this.

  • Some psychophysical limits this explains.

  • Further processing.



Right retina Research Group (with permission)

Receptive Field

Left retina

Fovea


Disparity selective neuron

Research Group (with permission)

*

*

Disparity-selective neuron

Right RF

R

Left RF

L


Basic building block
basic building-block Research Group (with permission)

  • inner product of image with receptive field

Pos(v)


…. Research Group (with permission)

….S= response

+ 1

+ 1

-0.1


=l Research Group (with permission)

=r

Left RF

+

S

+

Right RF


Output (spike rate) Research Group (with permission)

l1

r1

r2

l2

l2

r1

Input (membrane V)


Circuitry for complex cell

BS 2 Research Group (with permission)

BS 3

BS 4

Circuitry for complex cell

left right

binocular simple cells

RF1

BS 1

complex cell

RF2

Cx

If RF2 = -RF 1 in both eyes, then half squaring then summing is equivalent to simply squaring.


Energy model

square the result Research Group (with permission)

sum over many such subunits

add together

energy model

convolution of left eye’s image with jth left receptive field

convolution of right eye’s image with jth right receptive field


L Research Group (with permission)

R

L

R

Right Stimulus Position

Complex cell

Left Stimulus Position

Model

Ohzawa et al, 1990


Disparity selective neuron1
Disparity-selective neuron Research Group (with permission)

Right RF

R

Left RF

L


L Research Group (with permission)

R

L

R

L

R

L

R

Right Stimulus Position

Complex cell

Left Stimulus Position

Model

Ohzawa et al 1990


Disparity selective neuron2

Research Group (with permission)

*

*

Disparity-selective neuron

Right RF

R

Left RF

L


1 Research Group (with permission)

0.5

0

-0.5

-1

Left RF

Right RF

-d

d

0

Correlation

-50

0

50

Disparity (pixels)


1 Research Group (with permission)

0.5

0

-0.5

-1

Patern 1

Patern 2

Patern 3

Patern 4

Correlation

Patern 5

Mean

-50

0

50

Disparity (pixels)


d Research Group (with permission)

-d

0

Disparity


1 Research Group (with permission)

0.5

0

-0.5

-1

Right RF

Left RF

Correlation

0

Disparity


Deangelis ohzawa and freeman 1991
DeAngelis, Ohzawa and Freeman, (1991) Research Group (with permission)

Cat simple cell RF maps


For single subunits simple
For single subunits (simple) Research Group (with permission)

  • Odd symmetric disparity tuning implies phase disparity

  • Even symmetry around non-zero disparity implies position disparity

True for complex cells if:

  • All subunits have same phase disparity

  • All subunits have same position disparity.


Monkey complex cells Research Group (with permission)

duf043

duf065

60

40

Firing rate (spikes/s)

20

0

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

0.8

-1.4

-0.9

-0.4

0.1

0.6

1.1

1.4

Disparity (degrees)


So far
So far: Research Group (with permission)

  • Energy model measures cross-correlation after filtering.

  • V1 contains a bank of filters measuring these correlations after displacements of both phase and position.


Disparity selective neuron3

Research Group (with permission)

*

*

Disparity-selective neuron

Right RF

R

Left RF

L


the neuronal response Research Group (with permission)

cf: 0.06 cpd

80

60

response

[spikes/sec]

40

20

0

0

0.5

1

1.5

2

time [sec]


the neuronal response Research Group (with permission)

cf: 0.06 cpd

80

60

40

20

0

0

0.5

1

1.5

2

response

[spikes/sec]

cf: 0.5 cpd

80

60

40

20

0

0

0.5

1

1.5

2

time [sec]


f1 Research Group (with permission)

relative modulation

corrugation-frequency

[cpd]

relative modulation

cf: 0.06 cpd

80

60

40

f0

20

0

0

0.5

1

1.5

2

response

[spikes/sec]

cf: 0.5 cpd

80

60

40

20

0

0

0.5

1

1.5

2

time [sec]


SD Research Group (with permission)sf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

SDsf [cpd]

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

RM (f1/f0)

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

sf [cpd]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

1/(2πSDrf) [degree-1]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

response

[spikes/sec]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

vertical position [°]

output exponent:

1 2 4

2

1.5

1

corrugation cutoff [cpd]

0.5

0

n=19

r=0.45

0

0.5

1

1.5

2

1/(2*π*SD of RF height)

[degree-1]


Predicted from mean V1 response Research Group (with permission)

(mean ecentricity 3.7º)


Temporal impulse response lgn
Temporal impulse response (LGN) Research Group (with permission)

10ms

Reppas, Usrey and Ried (2000)


drifting luminance grating Research Group (with permission)

80

disparity modulation

1.5

40

40

response [spikes/sec]

tf cutoff for

drifting luminance grating [Hz]

1

20

relative modulation [f1/f0]

0

n=27

0.5

0

0

20

40

0

tf cutoff for

disparity modulation [Hz]

1

10

100

temporal frequency [Hz]

temporal frequency [Hz]

Temporal frequency tuning for contrast and disparity


Summary
Summary Research Group (with permission)

  • We don’t solve the correspondence problem dot-by-dot.

  • Is this enough?


Monocular response
Monocular response Research Group (with permission)

=

×


1 Research Group (with permission)

0.5

0

-0.5

-1

Correlation

-50

0

50

RF Disparity (pixels)


1 Research Group (with permission)

0.5

0

-0.5

-1

Correlation

-50

0

50

RF Disparity (pixels)


Disparity is two dimensional
Disparity is two-dimensional Research Group (with permission)


P’ Research Group (with permission)

P

direction of gaze

nodal point

fovea


Epipolar line Research Group (with permission)


Y Research Group (with permission)

Z

X


Y Research Group (with permission)

Z

X


Y Research Group (with permission)

Z

X


probability density function for disparities encountered during natural viewing

-15

-10

-5

vertical disparity (degrees)

0

5

10

15

-10

0

10

20

30

horizontal disparity (degrees)


probability density function for disparities encountered during natural viewing

-1

-0.5

vertical disparity (degrees)

0

0.5

1

-1

-0.5

0

0.5

1

horizontal disparity (degrees)


-6 during natural viewing

-4

-2

0

vertical disparity

2

4

6

-15

-10

-5

0

5

10

15

20

25

30

horizontal disparity


Preferred 2-D Disparity during natural viewing

0.6

0.6

0.4

0.2

0.0

Vertical Disparity (degrees)

-0.2

-0.4

-0.6

-0.8

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.6

Horizontal Disparity (degrees)


Stevenson and Schor (1997) during natural viewing


Schreiber et at 2001
Schreiber et at (2001) during natural viewing


Searching for matches
Searching for Matches during natural viewing

  • Not a 2-D problem.

  • Vertical extent of RF may be enough to deal with most epipolar lines.


1 during natural viewing

0.5

0

-0.5

-1

-50

0

50

correlation

RF Disparity (pixels)


1 during natural viewing

0.5

0

-0.5

-1

-50

0

50

correlation

RF Disparity (pixels)


Size disparity correlation
Size-disparity correlation during natural viewing

Smallman and MaCleod (1994)

1/(threshold contrast)

Spatial Period of Center Frequency

Disparity range (min)

Binocular phase

Center spatial frequency


Size-disparity correlation (2) during natural viewing

Prince and Eagle (1999)

R

L


Stimulus Disparity during natural viewing

1

0.5

0

-0.5

-1

-50

0

50

correlation

RF Disparity (pixels)


Stimulus Disparity during natural viewing

Stimulus Disparity

1

.5

0

-.5

-1

-50

0

50

correlation

0

50

RF Disparity (pixels)


2 cpd during natural viewing

0.5 cpd

Threshold

2 cpd + 0.5 cpd

2 cpd, half cycle

Farell, Li and McKee (2004)


1 during natural viewing

0.5

0

-0.5

-1

correlation

-50

0

50

Disparity (pixels)


Tsai and victor 2003
Tsai and Victor (2003) during natural viewing


Anti correlated stimuli
anti-correlated stimuli during natural viewing

left eye’s image

right eye’s image

black  white


1 during natural viewing

.5

0

-.5

-1

-50

0

50

correlation

0

50

RF Disparity (pixels)


Energy model simulation

4 during natural viewing

3.5

3

2.5

2

1.5

1

0.5

0

-50

-40

-30

-20

-10

0

10

20

30

40

50

energy model simulation

correlated stimuli

simulated firing rate

anti-correlated stimuli

disparity


200 during natural viewing

180

160

140

120

100

80

60

40

20

0

120

Cell rb313

Cell rb332

100

80

60

Firing Rate (spikes/s )

40

20

0

-0.4

-0.2

0.0

0.2

0.4

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

Disparity (degrees)

Correlated disparity

Anticorrelated disparity


Correlated disparity during natural viewing

Anticorrelated disparity

Energy model prediction

120

100

80

60

Firing Rate (spikes/s )

40

20

0

-0.4

-0.2

0.0

0.2

0.4


Correlated disparity during natural viewing

Anticorrelated disparity

120

Cell rb313

100

80

60

weaker response for anti-correlatedstimuli

Firing Rate (spikes/s )

40

20

0

-0.4

-0.2

0.0

0.2

0.4


What the energy model gets wrong
what the energy model gets wrong during natural viewing

 quantitative response to anticorrelation

  • real cells respond more weakly to anticorrelated stimuli


Monocular stimuli

left during natural viewing right

monocular stimuli

35

30

25

20

firing rate (spikes / s)

15

10

5

0


This cell is monocular

left during natural viewingright

“this cell is monocular”

35

30

25

20

firing rate (spikes / s)

15

10

5

0


Disparity tuning curve

35 during natural viewing

left right

left right

disparity tuning curve

30

25

20

firing rate (spikes / s)

15

10

left right

5

0

-1.5

-1

-0.5

0

0.5

1

disparity (degrees)


Left eye has purely inhibitory effect
left eye has purely inhibitory effect during natural viewing

35

30

25

-

20

firing rate (spikes / s)

15

10

5

0

-1.5

-1

-0.5

0

0.5

1

disparity (degrees)


1 during natural viewing

0.8

0.6

Disparity Discrimination Index (DDI)

0.4

0.2

0

0

0.2

0.4

0.6

0.8

1

Ocular Dominance Index


but -! during natural viewing

  • this isn’t possible in the energy model.


=ON region of RF during natural viewing

=OFF region of RF

the energy model says that each eye sends both excitatory and inhibitory input

receptive fields

BS


=ON region of RF during natural viewing

=OFF region of RF

the energy model says that each eye sends both excitatory and inhibitory input

receptive fields

BS


Disparity tuning curve1

35 during natural viewing

Inhibition from left eye

left right

disparity tuning curve

30

25

left right

20

firing rate (spikes / s)

15

10

uncorrelated

left right

5

0

-1.5

-1

-0.5

0

0.5

1

disparity (degrees)


Response rates to random dots during natural viewing

140

Ideal monocular neuron

120

100

80

Ideal binocular neuron

monocular

60

40

20

0

0

20

40

60

80

100

120

binocular uncorrelated


What the energy model gets wrong1
what the energy model gets wrong during natural viewing

 quantitative response to anticorrelation

  • real cells respond more weakly to anticorrelated stimuli

     cells where one eye always inhibits firing

  • not possible within the energy model


Energy model1

D = 2 during natural viewingL* R

energy model:

  • disparity tuning curve is the cross-correlation of the left and right eye’s receptive fields.

C = [vL+vR]2 = vL2 + vR2 + 2 vLvR


disparity tuning curve during natural viewing

left eye’s receptive field

right eye’s receptive field

0.35

0.35


How to test

Cx during natural viewing

S

S

S

how to test

  • measure receptive fields?


How to test1

Cx during natural viewing

S

S

S

how to test

  • measure receptive fields?

  • not possible for complex cells.

  • make the comparison in Fourier space.

  • this works for simple and complex cells.


Energy model2
energy model: during natural viewing

  • disparity tuning curve is the cross-correlation of the left and right eye’s receptive fields:

    D = 2 L* R

  • the Fourier power spectrum of the disparity tuning curve is the product of the Fourier amplitude spectra of the left and right eye’s receptive fields:

    FT2(D) = 2 FT(L)FT(R)


0 during natural viewing

0

2

4

6

8

-4

x 10

2

1

0

-1

-2

0

0

2

4

6

8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

position

spatial frequency tuning curve

receptive field (RF)

firing rate

spatial frequency

RF cross-section

Fourier spectrum

Fourier transform

amplitude

spatial frequency


If the energy model is right

0 during natural viewing

2

4

6

8

if the energy model is right:

  • then by obtaining the cell’s spatial frequency tuning….

  • we obtain the Fourier amplitude spectrum of the RF profile.

firing rate

spatial frequency


2 during natural viewing

2.5

1.5

2

1

1.5

4

1

0.5

3

0.5

0

0

2

4

6

8

2

0

0

2

4

6

8

1

0

0

2

4

6

8

monocular spatial frequency tuning curves

left eye

right eye

x

spatial frequency

spatial frequency

Fourier spectrum of disparity tuning curve

=

spatial frequency


Spatial frequency tuning

60 during natural viewing

50

40

firing rate (spikes/s)

30

20

10

0

0

0.1

1

10

0.1

1

10

spatial frequency (cycles per degree)

spatial frequency (cycles per degree)

spatial frequency tuning

left eye

right eye


Product of fitted monocular spatial frequency tuning curves

60 during natural viewing

0.05

50

0.04

0

40

30

0.03

20

0.02

10

0

product of fitted monocular spatial frequency tuning curves

0.01

0.1

0.2

0.5

1

2.5

5

10

15

spatial frequency (cycles per degree)


Disparity tuning curve2

100 during natural viewing

90

80

70

60

50

40

30

20

10

0

-1.5

-1

-0.5

0

0.5

1

1.5

disparity tuning curve

firing rate (spikes/s)

disparity (degrees)


60 during natural viewing

0.05

100

50

90

0.04

80

40

70

60

30

0.03

50

20

40

0.02

30

10

20

10

0

0.1

1

10

0

-1.5

-1

-0.5

0

0.5

1

1.5

firing rate (spikes/s)

firing rate (spikes/s)

disparity (degrees)

spatial frequency (cycles/degree)

0.05

0.04

0.03

normalized units

0.02

0.01

0

0.02

0.05

0.1

0.2

0.5

1

2.5

5

10

15

spatial frequency (cycles per degree)


Peak frequencies differ during natural viewing

too much power at DC

product of fitted spatial frequency tuning curves

0.05

0.04

Fourier transform of fitted disparity tuning curve (minus baseline)

0.03

normalized units

0.02

0.01

0

0.01

0.02

0.05

0.1

0.2

0.5

1

2.5

5

10

15


40 during natural viewing

duf065

Right

35

60

Left

30

25

40

20

Firing rate (spikes/s)

15

20

10

5.0

0.0

0

-1.4

-0.9

-0.4

0.1

0.6

1.1

1.4

0.05

0.10

1.0

10

30

Disparity (degrees)

Spatial Frequency (cpd)


duf043 during natural viewing

50

70

60

40

Right

Left

40

Firing rate (spikes/s)

20

20

0

0

0.05

0.10

1.0

10

30

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

0.8

Disparity (degrees)

Spatial Frequency (cpd)


What the energy model gets wrong2
what the energy model gets wrong during natural viewing

 quantitative response to anticorrelation

  • real cells respond more weakly than predicted to anticorrelated stimuli

     suppressive effect from one eye

  • not possible within the energy model

     mismatch between disparity frequency and response to gratings

  • real disparity tuning curves have more power at low frequencies than predicted


How can we fix the problem
how can we fix the problem? during natural viewing

  • one simple modification to the energy model.

  • keeps all the successes of the energy model.

  • but fixes all these problems at a stroke!


energy model during natural viewing

disparity-selective complex cell

receptive fields

images

binocular simple cell

BS

Cx


Read s modified version

= half-wave rectification during natural viewing

Read’s modified version

monocular simple cells

disparity-selective complex cell

receptive fields

images

binocular simple cell

MS

BS

Cx

MS


Read s modified version1
Read’s modified version during natural viewing

monocular simple cells

disparity-selective complex cell

receptive fields

images

binocular simple cell

MS

BS

Cx

MS


Suppression from one eye

input purely inhibitory during natural viewing

cell never fires

suppression from one eye

monocular simple cells

disparity-selective complex cell

receptive fields

images

binocular simple cell

MS

BS

Cx

MS


Problems our model solves
problems our model solves during natural viewing

  • suppressive effect from one eye

    • inhibitory synapse after monocular simple cell


0.05 during natural viewing

60

0.05

100

50

90

0.04

0.04

80

40

70

0.03

60

30

0.03

50

20

0.02

40

0.02

30

10

20

0.01

10

0

0.1

1

10

0

0

-1.5

-1

-0.5

0

0.5

1

1.5

0.02

0.05

0.1

0.2

0.5

1

2.5

5

10

15

firing rate (spikes/s)

firing rate (spikes/s)

disparity (degrees)

spatial frequency (cycles/degree)

normalized units

spatial frequency (cycles per degree)


0.05 during natural viewing

60

0.05

100

50

90

0.04

0.04

80

40

70

0.03

60

30

0.03

50

20

0.02

40

0.02

30

10

20

0.01

10

0

0.1

1

10

0

0

-1.5

-1

-0.5

0

0.5

1

1.5

0.02

0.05

0.1

0.2

0.5

1

2.5

5

10

15

firing rate (spikes/s)

firing rate (spikes/s)

disparity (degrees)

spatial frequency (cycles/degree)

normalized units

spatial frequency (cycles per degree)


100 during natural viewing

when vL and vR are negatively correlated, this tends to be negative

90

80

70

60

50

40

pulling the response down below the uncorrelated level

30

20

10

0

-1.5

-1

-0.5

0

0.5

1

1.5

firing rate (spikes/s)


100 during natural viewing

90

when vL and vR are negatively correlated, this is zero

80

70

60

50

40

30

pushing the response up closer to the uncorrelated level

20

10

0

-1.5

-1

-0.5

0

0.5

1

1.5

firing rate (spikes/s)


energy model our modified version during natural viewing

disparity tuning curve

0

0

-50

0

50

-50

0

50

disparity

disparity

no power at DC

increased power at DC

Fourier power spectrum

0

0

0

0.02

0.04

0.06

0

0.02

0.04

0.06

spatial frequency

spatial frequency


Threshold at zero
threshold at zero during natural viewing

monocular simple cells

receptive fields

binocular simple cell

complex cell

MS

BS

Cx

MS


Increased threshold
increased threshold during natural viewing

monocular simple cells

receptive fields

binocular simple cell

complex cell

MS

BS

Cx

MS


energy model our modified version during natural viewing

zero threshold

high threshold

disparity tuning curve

0

0

0

-50

0

50

-50

0

50

-50

0

50

disparity

disparity

disparity

no power at DC

increased power at DC

maximum power at DC

Fourier power spectrum

0

0

0

0

0.02

0.04

0.06

0

0.02

0.04

0.06

0

0.02

0.04

0.06

spatial frequency

spatial frequency

spatial frequency


Problems our model solves1
problems our model solves during natural viewing

  • suppressive effect from one eye

    • inhibitory synapse after monocular simple cell

  • mismatch between disparity frequency and response to gratings

    • threshold boosts power at low frequencies


Anticorrelation
anticorrelation during natural viewing

 image in one eye replaced with negative

 one of the convolutions changes sign

 disparity-modulated term inverts; amplitude unchanged:

 a consequence of the linearity of the model


Modified model
modified model during natural viewing

anticorrelation: convolution changes sign

clearly disparity-modulated term no longer simply inverts


MS during natural viewing

MS

-40

-20

0

20

40

Disparity

Energy model

Modified model

2

1.5

Response

1

0.5

0

-40

-40

-20

0

20

40

-20

0

20

40

Disparity

Disparity


Problems our model solves2
problems our model solves during natural viewing

  • suppressive effect from one eye

    • inhibitory synapse after monocular simple cell

  • mismatch between disparity frequency and response to gratings

    • threshold boosts power at low frequencies

  • quantitative response to anticorrelation

    • with high enough thresholds, arbitrarily low amplitude ratios can be obtained


Heterogeneity
heterogeneity during natural viewing

  • real neurons vary greatly in behavior.

  • some well-described by energy model.

  • complex cells have many binocular subunits:

  • perhaps some are like the energy model

    • linear binocular combination

  • others are like our modified version

    • threshold prior to binocular combination


Heterogeneity1

MS during natural viewing

BS

MS

BS

heterogeneity

some binocular subunits as in our model…

Cx

…others as in the original energy model

complex cells receive input from many binocular subunits.


Summary1
summary during natural viewing

  • the energy model gives a good qualitative account of disparity-tuned neurons.

  • it has been widely used in computational models.

  • there are a number of discrepancies when it is compared with quantitative data.


Summary2
summary during natural viewing

  • A simple, plausible modification removes these discrepancies.

  • Consequences for models of later processing largely unexplored.


Extrastriate cortex
Extrastriate cortex during natural viewing

  • V2, V4, MT, and MST all show responses to anticorrelated RDS, like V1.

  • IT does not response to anticorrelated RDS.

  • Does the solution have to be represented explicitly?


Conclusion
conclusion during natural viewing

  • Good understanding of the mechanisms of disparity selectivity in primary visual cortex, without invoking complex network interactions.

  • provides a firm basis for understanding the computations enabling stereo vision.


Put red lens over left eye, blue lens over right eye during natural viewing

Stereo anaglyph by Prof. Michael Greenhalgh, Australian National University (with permission).


Plus a prediction
plus… a prediction during natural viewing

  • Consider case where convolutions are equal and opposite: vL=-vR

  • Original energy model: they cancel out

  • Our version: no cancellation


Disparate drifting grating
disparate drifting grating during natural viewing

right eye

left eye


Typical simple cell response
typical simple cell response during natural viewing

  • one burst of firing per cycle of the stimulus.

firing rate

time (one stimulus cycle)


Phase difference 0 o
phase difference 0 during natural viewingo

right eye

MS

BS

MS

left eye


Phase difference 0 o1
phase difference 0 during natural viewingo

…half a cycle later

right eye

MS

BS

MS

MS

left eye


Phase difference 180 o
phase difference 180 during natural viewingo

right eye

MS

BS

+

MS

left eye


Phase difference 180 o1
phase difference 180 during natural viewingo

…half a cycle later

right eye

+

MS

BS

MS

left eye


o during natural viewing

0

o

60

o

120

interocular phase difference

o

180

o

240

o

300

time (2 stimulus periods)

energy model modified version


o during natural viewing

0

o

60

o

120

interocular phase difference

o

180

o

240

o

300

time (2 stimulus periods)

energy model modified version


-180 during natural viewingo

-90o

0o

90o

180o

interocular phase difference

spikes / s

time (1 stimulus period)


Summary3
summary during natural viewing

  • we postulate that some binocular simple cells receive input via monocular simple cells.

  • straightforward, physiologically plausible mechanism.

  • extends our repertoire so that we can account for all known observations.

  • even predicted something before it was observed!


Stereo anaglyph by Michael Greenhalgh, Australian National University.

Put red lens over left eye, blue lens over right eye


Long term goal of our work
long-term goal of our work University.

to understand:

  • the algorithm the brain uses for stereoscopic depth perception.

  • how this algorithm is implemented physiologically.

  • where this occurs within the brain.



1 University.

0.5

0

-0.5

-1

Patern 1

Patern 2

Patern 3

Patern 4

Correlation

Patern 5

Mean

-50

0

50

Disparity (pixels)


1 University.

0.5

0

-0.5

-1

Left RF

Right RF

-90

90

0

Correlation

90

0

-90

Disparity


Z University.

Y

“straight ahead”

yL

X

“straight ahead”

yR

xL

L

xR

R


1 University.

0.5

0

-0.5

-1

Patern 1

Patern 2

Patern 3

Patern 4

Correlation

Patern 5

Mean

-50

0

50

Disparity (pixels)


Basic building block1
basic building-block University.

  • inner product of image with receptive field

“ON” region

Pos(v)


Basic building block2

“OFF” region University.

basic building-block

  • inner product of image with receptive field

Pos(v)


Right RF University.

Left RF

d

0


Energy model simulation1

-50 University.

-40

-30

-20

-10

0

10

20

30

40

50

energy model simulation

correlated stimuli

simulated firing rate

uncorrelated stimuli

disparity


Shape of disparity tuning
shape of disparity tuning University.

  • a key prediction of the energy model.

  • demonstrating this result would be strong evidence for the energy model.


5.0 University.

4.0

3.0

Preferred Grating Frequency (cpd)

2.0

1.0

0.0

0.0

0.5

1.0

1.5

2.0

2.5

Disparity Frequency (cpd)


5.0 University.

4.0

3.0

Preferred Grating Frequency (cpd)

2.0

1.0

0.0

0.0

0.5

1.0

1.5

2.0

2.5

Disparity Frequency (cpd)


Peak University.

low

0.05

0.04

0.03

normalized units

0.02

0.01

0

0.01

0.02

0.05

0.1

0.2

0.5

1

2.5

5

10

15


Response at 0.05cpd University.

Response at peak

Peak frequency

5.0

1.0

4.0

0.8

3.0

0.6

Monocular grating

2.0

0.4

1.0

0.2

0.0

0.0

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.2

0.4

0.6

0.8

1.0

FT of disparity tuning


70 University.

45

40

60

35

50

30

40

25

20

30

3

15

2.5

20

2

10

1.5

1

10

5

0.5

0

0

0

-0.5

-1.5

-1

-0.5

0

0.5

1

0.1

1

10

0

0.5

1

1.5

2

2.5

3

3.5

4

firing rate (spikes/s)

firing rate (spikes/s)

disparity (degrees)

spatial frequency (cycles/degree)

normalized units

spatial frequency (cycles per degree)


1 University.

.5

0

-.5

-1

-50

0

50

correlation

0

50

RF Disparity (pixels)


ad