Iterative image registration lucas kanade revisited
Download
1 / 96

Iterative Image Registration: Lucas & Kanade Revisited - PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on

Iterative Image Registration: Lucas & Kanade Revisited. Kentaro Toyama Vision Technology Group Microsoft Research. Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Iterative Image Registration: Lucas & Kanade Revisited' - phoebe-stanley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Iterative image registration lucas kanade revisited

Iterative Image Registration:Lucas & Kanade Revisited

Kentaro Toyama

Vision Technology Group

Microsoft Research


Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges


History

  • Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

  • Shi & Tomasi (CVPR 1994)

  • Szeliski & Coughlan (CVPR 1994)

  • Szeliski (WACV 1994)

  • Black & Jepson (ECCV 1996)

  • Hager & Belhumeur (CVPR 1996)

  • Bainbridge-Smith & Lane (IVC 1997)

  • Gleicher (CVPR 1997)

  • Sclaroff & Isidoro (ICCV 1998)

  • Cootes, Edwards, & Taylor (ECCV 1998)

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

History

  • Lucas & Kanade (IUW 1981)

LK




Applications1

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK

Applications

  • Stereo


Applications2
Applications

  • Stereo

  • Dense optic flow

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Applications3
Applications

  • Stereo

  • Dense optic flow

  • Image mosaics

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Applications4
Applications

  • Stereo

  • Dense optic flow

  • Image mosaics

  • Tracking

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Applications5
Applications

  • Stereo

  • Dense optic flow

  • Image mosaics

  • Tracking

  • Recognition

?

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Lucas kanade

Derivation

#1

Lucas & Kanade


L k derivation 1

I0(x)

L&K Derivation 1


L k derivation 11

h

I0(x)

I0(x+h)

L&K Derivation 1


L k derivation 12
L&K Derivation 1

h

I0(x)

I(x)


L k derivation 13
L&K Derivation 1

h

I0(x)

I(x)


L k derivation 14
L&K Derivation 1

I0(x)

I(x)

R


L k derivation 15
L&K Derivation 1

I0(x)

I(x)


L k derivation 16
L&K Derivation 1

h0

I0(x)

I(x)


L k derivation 17
L&K Derivation 1

I0(x+h0)

I(x)


L k derivation 18
L&K Derivation 1

I0(x+h1)

I(x)


L k derivation 19
L&K Derivation 1

I0(x+hk)

I(x)


L k derivation 110
L&K Derivation 1

I0(x+hf)

I(x)


Lucas kanade derivation
Lucas & KanadeDerivation

#2


L k derivation 2

E(h) = S [ I(x) - I0(x+h) ]2

E(h) S [ I(x) - I0(x) - hI0’(x) ]2

xeR

xeR

L&K Derivation 2

  • Sum-of-squared-difference (SSD) error


L k derivation 21

SI0’(x)(I(x) - I0(x))

xeR

h

SI0’(x)2

xeR

L&K Derivation 2

S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2]

xeR

=0


Comparison

w(x)[I(x) - I0(x)]

S

I0’(x)

x

h

Sw(x)

x

SI0’(x)[I(x) - I0(x)]

x

h

SI0’(x)2

x

Comparison


Comparison1
Comparison

w(x)[I(x) - I0(x)]

S

I0’(x)

x

h

Sw(x)

x

SI0’(x)[I(x) - I0(x)]

x

h

SI0’(x)2

x



Original
Original

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R


Original1

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK

Original

  • Dimension of image

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

1-dimensional


Generalization 1a

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK

Generalization 1a

  • Dimension of image

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

2D:


Generalization 1b
Generalization 1b

  • Dimension of image

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Homogeneous 2D:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem a

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK

Problem A

Does the iteration converge?


Problem a1
Problem A

Local minima:


Problem a2
Problem A

Local minima:


Problem b

h is undefined if SI0’(x)2 is zero

xeR

Problem B

Zero gradient:

-SI0’(x)(I(x) - I0(x))

h

xeR

SI0’(x)2

xeR

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem b1

?

Problem B

Zero gradient:


Problem b2
Problem B’

Aperture problem:

-S (x)(I(x) - I0(x))

xeR

hy

S 2

xeR

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem b3

?

Problem B’

No gradient along

one direction:


Solutions to a b
Solutions to A & B

  • Possible solutions:

    • Manual intervention

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b1
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b2
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b3
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

    • Reliance on good features

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b4
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

    • Reliance on good features

    • Temporal filtering

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b5
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

    • Reliance on good features

    • Temporal filtering

    • Spatial interpolation / hierarchical estimation

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to a b6
Solutions to A & B

  • Possible solutions:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

    • Reliance on good features

    • Temporal filtering

    • Spatial interpolation / hierarchical estimation

    • Higher-order terms

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Original2
Original

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R


Original3
Original

  • Transformations/warping of image

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Translations:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem c
Problem C

What about other types of motion?


Generalization 2a
Generalization 2a

  • Transformations/warping of image

S

[

]

(

I

(

Ax

h

)

-

I0

(

x

)

E

A, h

)

=

+

2

x

e

R

Affine:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK



Generalization 2b
Generalization 2b

  • Transformations/warping of image

S

[

]

(

I

(

A x

)

-

I0

(

x

)

E

A

)

=

2

x

e

R

Planar perspective:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Generalization 2b1
Generalization 2b

Affine +

Planar perspective:


Generalization 2c
Generalization 2c

  • Transformations/warping of image

S

[

]

(

I

(

f(x, h)

)

-

I0

(

x

)

E

h

)

=

2

x

e

R

Other parametrized transformations

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Generalization 2c1
Generalization 2c

Other parametrized transformations


Problem b4

-SI0’(x)(I(x) - I0(x))

h

xeR

SI0’(x)2

xeR

Generalized aperture problem:

Problem B”

~

-(JTJ)-1 J (I(f(x,h)) - I0(x))

h

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem b5

?

Problem B”

Generalized

aperture problem:


Original4
Original

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R


Original5
Original

  • Image type

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Grayscale images

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Generalization 3
Generalization 3

  • Image type

S

||

||

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Color images

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Original6
Original

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R


Original7
Original

  • Constancy assumption

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Brightness constancy

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem c1
Problem C

What if illumination changes?


Generalization 4a
Generalization 4a

  • Constancy assumption

S

[

]

(

I

(

x

h

)

-

aI0

(

x

)+b

h,a,b

E

)

=

+

2

x

e

R

Linear brightness constancy

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK



Generalization 4b
Generalization 4b

  • Constancy assumption

S

[

]

I

(

x

h

)

-

lTB

(

x

)

h,l

E

(

)

=

+

2

x

e

R

Illumination subspace constancy

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem c2
Problem C’

What if the texture changes?


Generalization 4c
Generalization 4c

  • Constancy assumption

S

[

]

I

(

x

h

)

-

lTB

(

x

)

h,l

E

(

)

=

+

2

x

e

R

Texture subspace constancy

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem d
Problem D

Convergence is slower as #parameters increases.


Solutions to d
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d1
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

    • Selective parametrization

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d2
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

    • Selective parametrization

    • Offline precomputation

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d3
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

    • Selective parametrization

    • Offline precomputation

      • Difference decomposition

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d4
Solutions to D

  • Difference decomposition


Solutions to d5
Solutions to D

  • Difference decomposition


Solutions to d6
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

    • Selective parametrization

    • Offline precomputation

      • Difference decomposition

    • Improvements in gradient descent

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d7
Solutions to D

  • Faster convergence:

    • Coarse-to-fine, filtering, interpolation, etc.

    • Selective parametrization

    • Offline precomputation

      • Difference decomposition

    • Improvements in gradient descent

      • Multiple estimates of spatial derivatives

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Solutions to d8
Solutions to D

  • Multiple estimates / state-space sampling


Generalizations1
Generalizations

Modificationsmade so far:

S

[

]

I

(

x

h

)

-

I0

(

x

)

+

2

x

e

R


Original8
Original

  • Error norm

S

[

]

I0

(

I

(

x

h

)

-

(

x

)

E

h

)

=

+

2

x

e

R

Squared difference:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem e
Problem E

What about outliers?


Generalization 5a
Generalization 5a

  • Error norm

S

(

)

I0

(

r

I

(

x

h

)

-

(

x

)

E

h

)

=

+

x

e

R

Robust error norm:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Original9
Original

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R


Original10
Original

  • Image region / pixel weighting

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

Rectangular:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem e1
Problem E’

What about background clutter?


Generalization 6a
Generalization 6a

  • Image region / pixel weighting

S

[

]

I0

(

I

(

x

h

)

-

(

x

)

E

h

)

=

+

2

x

e

R

Irregular:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Problem e2
Problem E”

What about foreground occlusion?


Generalization 6b
Generalization 6b

  • Image region / pixel weighting

S

[

]

I0

(

I

(

x

h

)

-

(

x

)

w(x)

E

h

)

=

+

2

x

e

R

Weighted sum:

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK


Generalizations2
Generalizations

Modificationsmade so far:

S

[

]

I

(

x

h

)

-

I0

(

x

)

+

2

x

e

R


Generalizations summary
Generalizations: Summary

S

[

]

(

I

(

x

h

)

-

I0

(

x

)

E

h

)

=

+

2

x

e

R

S

(

)

(

I

(

f(x, h)

)

-

(

x

)

r

E

h

)

=

lB

w(x)

x

e

R


Foresight

SC

G

SI

CET

BAHH

ST

S

BJ

HB

BL

LK

Foresight

  • Lucas & Kanade (IUW 1981)

  • Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

  • Shi & Tomasi (CVPR 1994)

  • Szeliski & Coughlan (CVPR 1994)

  • Szeliski (WACV 1994)

  • Black & Jepson (ECCV 1996)

  • Hager & Belhumeur (CVPR 1996)

  • Bainbridge-Smith & Lane (IVC 1997)

  • Gleicher (CVPR 1997)

  • Sclaroff & Isidoro (ICCV 1998)

  • Cootes, Edwards, & Taylor (ECCV 1998)


Summary
Summary

  • Generalizations

    • Dimension of image

    • Image transformations / motion models

    • Pixel type

    • Constancy assumption

    • Error norm

    • Image mask

L&K ?

Y

Y

n

Y

n

Y


Summary1
Summary

  • Common problems:

    • Local minima

    • Aperture effect

    • Illumination changes

    • Convergence issues

    • Outliers and occlusions

L&K ?

Y

maybe

Y

Y

n


Summary2
Summary

  • Mitigation of aperture effect:

    • Manual intervention

    • Zero motion default

    • Coefficient “dampening”

    • Elimination of poor textures

    • Temporal filtering

    • Spatial interpolation / hierarchical

    • Higher-order terms

L&K ?

n

n

n

n

Y

Y

n


Summary3
Summary

  • Better convergence:

    • Coarse-to-fine, filtering, etc.

    • Selective parametrization

    • Offline precomputation

      • Difference decomposition

    • Improvements in gradient descent

      • Multiple estimates of spatial derivatives

L&K ?

Y

n

maybe

maybe

maybe

maybe


Hindsight
Hindsight

  • Lucas & Kanade (IUW 1981)

  • Bergen, Anandan, Hanna, Hingorani (ECCV 1992)

  • Shi & Tomasi (CVPR 1994)

  • Szeliski & Coughlan (CVPR 1994)

  • Szeliski (WACV 1994)

  • Black & Jepson (ECCV 1996)

  • Hager & Belhumeur (CVPR 1996)

  • Bainbridge-Smith & Lane (IVC 1997)

  • Gleicher (CVPR 1997)

  • Sclaroff & Isidoro (ICCV 1998)

  • Cootes, Edwards, & Taylor (ECCV 1998)


ad