Analysis of contour motions l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 72

Analysis of Contour Motions PowerPoint PPT Presentation


  • 138 Views
  • Uploaded on
  • Presentation posted in: General

Neural Information Processing Systems 2006. Analysis of Contour Motions. Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology. Visual Motion Analysis in Computer Vision. Motion analysis is essential in

Download Presentation

Analysis of Contour Motions

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Analysis of contour motions l.jpg

Neural Information Processing Systems 2006

Analysis of Contour Motions

Ce Liu William T. Freeman Edward H. Adelson

Computer Science and Artificial Intelligence Laboratory

Massachusetts Institute of Technology


Visual motion analysis in computer vision l.jpg

Visual Motion Analysis in Computer Vision

  • Motion analysis is essential in

    • Video processing

    • Geometry reconstruction

    • Object tracking, segmentation and recognition

    • Graphics applications

  • Is motion analysis solved?

  • Do we have good representation for motion analysis?

  • Is it computationally feasible to infer the representation from the raw video data?

  • What is a good representation for motion?


Seemingly simple examples l.jpg

Seemingly Simple Examples

Kanizsa square

From real video


Output from the state of the art optical flow algorithm l.jpg

Output from the State-of-the-Art Optical Flow Algorithm

Kanizsa square

Optical flow field

T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004


Output from the state of the art optical flow algorithm5 l.jpg

Output from the State-of-the-Art Optical Flow Algorithm

Dancer

Optical flow field

T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004


Optical flow representation aperture problem l.jpg

Optical flow representation: aperture problem

Lines

Corners

Flat regions

Spurious junctions

Boundary ownership

Illusory boundaries


Optical flow representation l.jpg

We need motion representation beyond pixel level!

Optical Flow Representation

Lines

Corners

Flat regions

Spurious junctions

Boundary ownership

Illusory boundaries


Layer representation l.jpg

J. Wang & E. H. Adelson 1994

Achieved with the help of spatial segmentation

Y. Weiss & E. H. Adelson 1994

Layer Representation

  • Video is a composite of layers

  • Layer segmentation assumes sufficient textures for each layer to represent motion

  • A true success?


Layer representation9 l.jpg

J. Wang & E. H. Adelson 1994

Layer representation is good, but the existing layer

segmentation algorithms cannot find the right layers

for textureless objects

Achieved with the help of spatial segmentation

Y. Weiss & E. H. Adelson 1994

Layer Representation

  • Video is a composite of layers

  • Layer segmentation assumes sufficient textures for each layer to represent motion

  • A true success?


Challenge textureless objects under occlusion l.jpg

Challenge: Textureless Objects under Occlusion

  • Corners are not always trustworthy (junctions)

  • Flat regions do not always move smoothly (discontinuous at illusory boundaries)

  • How about boundaries?

    • Easy to detect and track for textureless objects

    • Able to handle junctions with illusory boundaries


Analysis of contour motions11 l.jpg

Analysis of Contour Motions

  • Our approach: simultaneous grouping and motion analysis

    • Multi-level contour representation

    • Junctions are appropriated handled

    • Formulate graphical model that favors good contour and motion criteria

    • Inference using importance sampling

  • Contribution

    • An important component in motion analysis toolbox for textureless objects under occlusion


Three levels of contour representation l.jpg

Three Levels of Contour Representation

  • Edgelets: edge particles

  • Boundary fragments: a chain of edgelets with small curvatures

  • Contours: a chain of boundary fragments

Forming boundary fragments: easy (for textureless objects)

Forming contours: hard (the focus of our work)


Overview of our system l.jpg

Overview of our system

1. Extract boundary fragments

2. Edgelet tracking with uncertainty.

3. Boundary grouping and illusory boundary

4. Motion estimation based on the grouping


Forming boundary fragments l.jpg

(a)

(b)

(c)

(d)

Forming Boundary Fragments

  • Boundary fragments extraction in frame 1

    • Steerable filters to obtain edge energy for each orientation band

    • Spatially trace boundary fragments

    • Boundary fragments: lines or curves with small curvature

  • Temporal edgelet tracking with uncertainties

  • Frame 1: edgelet (x, y, q)

  • Frame 2: orientation energy of q

  • A Gaussian pdf is fit with the weight of orientation energy

  • 1D uncertainty of motion (even for T-junctions)


Forming contours boundary fragments grouping l.jpg

Forming Contours: Boundary Fragments Grouping

  • Grouping representation: switch variables(attached to every end of the fragments)

    • Exclusive: one end connects to at most one other end

    • Reversible: if end (i,ti) connects to (j,tj), then (j,tj) connects to (i,ti)

1

Arbitrarily possible connection

A legal contour grouping

0

Reversibility

Another legal contour grouping

1

1

0

0


Local spatial temporal cues for grouping l.jpg

Local Spatial-Temporal Cues for Grouping

Illusory boundaries corresponding to the groupings (generated by spline interpolation)

Motion stimulus


Local spatial temporal cues for grouping a motion similarity l.jpg

Velocity space

Local spatial-temporal cues for grouping: (a) Motion similarity

The grouping with higher motion similarity is favored

KL( ) < KL( )

Motion stimulus


Local spatial temporal cues for grouping b curve smoothness l.jpg

Local spatial-temporal cues for grouping: (b) Curve smoothness

The grouping with smoother and shorter illusory boundary is favored

Motion stimulus


Local spatial temporal cues for grouping c contrast consistency l.jpg

Local spatial-temporal cues for grouping: (c) Contrast consistency

The grouping with consistent local contrast is favored

Motion stimulus


The graphical model for grouping l.jpg

The Graphical Model for Grouping

  • Affinity metric terms

    • (a) Motion similarity

    • (b) Curve smoothness

    • (c) Contrast consistency

  • The graphical model for grouping

affinity

reversibility

no self-intersection


Motion estimation for grouped contours l.jpg

Motion estimation for grouped contours

  • Gaussian MRF (GMRF) within a boundary fragment

  • The motions of two end edgelets are similar if they are grouped together

  • The graphical model of motion: joint Gaussian given the grouping

This problem is solved in early work: Y. Weiss, Interpreting images by propagating Bayesian beliefs, NIPS, 1997.


Inference l.jpg

Inference

  • Two-step inference

    • Grouping (switch variables)

    • Motion based on grouping (easy, least square)

  • Grouping: importance sampling to estimate the marginal of the switch variables

    • Bidirectional proposal density

    • Toss the sample if self-intersection is detected

  • Obtain the optimal grouping from the marginal


Why bidirectional proposal in sampling l.jpg

Why bidirectional proposal in sampling?


Why bidirectional proposal in sampling24 l.jpg

Why bidirectional proposal in sampling?

Affinity metric of the switch variable (darker, thicker means larger affinity)

b1b2: 0.39

b1b3: 0.01

b1b4: 0.60

b4b1: 0.20

b4b2: 0.05

b4b3: 0.85

b2b1: 0.50

b2b3: 0.45

b2b4: 0.05

b3b1: 0.01

b3b2: 0.45

b3b4: 0.54

b1b2: 0.1750

b1b3: 0.0001

b1b4: 0.1200

Bidirectional proposal

Normalized affinity metrics


Why bidirectional proposal in sampling25 l.jpg

Why bidirectional proposal in sampling?

Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

b1b2: 0.39

b1b3: 0.01

b1b4: 0.60

b4b1: 0.20

b4b2: 0.05

b4b3: 0.85

b2b1: 0.50

b2b3: 0.45

b2b4: 0.05

b3b1: 0.01

b3b2: 0.45

b3b4: 0.54

b1b2: 0.62

b1b3: 0.00

b1b4: 0.38

Bidirectional proposal

(Normalized)

Normalized affinity metrics


Example of sampling l.jpg

Motion stimulus

Example of Sampling

Self intersection


Example of sampling27 l.jpg

Motion stimulus

Example of Sampling

A valid grouping


Example of sampling28 l.jpg

Motion stimulus

Example of Sampling

More valid groupings


Example of sampling29 l.jpg

Motion stimulus

Example of Sampling

More valid groupings


From affinity to marginals l.jpg

From Affinity to Marginals

Affinity metric of the switch variable (darker, thicker means larger affinity)

Motion stimulus


From affinity to marginals31 l.jpg

From Affinity to Marginals

Marginal distribution of the switch variable (darker, thicker means larger affinity)

Greedy algorithm to search for the best grouping based on the marginals

Motion stimulus


Experiments l.jpg

Experiments

  • All the results are generated using the same parameter settings

  • Running time depends on the number of boundary fragments, varying from ten seconds to a few minutes in MATLAB


Two moving bars l.jpg

Two Moving Bars

Frame 1


Two moving bars34 l.jpg

Two Moving Bars

Frame 2


Two moving bars35 l.jpg

Two Moving Bars

Extracted boundary fragments. The green circles are the boundary fragment end points.


Two moving bars36 l.jpg

Two Moving Bars

Optical flow from Lucas-Kanade algorithm. The flow vectors are only plotted at the edgelets


Two moving bars37 l.jpg

Two Moving Bars

Estimated motion by our system after grouping


Two moving bars38 l.jpg

Two Moving Bars

Boundary grouping and illusory boundaries (frame 1). The fragments belonging to the same contour are plotted in one color.


Two moving bars39 l.jpg

Two Moving Bars

Boundary grouping and illusory boundaries (frame 2). The fragments belonging to the same contour are plotted in one color.


Kanizsa square l.jpg

Kanizsa Square


Slide41 l.jpg

Frame 1


Slide42 l.jpg

Frame 2


Slide43 l.jpg

Extracted boundary fragments


Slide44 l.jpg

Optical flow from Lucas-Kanade algorithm


Slide45 l.jpg

Estimated motion by our system, after grouping


Slide46 l.jpg

Boundary grouping and illusory boundaries (frame 1)


Slide47 l.jpg

Boundary grouping and illusory boundaries (frame 2)


Dancer l.jpg

Dancer


Slide49 l.jpg

Frame 1


Slide50 l.jpg

Frame 2


Slide51 l.jpg

Extracted boundary fragments


Slide52 l.jpg

Optical flow from Lucas-Kanade algorithm


Slide53 l.jpg

Estimated motion by our system, after grouping


Slide54 l.jpg

Lucas-Kanade flow field

Estimated motion by our system, after grouping


Slide55 l.jpg

Boundary grouping and illusory boundaries

(frame 1)


Slide56 l.jpg

Boundary grouping and illusory boundaries

(frame 2)


Rotating chair l.jpg

Rotating Chair


Slide58 l.jpg

Frame 1


Slide59 l.jpg

Frame 2


Slide60 l.jpg

Extracted boundary fragments


Slide61 l.jpg

Estimated flow field from Brox et al.


Slide62 l.jpg

Estimated motion by our system, after grouping


Slide63 l.jpg

Boundary grouping and illusory boundaries (frame 1)


Slide64 l.jpg

Boundary grouping and illusory boundaries (frame 2)


Conclusion l.jpg

Conclusion

  • A contour-based representation to estimate motion for textureless objects under occlusion

  • Motion ambiguities are preserved and resolved through appropriate contour grouping

  • An important component in motion analysis toolbox

  • To be combined with the classical motion estimation techniques to analyze complex scenes


Thanks l.jpg

Thanks!

Analysis of Contour Motions

Ce Liu William T. Freeman Edward H. Adelson

Computer Science and Artificial Intelligence Laboratory

Massachusetts Institute of Technology

http://people.csail.mit.edu/celiu/contourmotions/


Backup slides l.jpg

Backup Slides


Why bidirectional proposal in sampling68 l.jpg

Why bidirectional proposal in sampling?


Why bidirectional proposal in sampling69 l.jpg

Why bidirectional proposal in sampling?

Affinity metric of the switch variable (darker, thicker means larger affinity)

b1b2: 0.39

b1b3: 0.01

b1b4: 0.60

b4b1: 0.20

b4b2: 0.05

b4b3: 0.85

b2b1: 0.50

b2b3: 0.45

b2b4: 0.05

b3b1: 0.01

b3b2: 0.45

b3b4: 0.54

b1b2: 0.1750

b1b3: 0.0001

b1b4: 0.1200

Bidirectional proposal

Normalized affinity metrics


Why bidirectional proposal in sampling70 l.jpg

Why bidirectional proposal in sampling?

Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

b1b2: 0.39

b1b3: 0.01

b1b4: 0.60

b4b1: 0.20

b4b2: 0.05

b4b3: 0.85

b2b1: 0.50

b2b3: 0.45

b2b4: 0.05

b3b1: 0.01

b3b2: 0.45

b3b4: 0.54

b1b2: 0.62

b1b3: 0.00

b1b4: 0.38

Bidirectional proposal

(Normalized)

Normalized affinity metrics


Sampling grouping switch variables l.jpg

Motion stimulus

Sampling Grouping (Switch Variables)


Slide72 l.jpg

Lucas-Kanade flow field

Estimated motion by our system, after grouping


  • Login