eecs 274 computer vision l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
EECS 274 Computer Vision PowerPoint Presentation
Download Presentation
EECS 274 Computer Vision

Loading in 2 Seconds...

play fullscreen
1 / 63

EECS 274 Computer Vision - PowerPoint PPT Presentation


  • 131 Views
  • Uploaded on

EECS 274 Computer Vision. Motion Estimation. Motion estimation. Aligning images Estimate motion parameters Optical flow Lucas-Kanade algorithm Horn-Schunck algorithm Slides based on Szeliski’s lecture notes Reading: FP Chapter 15, S Chapter 8. Why estimate visual motion?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EECS 274 Computer Vision' - juan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
eecs 274 computer vision

EECS 274 Computer Vision

Motion Estimation

motion estimation
Motion estimation
  • Aligning images
  • Estimate motion parameters
  • Optical flow
    • Lucas-Kanade algorithm
    • Horn-Schunck algorithm
  • Slides based on Szeliski’s lecture notes
  • Reading: FP Chapter 15, S Chapter 8
why estimate visual motion
Why estimate visual motion?
  • Visual Motion can be annoying
    • Camera instabilities, jitter
    • Measure it; remove it (stabilize)
  • Visual Motion indicates dynamics in the scene
    • Moving objects, behavior
    • Track objects and analyze trajectories
  • Visual Motion reveals spatial layout
    • Motion parallax
classes of techniques
Classes of techniques
  • Feature-based methods
    • Extract visual features (corners, textured areas) and track them
    • Sparse motion fields, but possibly robust tracking
    • Suitable especially when image motion is large (10s of pixels)
  • Direct methods
    • Directly recover image motion from spatio-temporal image brightness variations
    • Global motion parameters directly recovered without an intermediate feature motion calculation
    • Dense motion fields, but more sensitive to appearance variations
    • Suitable for video and when image motion is small (< 10 pixels)
optical flow vs motion field
Optical flow vs. motion field
  • Optical flow does not always correspond to motion field
  • Optical flow is an approximation of the motion field
  • The error is small at points with high spatial gradient under some simplifying assumptions
patch matching
Patch matching
  • How do we determine correspondences?
    • block matching or SSD (sum squared differences)
correlation and ssd
Correlation and SSD
  • For larger displacements, do template matching
    • Define a small area around a pixel as the template
    • Match the template against each pixel within a search area in next image
    • Use a match measure such as correlation, normalized correlation, or sum-of-squares difference
    • Choose the maximum (or minimum) as the match
    • Sub-pixel estimate (Lucas-Kanade)
discrete search vs gradient based
Discrete search vs. gradient based
  • Consider image I translated by
  • The discrete search method simply searches for the best estimate
  • The gradient method linearizes the intensity function and solves for the estimate
brightness constancy
Brightness constancy
  • Brightness Constancy Equation:
  • Minimize :
  • Linearize (assuming small (u,v))using Taylor series expansion:
estimating optical flow
Estimating optical flow
  • Assume the image intensity I is constant

Time = t

Time = t+dt

lucas kanade algorithm
Lucas-Kanade algorithm

Assume a single velocity for all pixels within an image patch

Minimizing

Hessian

LHS: sum of the 2x2 outer product of the gradient vector

matrix form14
Matrix form

for computational efficiency

computing gradients in x y t
Computing gradients in X-Y-T

y

time

j+1

k+1

j

k

x

i

i+1

likewise for Iy and It

local patch analysis
Local patch analysis
  • How certain are the motion estimates?
the aperture problem
The aperture problem
  • Algorithm: At each pixel compute by solving
  • Ais singular if all gradient vectors point in the same direction
    • e.g., along an edge
    • of course, trivially singular if the summation is over a single pixel or there is no texture
    • i.e., only normal flow is available (aperture problem)
  • Corners and textured areas are OK
iterative refinement
Iterative refinement
  • Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation
  • Warp one image toward the other using the estimated flow field

(easier said than done)

  • Refine estimate by repeating the process
optical flow iterative estimation

estimate update

Initial guess:

Estimate:

Optical flow: iterative estimation

x

x0

(usingd for displacement here instead of u)

optical flow iterative estimation24

estimate update

Initial guess:

Estimate:

Initial guess:

Estimate:

x

x0

Optical flow: iterative estimation
optical flow iterative estimation26
Optical flow: iterative estimation
  • Some implementation issues:
    • Warping is not easy (ensure that errors in warping are smaller than the estimate refinement)
    • Warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration
    • Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approximations to image intensity)
error functions
Error functions
  • Robust error function
  • Spatially varying weights
  • Bias and gain: images taken with different exposure
  • Correlation (and normalized cross correlation)
horn schunck algorithm
Horn-Schunck algorithm
  • Global method with smoothness constraint to solve aperture problem
  • Minimize a global energy functional with calculus of variations
horn schunck algorithm29
Horn-Schunck algorithm

See Robot Vision by Horn for details

horn schunck algorithm30
Horn-Schunck algorithm
  • Iterative scheme
  • Yields high density flow
  • Fill in missing information in the homogenous regions
  • More sensitive to noise than local methods
optical flow aliasing

actual shift

estimated shift

Optical flow: aliasing

Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity.

I.e., how do we know which ‘correspondence’ is correct?

nearest match is correct (no aliasing)

nearest match is incorrect (aliasing)

To overcome aliasing: coarse-to-fine estimation.

slide32

Jw

refine

warp

+

u=1.25 pixels

u=2.5 pixels

u=5 pixels

u=10 pixels

image J

image J

image I

image I

Pyramid of image J

Pyramid of image I

Coarse-to-fine estimation

slide33

Coarse-to-fine estimation

I

J

refine

J

Jw

I

warp

+

I

J

Jw

pyramid

construction

pyramid

construction

refine

warp

+

J

I

Jw

refine

warp

+

global parametric motion models
Global (parametric) motion models
  • 2D Models:
  • Affine
  • Quadratic
  • Planar projective transform (Homography)
  • 3D Models:
  • Instantaneous camera motion models
  • Homography+epipole
  • Plane+Parallax
motion models

Affine

Perspective

3D rotation

Translation

2 unknowns

6 unknowns

8 unknowns

3 unknowns

Motion models
example affine motion
Example: affine motion
  • Substituting into the B.C. equation

Each pixel provides 1 linear constraint in 6 global unknowns

Least square minimization (over all pixels):

parametric motion
Parametric motion

i.e., the product of image gradient with Jacobian of the correspondence field

parametric motion38
Parametric motion

the expressions inside the brackets are the same as the cases for simpler translation motion case

3d motion models

Instantaneous camera motion:

Global parameters:

Homography+Epipole

Local Parameter:

Global parameters:

Local Parameter:

Residual Planar Parallax Motion

Global parameters:

Local Parameter:

3D Motion Models
shi tomasi feature tracker
Shi-Tomasi feature tracker
  • Find good features (min eigenvalue of 22 Hessian)
  • Use Lucas-Kanade to track with pure translation
  • Use affine registration with first feature patch
  • Terminate tracks whose dissimilarity gets too large
  • Start new tracks when needed
correlation window size
Correlation window size
  • Small windows lead to more false matches
  • Large windows are better this way, but…
    • Neighboring flow vectors will be more correlated (since the template windows have more in common)
    • Flow resolution also lower (same reason)
    • More expensive to compute
  • Small windows are good for local search:more detailed and less smooth (noisy?)
  • Large windows good for global search:less detailed and smoother
robust estimation

velocity space

u2

+

+

u1

Robust estimation

Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers.

  • Sources of outliers (multiple motions):
    • specularities / highlights
    • jpeg artifacts / interlacing / motion blur
    • multiple motions (occlusion boundaries, transparency)

How to handle background and foreground motion

robust estimation47
Robust estimation

Standard Least Squares Estimation allows too much influence for outlying points

robust estimation48
Robust estimation

Robust gradient constraint

Robust SSD

robust estimation49

Problem: Least-squares estimators penalize deviations between data & model with quadratic error fn (extremely sensitive to outliers)

error penalty function

influence function

Redescending error functions (e.g., Geman-McClure) help to reduce the influence of outlying measurements.

error penalty function

influence function

Robust estimation
performance evaluation
Performance evaluation
  • See Baker et al. “A Database and Evaluation Methodology for Optical Flow”, ICCV 2007
  • Algorithms:
    • Pyramid LK: OpenCV-based implementation of Lucas-Kanade on a Gaussian pyramid
    • Black and Anandan: Author’s implementation
    • Bruhn et al.: Our implementation
    • MediaPlayerTM: Code used for video frame-rate upsampling in Microsoft MediaPlayer
    • Zitnick et al.: Author’s implementation
performance evaluation51
Performance evaluation
  • Difficulty: Data substantially more challenging than Yosemite
  • Diversity: Substantial variation in difficulty across the various datasets
  • Motion GT vs Interpolation: Best algorithms for one are not the best for the other
  • Comparison with Stereo: Performance of existing flow algorithms appears weak
motion representations
Motion representations
  • How can we describe this scene?
block based motion prediction
Block-based motion prediction
  • Break image up into square blocks
  • Estimate translation for each block
  • Use this to predict next frame, code difference (MPEG-2)
layered motion
Layered motion
  • Break image sequence up into “layers”:

 =

  • Describe each layer’s motion
layered motion55
Layered motion
  • Advantages:
  • can represent occlusions / disocclusions
  • each layer’s motion can be smooth
  • video segmentation for semantic processing
  • Difficulties:
  • how do we determine the correct number?
  • how do we assign pixels?
  • how do we model the motion?
background modeling mpeg 4
Background modeling (MPEG-4)
  • Convert masked images into a background sprite for layered video coding

+ + + + + +

=

what are layers
What are layers?
  • Intensities
  • Velocities
  • Layers
  • Alpha matting
  • Sprites
  • Wang and Adelson, “Representing moving images with layers”, IEEE Transactions on Image Processing, 1994
how do we estimate the layers
How do we estimate the layers?
  • compute coarse-to-fine flow
  • estimate affine motion in blocks (regression)
  • cluster with k-means
  • assign pixels to best fitting affine region
  • re-estimate affine motions in each region…
layer synthesis
Layer synthesis
  • For each layer:
    • stabilize the sequence with the affine motion
    • compute median value at each pixel
  • Determine occlusion relationships
applications
Applications
  • Motion analysis
  • Video coding
  • Morphing
  • Video denoising
  • Video stabilization
  • Medical image registration
  • Frame interpolation