Motion Computing in Image Analysis

1 / 36

# Motion Computing in Image Analysis - PowerPoint PPT Presentation

Motion Computing in Image Analysis. - Mani V Thomas CISC 489/689. Roadmap. Optic Flow Constraint Optic Flow Computation Gradient Based Approach Feature Based Approach Estimation Criterion Block Matching algorithms Conclusion.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Motion Computing in Image Analysis' - juliette

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Motion Computing in Image Analysis

- Mani V Thomas

CISC 489/689

• Optic Flow Constraint
• Optic Flow Computation
• Feature Based Approach
• Estimation Criterion
• Block Matching algorithms
• Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Importance of Visual Motion
• Apparent motion of objects on the image plane is a strong cue to understand structure and 3D motion
• Biological visual systems infer properties of the 3D world via motion
• Two sub-problems of motion
• Problem of correspondence estimation
• Which elements of a frame correspond to which elements of the next frame
• Problem of reconstruction
• Given the correspondence and the camera’s intrinsic parameters can we infer 3D motion and/or structure

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Apparent Motion
• Apparent motionof objects on the image plane
• Caution required!!
• Consider a perfectly uniform sphere that is rotating but no change in the light direction
• Optic flow is zero
• Perfectly uniform sphere that is stationary but the light is changing
• Optic flow exists
• Hope – apparent motion is very close to the actual motion

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Optic Flow Computation
• Two strategies for computing motion
• Differential Methods
• Spatio temporal derivatives for estimation of flow at every position
• Multi-scale analysis required if motion not constrained within a small range
• Dense flow measurements
• Matching Methods
• Feature extraction(Image edges, corners)
• Feature/Block Matching and error minimization
• Sparse flow measurements

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Optic Flow Computation
• Image Brightness Constancy assumption
• Let E be the image intensity as captured by the camera
• Using Taylor series to expand E
• Apparent brightness of moving objects remains constant
Optic Flow Computation
• Image Brightness Constancy assumption
• Apparent brightness of moving objects remains constant
• The are the image gradient while the are the components of the motion field

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Aperture Problem
• We can measure
• Terms that can be measured
• Terms to be computed
• Number of equations - 1
• The component of the motion field that is orthogonal to the spatial image gradient is not constrained by the image brightness constancy assumption
• Intuitively
• The component of the flow in the gradient direction is determined
• The component of the flow parallel to an edge is unknown

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

• Optic Flow Constraint
• Optic Flow Computation
• Feature Based Approach
• Estimation Criterion
• Block Matching algorithms
• Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Optic Flow Constraint
• How to get more equations for a pixel?
• Basic idea: impose additional constraints
• Most common is to assume that the flow field is smooth locally
• One method: pretend the pixel’s neighbors have the same (u,v)
• If we use a 5x5 window, that gives us 25 equations per pixel!
• We now have more equations than unknowns
• Solve the least squares problem
• Minimum least squares solution (in d) is given by
• First proposed by Lucas-Kanade in 1981
• Summation performed over all the pixels in the window
• When is the Lucas-Kanade equations solvable
• ATA should be invertible
• ATA should not be too small (effects of noise)
• Eigenvalues of ATA, 1 and 2 should not be small
• ATA should be well conditioned
• 1/2 should not be large (1 = larger eigenvalue)
Edge
• Gradient is large in magnitude
• Large 1 but small 2
Low texture region
• Small 1 and small 2
High texture region
• Gradients are different with large magnitudes
• Large 1 and large 2
• When our assumptions are violated
• Brightness constancy is not satisfied
• The motion is not small
• A point does not move like its neighbors
• Estimate velocity at each pixel by solving Lucas-Kanade equations
• Warp H towards I using the estimated flow field
• use image warping techniques
• Repeat until convergence

u=1.25 pixels

u=2.5 pixels

u=5 pixels

u=10 pixels

image H

image H

image I

image I

Gaussian pyramid of image H

Gaussian pyramid of image I

warp & upsample

run iterative L-K

.

.

.

image J

image H

image I

image I

Gaussian pyramid of image H

Gaussian pyramid of image I

run iterative L-K

• Optic Flow Constraint
• Optic Flow Computation
• Feature Based Approach
• Estimation Criterion
• Block Matching algorithms
• Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Feature Based Method
• Feature Extraction
• Maxima in first derivative of the Image
• Local peak in the first derivative
• Numerical Approximation
• Compute the motion parameters from the best bipartite graph
• Correspondence between the feature points in one image with those in the other

For more information: Ramesh Jain, Rangachar Kasturi, Brian Schunck: Machine Vision 1995 (140 - 159)

• Optic Flow Constraint
• Optic Flow Computation
• Feature Based Approach
• Estimation Criterion
• Block Matching algorithms
• Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Estimation Criterion
• Pixel domain Criterion
• MAE/MSE
• Lorentzian
• Correlation
• Frequency Domain Criterion
• Cross Correlation
• Phase Correlation
Estimation Criterion(contd.)
• Pixel Domain Criterion
• Estimation criterion aim at minimizing
• prediction error is sensitive to noise if number of pixels is not large or if region is poorly textured
• Common choice of estimation criterion
• Quadratic function is not good since a single large error can bias the estimate of the field
• Absolute value function is better than the quadratic since cost grows linearly with error
• Does not require multiplications and is better suited for real-time video encoders
Estimation Criterion(contd.)
• A more robust criterion is based on the Lorentzian function
• Grows slower than |x| for larger errors
• Similarity measure using Correlation
• Computationally complex because of the multiplications
• This criterion requires maximization
• Usually the normalized Cross correlation is computed

For more details: M. Black, “Robust Incremental Optical Flow”

Estimation Criterion(contd.)
• Frequency Domain Criterion
• Amplitudes of both the FT are independent of z
• Argument difference depends linearly on translation
• Global motion is recovered by evaluating the phase difference over a number of frequencies and solving the resulting system of equations
• In practice, this method will work only for a single object moving across a uniform background
Estimation Criterion(contd.)
• Phase Correlation
• In the case of a single global translation, the correlation surface becomes a Kronecker delta function
• In practice, there are numerous peaks which correspond to the dominant displacements between the two images
• The locations are relatively independent to illumination changes
• Optic Flow Constraint
• Optic Flow Computation
• Feature Based Approach
• Estimation Criterion
• Block Matching algorithms
• Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Block Matching Algorithms
• Sparse motion measurements
• Motion is spatially constant and temporally linear over a rectangular region of support
• The minimization problem is
• is an M x N block of pixels with the top-left corner co-ordinate at

-p

N

-p

N

M

M

p

p

Current Picture

Reference Frame

v

-p

N

(x,y)

(x+u,y+v)

M

-p

p

u

p

Block Matching Algorithms(contd.)
Block Matching Algorithms(contd.)
• Principle of Locality of Reference
• Block Matching algorithms
• Exhaustive Search
• Always finds the “deepest” minimum
• Computationally very expensive
• If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
• This corresponds to 29.89 GOPS for p=15 at 30fps for a 720x480 image (3 operations per pixel of one subtraction, one absolute value and one addition)
Block Matching Algorithms(contd.)
• Logarithmic Search
• Sub-optimal and may get trapped in a local minima
• Computationally feasible for real-time video encoders
• Search Method
• Divide the search space at [-p/2, -p/2]
• Search at (0,0) and at 8 major points at the perimeter of the rectangle at [-p/2, -p/2]
• Using best match position as starting point, search in the eight perimeter points at the half distance window
• If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
• This corresponds to 1.03 GOPS for p=15 at 30fps for a 720x480 image

For more information refer the work by Dr. Lai-Man Po and C. K. Cheung (http://www.ee.cityu.edu.hk/~lmpo/publications/index.html)

Block Matching Algorithms(contd.)
• Hierarchical Search
• Sub-optimal for regions containing detail and increased storage requirements
• Computationally feasible for real-time video encoders
• Search method
• Form several low resolution images by low pass filtering
• At the lowest resolution perform a sub-optimal search like log search
• Propagate search vectors to higher resolution images and perform search
• If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
• This corresponds to 507.38 MOPS for p=15 at 30fps for a 720x480 image
Conclusion
• Motion estimation
• Aperture problem
• Different algorithms to perform motion analysis