Lecture 9 Optical Flow, Feature Tracking, Normal Flow. Gary Bradski Sebastian Thrun. *. http://robots.stanford.edu/cs223b/index.html. * Picture from Khurram Hassan-Shafique CAP5415 Computer Vision 2003. Q from stereo about Essential Matrix. (from Trucco P-153).
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Lecture 9Optical Flow, Feature Tracking, Normal Flow
Gary Bradski
Sebastian Thrun
*
http://robots.stanford.edu/cs223b/index.html
* Picture from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
(from Trucco P-153)
Rank (R) =3
Rank (S) =2
Why is this zero if it’s not orthogonal?
So,
Why is this zero if it’s not orthogonal?
Answer: We’re dealing with equations of lines in
homogeneous coordinates.
Remember from Sebastian’s lecture, projective equations
are nonlinear because of the scale factor (1/Z). By
adding a generic scale, we get simple linear equations.
Thus, a point in the image plane is expressed as:
For a line:
Equation
Thus,
represents the projection of the line pl onto the right image
plane.
is the equation of the line in the right image written in terms
of the point pl. That is, a statement that the point pl lies on
the that line.
Image tracking
3D computation
Image sequence
(single camera)
Tracked sequence
3D structure
+
3D trajectory
Optical Flow
Velocity vectors
Common assumption:
The appearance of the image patches do not change (brightness constancy)
Note: more elaborate tracking models can be adopted if more frames are process all at once
Perhaps an aperture problem discussed later.
Optical Flow Break Down
* From Marc Pollefeys COMP 256 2003
* Slide from Michael Black, CS143 2003
* Slide from Michael Black, CS143 2003
* Slide from Michael Black, CS143 2003
{
Because no change in brightness with time
Ix
v
It
Brightness Constancy Assumption:
?
Temporal derivative
Spatial derivative
Assumptions:
Temporal derivative at 2nd iteration
Can keep the same estimate for spatial derivative
Iterating helps refining the velocity vector
Converges in about 5 iterations
For all pixel of interest p:
Requirements:
2D:
1D:
Shoot! One equation, two velocity (u,v) unknowns…
We get at most “Normal Flow” – with one point we can only detect movement
perpendicular to the brightness gradient. Solution is to take a patch of pixels
Around the pixel of interest.
* Slide from Michael Black, CS143 2003
Motion along just an edge is ambiguous
Aperture problem
The Math is very similar:
Window size here ~ 11x11
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
What are the potential causes of errors in this procedure?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
It-1(x,y)
It-1(x,y)
It-1(x,y)
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
- use image warping techniques
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
u=1.25 pixels
u=2.5 pixels
u=5 pixels
u=10 pixels
image It-1
image It-1
image I
image I
Gaussian pyramid of image It-1
Gaussian pyramid of image I
warp & upsample
run iterative L-K
.
.
.
image J
image It-1
image I
image I
Gaussian pyramid of image It-1
Gaussian pyramid of image I
run iterative L-K
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
* From Marc Pollefeys COMP 256 2003
Generalization
* From Marc Pollefeys COMP 256 2003
* From Marc Pollefeys COMP 256 2003
* Slide from Michael Black, CS143 2003
Horn & Schunck algorithm
Additional smoothness constraint :
besides Opt. Flow constraint equation term
minimize es+aec
* From Marc Pollefeys COMP 256 2003
The above solution requires that G be of full rank, that is, on a corner. Simplified,
what basically happens for the solution in Horn and Schunck is that:
which is always full rank.
Horn & Schunck algorithm
In simpler terms: If we want dense flow, we need to regularize what happens
in ill conditioned (rank deficient) areas of the image. We take the old cost function:
And add a regularization term to the cost:
where ||d|| is some length metric, typically Euclidian length. When you solve, what
happens to our former solution
?
Regularized flow
Optical flow
Dense Optical Flow ~ Michael Black’s method
Michael Black took this one step further, starting from the regularized cost:
He replaced the inner distance metric, a quadradic:
with something more robust:
?
Where looks something like
Basically, one could say that Michael’s method adds ways to handle
occlusion, non-common fate, and temporal dislocation
Davis, Bradski, WACV 2000
Bradski Davis, Int. Jour. of Mach.
Vision Applications 2001
MHG
silhouette
MHI
Motion Segmentation Algorithm
Motion Segmentation Algorithm
Motion Segmentation Algorithm
Segmented
Motion
Segmented
Motion
Actually need a two-pass algorithm for labeling all motion segments:
Overlay silhouettes, take gradient for normal optical
flow. Flood fill to segment motions.
Motion
Segmentation
Motion
Segmentation
Pose
Recognition
Gesture
Recognition