1 / 40

Optical Flow and Motion in Computer Vision Applications

Explore optical flow algorithms for motion estimation in computer vision, including tracking objects, aligning images, 3D shape reconstruction, face tracking, and behavior analysis. Learn about applications like face recognition, rendering, and animation.

revilla
Download Presentation

Optical Flow and Motion in Computer Vision Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Computer Vision Spring 2010 15-385,-685 Instructor: S. Narasimhan Wean 5409 T-R 10:30am – 11:50am Lecture #13

  2. Optical Flow and Motion Lecture #13

  3. Optical Flow and Motion • We are interested in finding the movement of scene objects from time-varying images (videos). • Lots of uses • Track object behavior • Correct for camera jitter (stabilization) • Align images (mosaics) • 3D shape reconstruction • Special effects

  4. Tracking – Rigid Objects (Simon Baker, CMU)

  5. Tracking – Non-rigid Objects (Comaniciu et al, Siemens)

  6. Face Tracking (Simon Baker et al, CMU)

  7. Applications of Face Tracking • User Interfaces: • Mouse Replacement: Head Pose and Gaze Estimation • Automotive: Windshield Displays, Smart Airbags, Driver Monitoring • Face Recognition: • Pose Normalization • Model-Based Face Recognition • Lipreading/Audio-Visual Speech Recognition • Expression Recognition and Deception Detection • Rendering and Animation: • Expression Animations and Transfer • Low-Bandwidth Video Conferencing • Audio-Visual Speech Synthesis (Simon Baker, CMU)

  8. 3D Structure from Motion (David Nister, Kentucky)

  9. 3D Structure from Motion (David Nister, Kentucky)

  10. Behavior Analysis Query Result (Michal Irani, Weizmann)

  11. Scene point velocity: Image velocity: Perspective projection: Motion field Motion Field • Image velocity of a point moving in the scene

  12. Optical Flow • Motion of brightness pattern in the image • Ideally Optical flow = Motion field

  13. Optical Flow Motion Field No motion field but shading changes Motion field exists but no optical flow

  14. Problem Definition: Optical Flow • How to estimate pixel motion from image H to image I? • Find pixel correspondences • Given a pixel in H, look for nearby pixels of the same color in I • Key assumptions • color constancy: a point in H looks “the same” in image I • For grayscale images, this is brightness constancy • small motion: points do not move very far

  15. Optical Flow Constraint Equation Optical Flow: Velocities Displacement: • Assume brightness of patch remains same in both images: • Assume small motion: (Taylor expansion of LHS up to first order)

  16. Optical Flow Constraint Equation Divide by and take the limit Constraint Equation NOTE: must lie on a straight line We can compute using gradient operators! But, (u,v) cannot be found uniquely with this constraint!

  17. Finding Gradients in X-Y-T y time j+1 k+1 j k x i i+1

  18. Optical Flow Constraint • Intuitively, what does this constraint mean? • The component of the flow in the gradient direction is determined • The component of the flow parallel to an edge is unknown

  19. Optical Flow Constraint

  20. Aperture Problem

  21. Aperture Problem

  22. Computing Optical Flow • Formulate Error in Optical Flow Constraint: • We need additional constraints! • Smoothness Constraint (as in shape from shading and stereo): • Usually motion field varies smoothly in the image. • So, penalize departure from smoothness: • Find (u,v) at each image point that MINIMIZES: weighting factor

  23. Discrete Optical Flow Algorithm • Consider image pixel • Departure from Smoothness Constraint: • Error in Optical Flow constraint equation: • We seek the set that minimize: NOTE: show up in more than one term

  24. Discrete Optical Flow Algorithm • Differentiating w.r.t and setting to zero: • are averages of around pixel Update Rule:

  25. Example

  26. Optical Flow Result

  27. Low Texture Region - Bad • gradients have small magnitude

  28. Edges – so,so (aperture problem) • large gradients, all the same

  29. High Textured Region - Good • gradients are different, large magnitudes

  30. Focus of Expansion (FOE) • Motion of object = - (Motion of Sensor) • For a given translatory motion and gaze direction, the world • seems to flow out of one point (FOE). After time t, the scene point moves to:

  31. Focus of Expansion (FOE) • As t varies the image point moves along a straight line in the image • Focus of Expansion: Lets backtrack time or

  32. Focus of Expansion (FOE) - Example http://homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/LIB/bandb7_12.pdf

  33. Revisiting the Small Motion Assumption • Is this motion small enough? • Probably not—it’s much larger than one pixel (2nd order terms dominate) • How might we solve this problem?

  34. Reduce the Resolution!

  35. u=1.25 pixels u=2.5 pixels u=5 pixels u=10 pixels image H image I image H image I Gaussian pyramid of image H Gaussian pyramid of image I Coarse-to-fine Optical Flow Estimation

  36. upsample run iterative OF . . . image J image I image H image I Gaussian pyramid of image H Gaussian pyramid of image I Coarse-to-fine Optical Flow Estimation run iterative OF

  37. Image Alignment • Goal: Estimate single (u,v) translation (transformation) for entire image

  38. Mosaicing (Michal Irani, Weizmann)

  39. Mosaicing (Michal Irani, Weizmann)

  40. Next Class • Structured Light and Range Imaging • Reading  Notes

More Related