1 / 69

Recent progress in optical flow

Recent progress in optical flow. progress. Presented by : Darya Frolova and Denis Simakov. Optical Flow is not in favor. Very popular slide:. Often not using Optical Flow is stated as one of the main advantages of a method.

titus
Download Presentation

Recent progress in optical flow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Recent progress in optical flow progress Presented by: Darya Frolova and Denis Simakov

  2. Optical Flow is not in favor Very popular slide: Often not using Optical Flow is stated as one of the main advantages of a method Optical Flow methods have a reputation of either unreliable or slow Recent works claim: Optical Flow can be computed fastand accurately

  3. more methods many methods Optical Flow Research: Timeline Horn&Schunck Lucas&Kanade 1981 1992 1998 now Benchmark:Galvin et.al. Benchmark:Barron et.al. Seminal papers A slow and not very consistent improvement in results, but a lot of useful ingredients were developed

  4. In This Lecture We will describe : • Ingredients for an accurate and robust optical flow • How people combine these ingredients • Fast algorithms Papers: • Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005 • High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005 • Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005 • Towards ultimate motion estimation: Combining highest accuracy with real-time performanceA. Bruhn, J. Weickert, 2005 • Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006

  5. What is Optical Flow?

  6. Definitions The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] + = frame #2 frame #1 flow field The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck]

  7. flow (2) flow (1): true motion Ambiguity of optical flow Frame 1

  8. Applications optical flow • video compression • 3D reconstruction • segmentation • object detection • activity detection • key frame extraction • interpolation in time motion field We are usually interested in actual motion

  9. Outline • Ingredients for an accurate and robust optical flow • Local image constraints on motion • Robust statistics • Spatial coherence • How people combine these ingredients • Fast algorithms

  10. Local image constraints

  11. Brightness Constancy u frame t+1 v frame t

  12. Complex dependence on Linearized brightness constancy Deviation from brightness constancy (we want it to be zero) Linearize:

  13. Linearized brightness constancy Let us square the difference: J – “motion tensor”, or “structure tensor”

  14. Averaged linearized constraint J is a function of x, y (a matrix for every point) Combine over small neighborhoods (more robust to noise): = J *

  15. Method of Lucas&Kanade • Solve independently for each point [ Lucas&Kanade 1981] linear system Can be solved for every point where matrix is not degenerate

  16. Lukas&Kanade - Results Rubik cube Hamburg taxi flow field flow field

  17. Brightness is not always constant Rotating cylinder Brightness constancy does not always hold Gradient constancy holds intensity intensity derivative position position

  18. Local constraints - Summary We have seen linearized • brightness constancy averaged linearized averaged linearized • gradient constancy

  19. Local constraints are not enough!

  20. Local constraints work poorly Optical flow direction using only local constraints input video color encodes direction as marked on the boundary

  21. Where local constraints fail Uniform regions Motion is not observable in the image (locally)

  22. Where local constraints fail “Aperture problem” We can estimate only one flow component (normal)

  23. Where local constraints fail Occlusions We have not seen where some points moved Occluded regions are marked in red

  24. Obtaining support from neighbors • Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong

  25. Robust combination of partially reliable data or: How to hold elections

  26. L2: L1: xi → xi + ∆ Influence of xi on E: equal for all xi proportional to Outliers influence the most Majority decides Toy example Find “best” representative for the set of numbers xi

  27. Oligarchy Democracy Elections and robust statistics many ordinary people a very rich man wealth Votes proportional to the wealth One vote per person like in L1 norm minimization like in L2 norm minimization

  28. usual: L2 robust regularized robust: L1 ε • easy to analyze and minimize • sensitive to outliers • robust in presence of outliers • non-smooth: hard to analyze • smooth: easy to analyze • robust in presence of outliers Combination of two flow constraints [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

  29. Spatial Propagation

  30. Obtaining support from neighbors • Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong

  31. - flow in the x direction • flow in the y direction • gradient Homogeneous propagation This constraint is not correct on motion boundaries => over-smoothing of the resulting flow [Horn&Schunck 1981]

  32. Robustness to flow discontinuities ε (also known as isotropic flow-driven regularization) [T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004] High accuracy optical flow estimation based on a theory for warping

  33. Selective flow filtering • We want to propagate information • without crossing image and flow discontinuities • from “good” points only (not occluded) Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

  34. I I I I x x x Bilateral filter Unilateral (usual) Bilateral x Preserves discontinuities! [C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

  35. occluded regions Using of bilateral filter - Example cyan rectangle moves to the right and occludes background region marked by red [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

  36. Learning of spatial coherence • Come to the next lecture …

  37. Spatial coherence: Summary Homogeneous propagation - oversmoothing Robust statistics with homogeneous propagation - preserves flow discontinuities Bilateral filtering - combines information from regions with similar flow and similar intensities Handles occlusions

  38. Two more useful ingredients in brief – one slide each

  39. 2D vs. 3D Several frames allow more accurate optical flow estimation 2 frames: Several frames:

  40. MultiscaleOptical Flow Linearization: valid only for small flow pyramid for frame 1 pyramid for frame 2 frame 1warped ? + upsample + (other names: “warping”, “coarse-to-fine”, “multiresolution”)

  41. Methods How to make tasty soup with these ingredients: several recipes

  42. Outline • Ingredients for an accurate and robust optical flow • How people combine these ingredients • Lukas & Kanade meet Horn & Schunck • The more ingredients – the better • Bilateral filtering and occlusions • Fast algorithms

  43. Combining ingredients • Spatial coherency • Homogeneous • Flow-driven • Bilateral filtering + occlusions • Local constraints • Brightness constancy • Image gradient constancy Energy = ∫ϕ(Data) + ∫ϕ(“Smoothness”) Combined using robust statistics Computed coarse-to-fine Use several frames

  44. Combining Local and Global Remember: Lucas&Kanade Horn&Schunk Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002]

  45. Sensitivity to noise – quantitative results frame t+1 Error measure: angle between true and computed flow in (x,y,t) space frame t ground truth flow

  46. The more ingredients - the better brightness constancy spatial coherence gradient constancy [Bruhn, Weickert, 2005]Towards ultimate motion estimation: Combining highest accuracy with real-time performance

  47. Quantitative results Angular error Method Yosemite sequence with clouds Average error decreases, but standard deviation is still high….

  48. Influence of each ingredient For Yosemite sequence with clouds

  49. Handling occlusions bilateral filtering of flow:preserve intensity and flow discontinuities; model occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection

  50. Qualitative results

More Related