1 / 55

Depth and Motion Discontinuities

Depth and Motion Discontinuities. Stan Birchfield Ph.D. oral defense Stanford University January 1999. Discontinuities. PHOTOMETRIC GEOMETRIC. intensity, color, texture, . camera. camera. surface. surface. Depth. Motion. Importance of Discontinuities. object. camera 1. camera 2.

toshi
Download Presentation

Depth and Motion Discontinuities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Depth and Motion Discontinuities Stan Birchfield Ph.D. oral defense Stanford University January 1999

  2. Discontinuities • PHOTOMETRIC • GEOMETRIC intensity, color, texture, ... camera camera surface surface Depth Motion

  3. Importance of Discontinuities object camera 1 camera 2 • Fundamental to image understanding • Object boundaries(well-defined) • Assumption of continuity • Occlusion

  4. Motivation IMAGES LOW-LEVEL FILTERS HIGH-LEVEL TASKS segmentation tracking detection classification camera control video retrieval

  5. Evidence for Discontinuities • LOCAL • T-junctions[Parida et al. 1997, Ruzon & Tomasi 1999] • Occlusion [Toh & Forrest 1990, Wixson 1993] • Multi-modal depth or motion[Spoerri & Ullman 1987, Little & Gillett 1990] • Changes in intensity, color, texture, …[Canny 1986, Wang & Binford 1994] • GLOBAL • “Continuity of discontinuities”[Marr & Poggio 1979] • Changes in depth or motion field[Thompson et al. 1982, Birchfield & Tomasi 1998]

  6. Complications from Sampling • Amount of discontinuity • Curvature ?

  7. Finding and Using Discontinuities Depth discontinuities from stereo Motion discontinuities from a sequence Maximum flow for stereo and motion Using discontinuities to track heads

  8. Depth Discontinuities from a Stereo Pair L intensity R epipolar constraint lamp disparity wall pixel Use dynamic programming to minimize: discontinuity penalty dissimilarity

  9. Discontinuity Penalty L L intensity R R u(l ) i disparity pixel GOOD BAD l i u(3) < 3u(1) GOOD u(3) = 3u(1) BAD penalty occlusion length

  10. Stereo Algorithm • Matches pixels directly with convex penalty function(no preprocessing or windows) • Handles large untextured regions • Solves sampling problem • Handles slanted surfaces • Is fast

  11. Handling Untextured Regions Assumption: Depth discontinuities are accompanied by intensity edge Naïve constraint: Depth discontinuities lie near intensity edge Our constraint: Depth discontinuities lie to a particular side of intensity edge Discontinuities in left (right) scanline lie to the left (right) of intensity edge Intensity edges (x-derivative with low threshold) With naïve constraint With our constraint

  12. Discontinuity Lies to the Left of an Edge far object Physical setup near object left camera right camera depth discontinuity Matches violate assumption edge Obvious matches Matches are consistent with assumption depth discontinuity edge

  13. The Problem of Image Sampling Our measure Left scanline Right scanline Absolute difference

  14. A Dissimilarity Measure That is Insensitive to Image Sampling d(xL,xR) xL xR Our dissimilarity measure: d(xL,xR) = min{d(xL,xR) ,d(xR,xL)}

  15. Analysis of Dissimilarity Measure intensity function Concave/convex regions: measure is guaranteed to work Inflection points: measure works in practice when lens is defocused to remove aliasing (inflection points @ linear {convex, concave})

  16. Analysis of Dissimilarity Measure At a hypothesized disparity of 10 pixels maximum average As a function of lens defocus absolute difference our measure { 1 defocus aliasing Tc = fc cutoff frequency

  17. Speeding Computation TWO STEPS: 1. Reliable overruns unreliable 2. Background overruns foreground Fast postprocessing occlusion RIGHT Pruning bad nodes Standard Disparity map time LEFT Ours maximum disparity depth discontinuity

  18. Handling Slanted Surfaces Independent scanlines: no coherence disparity Postprocessing: forbid propagation if disparity changes by just one level column

  19. Results

  20. More Results [Images from JISCT data set]

  21. Finding and Using Discontinuities Depth discontinuities from stereo Motion discontinuities from a sequence Maximum flow for stereo and motion Using discontinuities to track heads

  22. Motion Discontinuities from a Sequence • Similar to stereo(matching pixels to fit piecewise-smooth function) • But different(2D search, non-rigid transformation, many images) • How far can we get using sparse features?

  23. Motion Video

  24. Difficulties of Image Sequences feature position • Instantaneous velocityfrom sampled positions • Accumulation of evidence • Frame of reference, motion model ? time feature 1 threshold position feature 2 background time feature threshold position background time

  25. Image Strain

  26. Motion Algorithm • Select and track features • Group • Trace boundariesbetween groups

  27. Finding and Using Discontinuities Depth discontinuities from stereo Motion discontinuities from a sequence Maximum flow for stereo and motion Using discontinuities to track heads

  28. Why Maximum Flow? • Our stereo algorithm (Dynamic programming) • Good results on difficult images • Fast • But, suboptimal(processes rows, then columns independently) • Newer algorithms (Maximum flow) • Set up graph, find maximum flow ---minimum cut yields disparity or motion map • Able to look at whole image

  29. Minimizing a 2D Cost Function 1D: 2D: minimum cut = disparity surface GLOBAL solves disparity disparity ? pixel Discontinuity penalty: u(l ) p,q Local (GOOD) (BAD) Global LOCAL l p,q Minimize: d

  30. Maximum Flow for Stereo and Motion Global algorithm [Roy & Cox 1998, Ishikawa & Geiger 1998] penalty BAD discontinuity amount Local algorithm [Boykov et al. 1998] GOOD

  31. Challenges for Maximum Flow With our intensity variation constraint and large penalty … and slant of table With large penalty With small penalty … and slant of surfaces

  32. Finding and Using Discontinuities Depth discontinuities from stereo Motion discontinuities from a sequence Maximum flow for stereo and motion Using discontinuities to track heads

  33. Problem ZOOM TILT PAN APPLICATIONS: * video conferencing * distance learning CHALLENGES: * rotation * multiple people * zoom

  34. Previous Methods Contour: Y N Skin color: N ? Background subtraction: Y N Moving people in background 3D Rotation Template: N Y

  35. Head Tracking Algorithm Ellipse: vertical aspect ratio = 1.2 state s = (x,y,s) s (x,y) ellipse normal gradient gradient module ellipse in frame t-1 ellipse in frame t Skin Hair Model frame t color module Current Intersection INPUT OUTPUT LOCAL SEARCH MODEL

  36. Evaluation of Head Tracker 1. Tracks head in real time on standard hardware 2. Insensitive to - full 360-degree out-of-plane rotation - arbitrary camera movement (including zoom) - multiple moving people - severe but brief occlusion - hair/skin color, hair length, facial hair, glasses

  37. Headtracker Video

  38. Finding and Using Discontinuities Depth discontinuities from stereo Motion discontinuities from a sequence Maximum flow for stereo and motion Using discontinuities to track heads

  39. Contributions • Precise, unified definitions and relationship • Pixel-to-pixel stereo algorithm [ICCV 1998] • good results on difficult images • intensity variation constraint • dissimilarity measure [PAMI 1998] • preserving slant • fast (pruning search nodes, postprocessor) • Motion discontinuities(sparse features, image strain, grouping algorithm) • Maximum flow techniques(edge weights for global algorithm, smoothness, left border ; applied to motion) • Elliptical head tracker [Asilomar 1997, CVPR 1998] • advanced state of the art by tracking people undergoing 3D rotation in front of a dynamic background

  40. Thank you

  41. Appendix: Extra slides

  42. Comparison of Our Algorithm with Maxflow Our algorithm Local maxflow algorithm

  43. Depth Discontinuity • Proposition 1: A depth discontinuity is a point in the image plane whose projection ray grazes a surface in the world. object image plane Depth discontinuities focal point

  44. Definition of Jump • The jump of a discontinuity is the smallest e>0 such that there is a d>0 for which, if ||x-x0||<d, then |f(x)-f(x0)| > e f(x) jump e x x0 d

  45. Comparison with Intensity Edges Original image Intensity edges Depth discontinuities • More closely tied to actual object boundaries • Not distracted by • change in albedo (reflectance) • change in lighting (e.g., shadows)

  46. Comparison with Layers Original image Motion discontinuities

  47. Comparison with Segmentation Original image Depth discontinuities A plausible segmentation • Well-defined (not task-dependent) • Curves are not necessarily closed

  48. Epipolar Constraint world point epipolar line epipolar plane center of projection Left camera Right camera

  49. Comparison of Dissimilarity Measure with Other Methods spurious matches absolute difference our measure Computing time subpixel resolution

  50. Propagating Information Between Scanlines • Calculate reliabilities & thresholdDisparities:[ 4 4 4 4 4 7 7 7 2 ]TReliabilities:[ 5 5 5 5 5 3 3 3 1 ]Tunreliable < reliable < very reliable • Propagate very reliable when • neighbor unreliable, OR • disparity(neighbor) > disparity(pixel) + 2 • Repeat in x-direction Fast (30% increase instead of 800%)

More Related