stereo matching energy minimization
Download
Skip this Video
Download Presentation
Stereo Matching & Energy Minimization

Loading in 2 Seconds...

play fullscreen
1 / 42

Stereo Matching & Energy Minimization - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

Stereo Matching & Energy Minimization. Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski. Stereo Matching. What are some possible algorithms? match “features” and interpolate match edges and interpolate match all pixels with windows (coarse-fine) use optimization:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Stereo Matching & Energy Minimization' - dana-sutton


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
stereo matching energy minimization

Stereo Matching &Energy Minimization

Vision for GraphicsCSE 590SS, Winter 2001Richard Szeliski

stereo matching
Stereo Matching
  • What are some possible algorithms?
    • match “features” and interpolate
    • match edges and interpolate
    • match all pixels with windows (coarse-fine)
    • use optimization:
      • iterative updating
      • energy minimization (regularization, stochastic)
      • dynamic programming
      • graph algorithms

Vision for Graphics

feature based stereo
Feature-based stereo
  • Match “corner” (interest) points
  • Interpolate complete solution

Vision for Graphics

data interpolation
Data interpolation
  • Given a sparse set of 3D points, how do we interpolate to a full 3D surface?
  • Scattered data interpolation [Nielson93]
  • triangulate
  • put onto a grid and fill (use pyramid?)
  • place a kernel function over each data point
  • minimize an energy function

Vision for Graphics

energy minimization
Energy minimization
  • 1-D example: approximating splines

dx,y

zx,y

Vision for Graphics

relaxation
Relaxation
  • Iteratively improve a solution by locally minimizing the energy: relax to solution
  • Earliest application: WWII numerical simulations

zx,y

dx+1,y

dx,y

dx+1,y

Vision for Graphics

relaxation1
Relaxation
  • Try solving this problem yourself:
    • Make up a bunch of zx,y
    • Guess best values for dx,y

Vision for Graphics

relaxation2
Relaxation
  • How can we get the best solution?
  • Differentiate energy function, set to 0

Vision for Graphics

non quadratic energy
Non-quadratic energy
  • How about minimizing this cost function?

Vision for Graphics

discrete optimization space
Discrete optimization space
  • What if you have discrete (e.g., binary) values?

0

0

0

0

0

1

1

1

1

1

Vision for Graphics

dynamic programming
Dynamic programming
  • Evaluate best cumulative cost at each pixel

0

0

0

0

0

1

1

1

1

1

Vision for Graphics

dynamic programming1
Dynamic programming
  • 1-D cost function

Vision for Graphics

dynamic programming2
Dynamic programming
  • Can we apply this trick in 2D as well?

dx-1,y

dx,y

dx-1,y-1

dx,y-1

No: dx,y-1 anddx-1,y may depend on different values of dx-1,y-1

Vision for Graphics

graph cuts
Graph cuts
  • Solution technique for general 2D problem

Vision for Graphics

graph cuts1
Graph cuts
  • Two different kinds of moves:

Vision for Graphics

graph cuts2
Graph cuts
  • a-b swap: interchange a and b labels

Vision for Graphics

graph cuts3
Graph cuts
  • a expansion: add pixels to a class

Vision for Graphics

neighborhood size review
Neighborhood size (review)
  • Smaller neighborhood: more details
  • Larger neighborhood: fewer isolated mistakes
  • w = 3 w = 20

Vision for Graphics

plane sweep stereo
Plane sweep stereo
  • Re-order (pixel / disparity) evaluation loopsfor every pixel, for every disparity for every disparity for every pixel compute cost compute cost

Vision for Graphics

stereo matching framework
Stereo matching framework
  • For every disparity, compute raw matching costsWhy use a robust function?
    • occlusions, other outliers
  • Can also use alternative match criteria

Vision for Graphics

stereo matching framework1
Stereo matching framework
  • Aggregate costs spatially
  • Here, we are using a box filter(efficient moving averageimplementation)
  • Can also use weighted average,[non-linear] diffusion…

Vision for Graphics

stereo matching framework2
Stereo matching framework
  • Choose winning disparity at each pixel
  • Can interpolate to sub-pixel accuracy

Vision for Graphics

linear diffusion
Linear diffusion
  • Average energy with neighbors
  • window diffusion

Vision for Graphics

linear diffusion1
Linear diffusion
  • Average energy with neighbors + starting value
  • window diffusion

Vision for Graphics

dynamic programming3
Dynamic programming
  • 1-D cost function [Intille & Bobick, IJCV 99]

Vision for Graphics

dynamic programming4
Dynamic programming
  • Disparity space image and min. cost path

Vision for Graphics

dynamic programming5
Dynamic programming
  • Sample result (note horizontal streaks)

Vision for Graphics

graph cuts4
Graph cuts
  • a-b swap
  • expansion

modify smoothness penalty based on edges

compute best possible match within integer disparity

Vision for Graphics

bayesian inference
Bayesian inference
  • Formulate as statistical inference problem
  • Prior model pP(d)
  • Measurement model pM(IL, IR| d)
  • Posterior model
  • pM(d | IL, IR)  pP(d) pM(IL, IR| d)
  • Maximum a Posteriori (MAP estimate):
  • maximize pM(d | IL, IR)

Vision for Graphics

markov random field
Markov Random Field
  • Probability distribution on disparity field d(x,y)
  • Enforces smoothness or coherence on field

Vision for Graphics

measurement model
Measurement model
  • Likelihood of intensity correspondence
  • Corresponds to Gaussian noise for quadratic r

Vision for Graphics

map estimate
MAP estimate
  • Maximize posterior likelihood
  • Equivalent to regularization (energy minimization with smoothness constraints)

Vision for Graphics

why bayesian estimation
Why Bayesian estimation?
  • Principled way of determining cost function
  • Explicit model of noise and prior knowledge
  • Admits a wider variety of optimization algorithms:
    • gradient descent (local minimization)
    • stochastic optimization (Gibbs Sampler)
    • mean-field optimization
    • graph theoretic (actually deterministic) [Zabih]

Vision for Graphics

mean field interpretation
Mean-field interpretation
  • Bayesian non-linear diffusion rule:
    • update your probability distribution assuming your neighbors’ distributions are independent (valid for Markov chain)
  • Equivalent to finding best factored approximation
  • P(d|IL,IR) ~ Q(d) = iQi(di)

Vision for Graphics

mean field interpretation1
Mean-field interpretation
  • log MAP estimate
  • -log P(d|IL,IR) = ijEij(di,dj) + iEi(di)
  • = ijsiAijsi + ibisi
  • Kullback-Leibler divergence
  • DKL =H(Q) - dQ(d) log P(d)
  • = ik qik log qik + ijsiAijsi + ibisi

Vision for Graphics

mean field interpretation2
Mean-field interpretation
  • minimize K-L divergence with
  • k qik = 1
  • update rule:
  • qik  exp[ - ( jaijkqj +bik )]
  • = exp[- ( jlEij(di=k,dj=l)p(dj=l) + Ei(di=k))]

Vision for Graphics

depth map results
Depth Map Results
  • Input image Sum Abs Diff
  • Mean field Graph cuts

Vision for Graphics

stereo with non linear diffusion
Stereo with Non-Linear Diffusion
  • Advantages:
    • works very well in non-occluded regions
  • Disadvantages:
    • restricted to two images (not)
    • gets confused in occluded regions
    • can’t handle mixed pixels

Vision for Graphics

summary
Summary
  • Applications
  • Image rectification
  • Matching criteria
  • Local algorithms (aggregation)
    • area-based; iterative updating
  • Optimization algorithms:
  • energy (cost) formulation
  • Markov Random Fields
    • mean-field; dynamic programming;
    • stochastic; graph algorithms

Vision for Graphics

more stereo next 2 lectures
More stereo…(next 2 lectures)
  • Multi-image stereo
  • Volumetric techniques
  • Graph cuts
  • Transparency
  • Surfaces and level sets

Vision for Graphics

bibliography
Bibliography
  • See the references in the readings…
  • Y. Boykov, O. Veksler, and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, Unpublished manuscript, 2000.
  • A.F. Bobick and S.S. Intille. Large occlusion stereo. International Journal of Computer Vision, 33(3), September 1999. pp. 181-200
  • D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998
  • R. Szeliski. Stereo algorithms and representations for image-based rendering. In British Machine Vision Conference (BMVC'99), volume 2, pages 314-328, Nottingham, England, September 1999.
  • R. Szeliski and R. Zabih. An experimental comparison of stereo algorithms. In International Workshop on Vision Algorithms, pages 1-19, Kerkyra, Greece, September 1999.
  • G. M. Nielson, Scattered Data Modeling, IEEE Computer Graphics and Applications, 13(1), January 1993, pp. 60-70.

Vision for Graphics

ad