JetStream : Probabilistic Contour Extraction with Particles

1. JetStream: Probabilistic Contour Extraction with Particles Patrick Perez, Andrew Blake, and Michel Gangnet, Microsoft Research, St George House, 1 Guildhall Street, Cambridge, CB2 3NH, UK http://research.microsoft.com/vision Presented by: Vladan Radosavljevic

2. Outline • Introduction • Image Gradient • Related Work • Probabilistic contour tracking • Tracking framework • Dynamics • Measurement • Iterative computation of posterior - particles • Model Ingredients • Likelihood ratio • Dynamics • Proposal Sampling Function • Experimental Results • Conclusion

3. Motivation • Contour extraction • from segmenting images with closed contours • to the extraction of linear structures of particular interest such as roads

4. The gradient direction is given by: • how does this relate to the direction of the edge? • The edge strength is given by the gradient magnitude Introduction - image gradient • The gradient of an image: • The gradient points in the direction of most rapid change in intensity

5. Introduction • Most approaches to contour extraction rely on some minimal cost principle • k is the curvature, s is the arc-length • y(r(s)) scalar or vector derived at location r(s) from the raw image I • Often y(r(s)) is the gradient norm • This function captures some kind of regularity on candidate curves, for example - rewarding, by a lower cost, the presence of large gradients along the contour

6. Introduction – related work • First approach - dynamic programming • optimal curve in the form of chain pixels • unless optimality is abandoned, and huge storage resources are available, there are tight restrictions on the form of cost function • Second approach - growing a contour from the seed point according to cost function • given the current contour, a new segment is appended to it according both to a shape prior (mainly smoothness) and to the evidence provided by the data at the location under concern • deterministic: complete discontinuous curves provided by edge detectors

7. Introduction – related work • A probabilistic point of view: • the contours are seen as the paths of a stochastic process • tracking problem • We are interesting in the method that is able: • to avoid spurious distracting contours • to track the multiple off-springs starting at branching contours • to interpolate over transient evidence “gaps” • JetStream – a method with particle filtering

8. Probabilistic contour tracking • Tracking contours in still images is an unconventional tracking problem because of the absence of a real notion of time • The “time” is only associated with the growing of the contour in the image plane – consider a spatial chain as a temporal chain • Contrary to standard tracking problems where data arrive one bit after another as time passes by, the whole set of data y is standing there at once • There is no straightforward way of tuning the “speed”, or equivalently the length of successive moves

9. Tracking framework - dynamics • xi – points in the plane R2 • (x0, x1,...,xn) - curve in some standard way, e.g., the xi’s are the vertices of a polyline, starting from x0then moving along the contour to xn • dynamics: • assuming second order dynamics with some kernel q: • then a priori density is:

10. Tracking framework - measurement • measurement y (observed image) conditioned on x0:n is considered as independent spatial process which is not true in reality but it is a reasonable approximation: • where Ώ is a discrete set of measurements locations in the image plane • the conditional distributions depend only on whether or not point u is contained in contour x0:n • in particular, for any two points u along the contour the corresponding conditional distributions are identical, similarly for any two points in the background

11. Tracking framework - measurement • Therefore, each is either ponif u belongs to the contour x0:n, or poffif not: • Finally: • where

12. Iterative computation of posterior • Function can be considered as a cost function • Expressed as the minimization of this function, the contour extraction problem then amounts to seeking the maximum a posteriori (MAP) estimate • Posterior densities can be computed recursively: • but there is no closed form of the pi • pican be approximatedby a finite set of M sample paths (the ‘particles’) • Best path at step i can be which is a Monte Carlo approximation of posterior expectation

13. Iterative computation of posterior • Prediction: each path is grown one step by sampling from the proposal density function f • If the paths are samples from pithen the extended paths are samples from f*pi • Since we want samples from distribution pi+1, extended paths are weighted according to ratio • The resulting weighted path set now provides an approximation of the target distribution pi+1 • M paths are drawn with replacement from the weighted set • The weights are:

14. Likelihood ratio l • Measurement: the norm of the luminance (or color) gradient • poffpon • poff – exponential distribution • pon - complex mixture, better keep as less informative as possible

15. Likelihood ratio l • The direction of the gradient also retains precious information that a data model based only on gradient norm neglects: the distribution of Ψ is symmetric, and it becomes tighter as the norm of the gradient increases

16. Likelihood ratio l • However, at corners, the norm of the gradient is usually large but its direction cannot be accurately measured • Using a standard corner detector, each pixel u is associated with a label c(u) = 1 if a corner is detected, and 0 otherwise • Where a corner has been detected assumption is that distribution is uniform

17. Dynamics q • Because of the absence of natural time, it is better to consider a dynamics with fixed step length d. The definition of second order dynamics then amounts to specifying an a priori probability distribution on direction change Θi • Finally: • To allow for abrupt direction changes at the locations where corners have been detected, the normal distribution is mixed with a small proportion of uniform distribution

18. Proposal sampling function f • With choice f = q, corners will be mostly ignored since the expected number of particles undertaking drastic direction changes is vM, where typically ν= 0.01 and M = 100 • At locations where no corners are detected, the proposal density is the normal component of the dynamics. • If location lies on a detected corner, the next proposed location is obtained by turning of an angle picked uniformly • Therefore:

19. Proposal sampling function f

20. JetStream - iteration

21. Experimental Results – Interactive cut-out • The extraction of a region of interest from one image • In practice, JetStream is run for a fixed number n of steps (100 in our experiments) from initial conditions x0:1 chosen by the user. • If the result is satisfactory, n more steps are undertaken. • If not, a restart region within the particle flow, and an associated restart direction, can be chosen by the user

22. Experimental Results – Interactive cut-out • More sophisticated user interaction • Provide the user with the facility to place one or more dams, defined as regions Rk where:

23. Experimental Results – Interactive cut-out

24. Experimental Results – Road extraction • In the specific context of road extraction, one is in fact interested in recovering “ribbons” • Using JetStream as defined previously results in paths jumping from one side of the ribbon to the other

25. Experimental Results – Road extraction • The state space is extended to include a width variable mi, which indicates the distance at step i between the two sides of the ribbon • It expresses that xi+ and xi- defined as • are on a contour while xi is not:

26. Experimental Results – Road extraction • A simple first order dynamics is chosen for m:

27. Experimental Results – Road extraction

28. Conclusion • JetStream is applicable if there are not sharp corners in the images • If sharp corners exist or the image’s intensity distribution is complex JetStream doesn’t provide good results

29. References • [1] M. Pérez, A. Blake, and M. Gangnet. JetStream : Probabilistic contour extraction with particles. In Int. Conf. on Computer Vision, ICCV’ 2001, Vancouver, Canada, July 2001.