An Investigation Into Segmenting Traffic Images Using Various Types of Graph Cuts

1. An Investigation Into Segmenting Traffic Images Using Various Types of Graph Cuts Jonathan Dinger

2. Analysis of Traffic Videos Using Computer Vision Traffic footage example

3. Segmentation • Important step in video analysis • Background subtraction is often used

4. Background Subtraction • Uses pixelwise computations • Performance could be better • Better segmentation = better traffic detection

5. Graph Cuts • Use interaction between neighboring pixels • Keep objects segmented together • Better segmentations

6. Graphs • G = (V,E) • Two vertices in V called source and sink • s and t, respectively • Remaining vertices called M • Vertices in M connected to both s and t (T-links) • Vertices in M connected to neighboring vertices (N-links)

7. Graphs • T-links are uni-directional • N-links are bi-directional • Each edge has a weight • also known as a capacity • Each pixel has its own vertex

8. Definitions • Path • List of vertices connected by edges • s-t cut • Removal of edges such that all vertices have a path to either the source or the sink, but not both • Flow • Each edge has a capacity • That much flow can be pushed through each edge • Flow through a graph is the cumulative amount of flow going from the source to the sink

9. Definitions • Minimum cut/maximum flow • A cut with the smallest cost (weight) • Maximum flow that can be pushed from source to sink (capacity) • By max-flow min-cut theorem, these are equal • Graph cuts • Minimum cut on a graph

10. Augmenting Paths Algorithm • Method to find maximum flow • Augmenting path • Path where flow can be increased in all edges between vertices in path • Run search • Find augmenting path from source to sink • Add more flow to that path • Loop back to 1.

11. Augmenting Paths Example Step 1. Step 2. Step 3. Step 4.

12. Augmenting Paths Example Step 5. Step 6. Step 7. Step 8.

13. Implementations • Two separate implementations • Our implementation • Kolmogorov’s implementation http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software/maxflow-v3.01.src.tar.gz

14. Banded Graph Cuts (BGC) • Downsampled images • Cut performed over smaller area • Then upsample, and perform cut over band • Faster than graph cuts • Less detailed than graph cuts

15. Banded graph cuts

16. BGC with Laplacian Augmentation • Downsampling images loses information • Use Laplacian pyramid to store lost data • Computes difference between image and image gained by downsampling and upsampling again • Add back some of the lost detail • Resegment in areas where detail was lost

17. BGC with Laplacian Augmentation

18. GC vs BGC • BGC faster, less accurate • GC slightly slower, more accurate Graph cut Banded graph cut

19. Multi-way Graph Cut • 3 or more labels • α-β swap • Loop through label pairs • Run graph cut on current pair of labels • If current label of a pixel is not one of the pair, do not use pixel in graph cut • Graph cuts will swap some pixels with label α to label β and vice versa

20. Traffic Segmentation • Graph cut on each image • Background image computed per pixel • N is the number of images, xi is the grayscale value of the current pixel, and μ is the average grayscale value over all image frames

21. Traffic Segmentation • Find variance of each pixel • N, xi, and μ are as above • σ2 is the variance • Threshold the variance • If variance is below the threshold, do not include pixel in graph cut • Assume non-varying pixels are background pixels • Avoid divide-by-zero errors in weights

22. T-link Weights • Exponentials • x is the grayscale value of the current pixel • μ and σ2 are as above • β is a constant that forces the two functions to be equal at α standard deviations

23. T-link Weights • Absolute differences • x and μ are as above • L is the maximum possible distance between x and μ, so for grayscale images • K is a shift constant that forces f3 and f4 to be equal at α standard deviations

24. More T-link Weights • Simple Grayscale Differences • x is the grayscale value of the pixel. Cm and Cn are two grayscale values used as a basis for segmentation

25. N-link Weights • Distance • Euclidean distance between two neighboring pixels with coordinates (x1, y1) and (x2, y2) • Smoothing (similarity) • x and y are the grayscale values of two neighboring pixels. • is the maximum possible difference between the pixel values • γ is a modifier that defines the amount of smoothing that takes place

26. Results for Graph Cuts Image Graph cut Banded graph cut Augmented BGC Our graph cut Multi-way cut

27. Timing Results Computing times for cut results in milliseconds (ms)

28. Results for Traffic Segmentation (distance) Image Background Variance Exponential cut Absolute difference cut

29. More results (distance) Image Exponential Absolute difference

30. More results (distance) Image Exponential Absolute difference

31. Comparison of Distance Model with Background Subtraction Image Exponential Background subtraction

32. Results (smoothing) Image Exponential Absolute difference

33. More results (smoothing) Image Exponential Absolute difference

34. Comparison of Smoothing Model with Background Subtraction Image Absolute difference Background subtraction

35. Video Results (smoothing) Image Graph cut Background Subtraction

36. Conclusions • Segmentation performance without smoothing comparable to background subtraction • Background subtraction is faster, easier • Smoothing model • Segments larger pieces of vehicles into one section • Vehicle segmentations more “solid” • Absolute difference T-link weights combined with smoothing N-link weights give best results

37. Future Work • Use multi-way cuts to add shadow segmentation • Extend to RGB