Advanced Methods for Image Segmentation

1. Advanced Methods for Image Segmentation Ilya Pollak Purdue University November 10, 2008

2. Outline • Image segmentation examples • Different classes of image segmentation methods • Methods based on scale-spaces: • Linear Gaussian scale-space • Perona-Malik equation • Stabilized inverse diffusion equation (SIDE) • Vector-valued SIDEs • Applications to Digital Microscopy Data

3. Segmentation of a Freckle Defect in Single Crystal Nickel Images freckle021 freckle055 freckle059 freckle108

4. Segmentation of Dermatoscopic Images of Skin Lesions

5. Segmentation of a SAR Image Original Method 1 Method 2

6. Document Image Segmentation Document and its segmentation Classification of regions

7. Activation Detection in Functional MRI Left hemifield stimulus Right hemifield stimulus

8. References • L. Huffman, J. Simmons and I. Pollak. Segmentation of digital microscopy data for the analysis of defect structures in materials using nonlinear diffusions. Presented at the Conference on Computational Imaging, IS&T/SPIE 20th Annual Symposium on Electronic Imaging Science and Technology, January 27-31, 2008, San Jose, CA. In Computational Imaging VI, Proceedings of SPIE, C.A. Bouman, E.L. Miller, and I. Pollak, Eds. • X. Dong and I. Pollak. Multiscale Segmentation with Vector-Valued Nonlinear Diffusions onArbitrary Graphs. IEEE Trans. Im. Proc., 15(7):1993-2005, July 2006. • M.G. Fleming, C. Steger, J. Zhang, J. Gao, A.B. Cognetta, I. Pollak and C.R. Dyer. Techniques for a structural analysis of dermatoscopic imagery. Computerized Medical Imaging and Graphics, 22(5):375-389, 1998. • I. Pollak, A.S. Willsky and H. Krim. Image segmentation and edge enhancement with stabilized inverse diffusion equations. IEEE Trans. Im. Proc., 9(2):256-266, Feb 2000. • W. Wang, I. Pollak, T.-S. Wong, C.A. Bouman, M.P. Harper and J.M. Siskind. Hierarchical stochastic image grammars for classification and segmentation. IEEE Trans. Im. Proc., 15(10):3033-3052, October 2006. • J. Wei, T. Talavage and I. Pollak. Modeling and activation detection in fMRI data analysis. In Proc. IEEE Statistical Signal Processing Workshop, pp. 141-145, August 26-29, 2007, Madison, WI.

9. Some Classes of Segmentation Methods • Bayesian classification (C. Bouman, Tue AM): • Construct a prior model for each pixel class, estimate pixel classes from the observed data • Active contours (S. Acton, J. Kovačević, Tue AM): • Throw several curves onto the image and let them evolve towards objects of interest by minimizing an energy • Graph-based methods (S. Wang, Tue AM): • E.g., compute an optimal graph cut • Variational methods (C. Bajaj, Tue PM): • Set up and solve a global variational problem (e.g. Mumford-Shah) • Region merging: • Recursively merge regions to reduce an energy • Multiscale methods: • Repeatedly coarsen with “low-pass” filters, segment coarsened versions

10. To simplify, consider 1D segmentation first Take the first row of pixels and plot their intensities as a function of position:

11. Noise Removal with a Linear Gaussian Scale-Space = * = * * =

12. Linear Gaussian Scale-Space and Heat Equation = * where Heat equation

13. 1D Example

14. 2D Example Fine scale Coarse scale

15. Perona-Malik Equation • Edge sharpening for • Ill-posed, • Semi-discrete and discrete versions are well-posed:

16. Semi-Discrete Perona-Malik Equation We focus on the semi-discrete equation, which is a system of ODE’s: for n=1,…,N

17. 2D Example Large K Small K

18. Stabilized Inverse Diffusion Equations (SIDE’s) • Pollak, Willsky, Krim, Trans. Image Proc., Feb. 2000. • The limit of Perona-Malik as K approaches zero. • Semi-discrete version is well-posed. • Sliding modes on the surfaces • I.e., • Scale-space consists of piecewise-constant signals. • Will converge to a constant within finite time.

19. Another 1D Example

20. SIDEs (continued) • The solution automatically produces fine-to-coarse sequence of segmentations. • This process is a multiscale region merging algorithm which starts with singleton regions. • Sliding mode dynamics for the i-th region of length and intensity • In 2D, • where

21. 2D Example 1000 regions 100 regions 2 regions

22. 2D Examples

23. Segmentation of Vector-Valued Images: Motivation How to segment an image composed of several textures whose average intensities may not be very different? One possible answer: convert it into several “feature” images which associate different intensities with different textures

24. Scalar-Valued Image Vector-Valued Image Filter 1 Filter 2 Filter N Original image Filter bank Vector-valued feature image

25. Example: Gabor Energy Features • Useful for analyzing textures at different scales, frequencies, orientations • Filter an image with Gabor filter pairs at many scales, frequencies, orientations • For each Gabor filter pair, take the energy image

26. Gabor Energy Features

27. SIDE as a Gradient Descent + = +

28. SIDE as a Gradient Descent

29. From Scalar-Valued to Vector-Valued SIDEs • Define an inner product between feature (or color) vectors. • Define an inner product between two vector-valued images in • Perform recursive region merging, with gradient descent on between merges.

30. Vector-Valued SIDE • is the vector intensity of region Riat scale t • a(Ri) is an application-specific positive weight function, e.g., the area of region Ri • b(Ri,Rj) is an application-specific positive weight function which relates neighboring regions Ri and Rj, e.g., the length of the boundary between Ri and Rj • E(x) is the energy function, e.g., • This choice of E(x) pushes intensities μ of neighboring regions to equality, therefore encouraging a coarse segmentation

31. Multiscale Segmentation Algorithm • Given a segmentation S of image u, evolve the descent equation until the intensities of some pair of neighbor regions are equal. • Merge the two regions by removing them both from S and adding their union to S. • If the desired number of regions is reached, stop. Else, go to Step 1.

32. Texture Segmentation Example

33. Texture Segmentation Example

34. Another Example

35. Segmentation of a Natural Image

36. Segmentation of a Freckle Defect in Single Crystal Nickel Images freckle021 freckle055 freckle059 freckle108

37. Multi-Tilt Segmentations Combining segmentations from multiple images + + + =

38. Segmentation Fusion Locate landmarks visible in every image Register images using landmarks Combine SIDE segmentations of each image to create a composite segmentation

39. Carbides as Landmarks Carbides appear as dark spots in the material visible at most of the tilt angles. These are used as landmarks. Some visible carbides

40. Carbides detected by thresholding the intensity of pixels in regions where less than 1% of pixels are above threshold Landmark Detection Carbide pixel extraction

41. Image Registration Unaligned segmentation boundaries (4001 in red, 7001 in green) Aligned through 2D correlation Aligned by affine-transforming 7001 to match 4001

42. Segmentation Fusion Directly overlaying segmentations from multiple tilts produces many new “regions” due to slight differences in SIDE outputs on different images. Zoom in on shared region borders Overlaid boundaries of 4001 and transformed 7001

43. Removing Small Extraneous Regions Assign a unique region label to every region in the original individual segmentations. Each region in the combined segmentation then has two original region labels from the original segmentations. • The overlap of Region A from image 1 with Region B from image 2 may produce multiple new contiguous regions. New region labels indicated by color

44. New regions are defined as “extraneous” if they make up less than ε% of both of the original regions the pixels belonged to. Extraneous regions are then combined with the neighboring region that shares one of the same original region labels and contains the most pixels from that original region. Removing extraneous regions results in 426 regions total Direct overlay contains 1960 regions Zoom of original overlay labeling Zoom of overlay labeling after extraneous region removal Removing Small Extraneous Regions

45. Merged 4001-5001-6001 regions Merged 4001-5001 regions 4001 regions Merged 4001-5001-6001-7001 regions Transformed 5001 regions Transformed 6001 regions Transformed 7001 regions Segmentation Fusion for Four Tilts

46. Segmentations and Images Combined region boundaries 4001 portion 5001 portion 6001 portion 7001 portion

47. Summary • SIDE is a flexible, robust segmentation method • Once parameters are selected, no human interaction • Can work in conjunction with any feature extraction method and any image registration/fusion method • Has been successfully applied to natural images, medical images, and microscopy images of materials

48. Acknowledgments • Jeff Simmons of AFRL • Data: Michael Uchic and Jonathan Spowart of AFRL • Past Funding: AFRL, Wright-Patterson AFB (Dr. Dallis Hardwick, program manager) under subcontract USAF-5212-STI-SC-0026 from GeneralDynamics Information Technology, Inc. (May-Nov 2007) • Future Funding: ??????

49. Future Work • Improvements to multi-tilt fusion • Joint 3D/4D segmentation • Applications to other images of materials • Investigation of feature extraction methods • Parameter learning • Prior modeling: designing penalty functions a and b and energy function E • Theoretical analysis: • Total-variation minimization for u0 = x + w • Inverse problems: u0 = Ax + w • Non-Gaussian noise