Variational Pairing of Image Segmentation and Blind Restoration

1. Variational Pairing of Image Segmentation and Blind Restoration Leah Bar Nir Sochen*Nahum Kiryati School of Electrical Engineering *Dept. of Applied Mathematics Tel Aviv University

2. Segmentation: images meet concepts Borrowed from Georges Koepfler

3. Segmentation: images meet concepts Borrowed from Georges Koepfler Formalization (Mumford & Shah) Segmentation by minimizing a functional min [(fidelity to image) + β (gradients within segments) + α (total edge length)]

4. Segmentation: images meet concepts Borrowed from Georges Koepfler Formalization (Mumford & Shah) Segmentation by minimizing a functional min [(fidelity to image) + β (gradients within segments) + α (total edge length)] Calculus of Variations PDE’s Numerical Techniques Linear Systems of Equations

5. Mumford-Shah Segmentation fidelity to image gradients within segments total edge length Ω: image domainK: edge setf: segmented imageg: observed image

6. Mumford-Shah Segmentation fidelity to image gradients within segments total edge length Ω: image domainK: edge setf: segmented imageg: observed image Problem: Discontinuities in the domains (Ω/K, K) make minimization difficult

7. Mumford-Shah Segmentation fidelity to image gradients within segments total edge length Ω: image domainK: edge setf: segmented imageg: observed image Problem: Discontinuities in the domains (Ω/K, K) make minimization difficult Solution: Continuous approximation of F(f,K) (Gamma-convergence framework)

8. Mumford-Shah Segmentation fidelity to image gradients within segments total edge length Ω: image domainK: edge setf: segmented imageg: observed image Problem: Discontinuities in the domains (Ω/K, K) make minimization difficult Solution: Continuous approximation of F(f,K) (Gamma-convergence framework) fidelity to image gradients in segments total edge length v(x):smooth function v(x)~0 at edges v(x)~1 otherwise (in segments) (Ambrosio & Tortorelli, 1990)

9. In a blurred image, edges are degraded and segmentation is difficult.

10. Image Restoration Given the image g and the blur kernel h, restore the original image f . 1. Brute force ... ill posed. Minimize 2. Tikhonov regularization ... oversmoothing. Minimize 3. Total Variation (TV) regularization ... better edge preservation. Minimize

11. Blind Image Restoration Given the image g, restore the original image f (and the blur-kernel h). - Ill posed (1): sensitivity to small changes in g. - Ill posed (2): maybe the original image was already blurred?

12. Blind Image Restoration Given the image g, restore the original image f (and the blur-kernel h). - Ill posed (1): sensitivity to small changes in g. - Ill posed (2): maybe the original image was already blurred? Chan & Wong (1998) TV-regularization with respect to both the image and the kernel. Minimize - The restored image is very sensitive to the recovered kernel - The recovered kernel depends on the contents of the image (bad news)

13. Chan & Wong - The recovered kernel depends on the contents of the image. source image isotropic blur blind restoration recovered kernel

14. Chan & Wong (1998) - Performance original blurred restored isotropic gaussian kernel, =2.1 - The restored image is very sensitive to the recovered kernel. - The recovered kernel depends on the contents of the image. recovered kernel

15. In blind image restoration, one can’t get it all. Borrowed from Mickey Mouse (The Sorcerer’s Apprentice)

16. Some related work... (Blind) Restoration Segmentation You & Kaveh, 1996 Vogel & Oman, 1998 Mumford & Shah, 1989 Kim et al, 2002 Chan & Wong, 1998 Chambolle, 1995 Carasso, 2001 Hewer et al, 1998 Mathematics, Foundations Ambrosio & Tortorelli, 1992 Tikhonov & Arsenin, 1977 Rudin, Osher & Fatemi, 1992 Aubert & Kornprobst, 2002

17. The suggested approach Why? - Segmentation is hard, but easier if the image is sharp - Blind restoration is hard, but easier if the edges are known What? Blind restoration and segmentation as mutually supporting processes How? Unified variational framework, iterative algorithm

18. Combined objective functional • Mumford-Shah segmentation + blind restoration • Make it work: Use the Γ-convergence approximation • Make it work well: Use a parametric blur-kernel fidelity, parametric blur gradients in segments “smooth v” “total edge length” “wide kernel” Reminderv(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments)

19. Minimizing the functional • Iterate • Minimize with respect to v(segmentation / edge detection) • Minimize with respect to f(image restoration) • Minimize with respect to σ(blur-kernel recovery)

20. Iterative Minimization Equations • Minimization with respect to v(Euler equation)

21. Iterative Minimization Equations • Minimization with respect to v(Euler equation) • Minimization with respect to f(Euler equation)

22. Iterative Minimization Equations • Minimization with respect to v(Euler equation) • Minimization with respect to f(Euler equation) • Minimization with respect to σ(derivative)

23. Iterative Minimization Equations • Minimization with respect to v(Euler equation) • Minimization with respect to f(Euler equation) • Minimization with respect to σ(derivative) Calculus of Variations PDE’s Numerical Techniques Linear Systems of Equations

24. Frequently Asked Questions • What are the initial values? We use f=g (output=input), v=1 (no edges) and σ=ε(small blur). • What is the stopping condition? We stop when the radius σ of the recovered kernel has converged. • Does it converge? To a global optimum? • Nice theoretical properties • Excellent experimental behavior • Additional analytic work in progress Typical convergence: σ vs. iteration number

25. Experimental Results (1): Known Blur Kernel Blurred

26. Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration

27. Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration Suggested restoration

28. Experimental Results (1): Known Blur Kernel Blurred Lucy-Richardson restoration Suggested restoration Suggested edges (v function)

29. Experimental Results (2): Blind Blurred

30. Experimental Results (2): Blind Blurred Chan-Wong restoration

31. Experimental Results (2): Blind Blurred Chan-Wong restoration Suggested restoration

32. Experimental Results (2): Blind Blurred Chan-Wong restoration Suggested restoration Suggested edges (v function)

33. Experimental Results (3): Blind Blurred

34. Experimental Results (3): Blind Blurred Chan-Wong restoration

35. Experimental Results (3): Blind Blurred Chan-Wong restoration Suggested restoration

36. Experimental Results (3): Blind Blurred Chan-Wong restoration Suggested restoration Suggested edges (v function)

37. Experimental Results (4): Blind Blurred

38. Experimental Results (4): Blind Blurred Chan-Wong restoration

39. Experimental Results (4): Blind Blurred Chan-Wong restoration Suggested restoration

40. Experimental Results (4): Blind Blurred Chan-Wong restoration Suggested restoration Suggested edges (v function)

41. Conclusions Image segmentation and (blind) restoration, sont les mots qui vont tres bien ensemble* . The whole is larger than the sum of its parts (in this case). Blind restoration is easier if you can use a parametric blur model. *these are words that go together well.