1 / 34

Super-Resolution

Super-Resolution. Digital Photography CSE558, Spring 2003 Richard Szeliski. Super-resolution. convolutions, blur, and de-blurring Bayesian methods Wiener filtering and Markov Random Fields sampling, aliasing, and interpolation multiple (shifted) images prior-based methods MRFs

clem
Download Presentation

Super-Resolution

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Super-Resolution Digital PhotographyCSE558, Spring 2003Richard Szeliski

  2. Super-resolution • convolutions, blur, and de-blurring • Bayesian methods • Wiener filtering and Markov Random Fields • sampling, aliasing, and interpolation • multiple (shifted) images • prior-based methods • MRFs • learned models • domain-specific models (faces)- Gary Super-Resolution

  3. Linear systems • Basic properties • homogeneity T[a X] = a T[X] • additivity T[X1+X2] = T[X1]+T[X2] • superposition T[aX1+bX2] = aT[X1]+bT[X2] • Linear system  superposition • Examples: • matrix operations (additions, multiplication) • convolutions Super-Resolution

  4. Signals and linear operators • Continuous I(x) • Discrete I[k] or Ik • Vector form I • Discrete linear operator y = A x • Continuous linear operator: • convolution integral • g(x) = sh(,x) f() d, h(,x): impulse response • g(x) = s h(-x) f() d= [f * h](x)shift invariant Super-Resolution

  5. 2-D signals and convolutions • Continuous I(x,y) • Discrete I[k,l] or Ik,l • 2-D convolutions (discrete) • g[k,l] = m,n f[m,n] h[k-m,l-n] • = m,n f[m,n] h1[k-m]h2[l-n] separable • Gaussian kernel is separable and radial • h(x,y) = (22)-1exp-(x2+y2)/2 Super-Resolution

  6. Convolution and blurring Super-Resolution

  7. Separable binomial low-pass filter Super-Resolution

  8. Fourier transforms • Project onto a series of complex sinusoids • F[m,n] = kf[k,l] e-i 2(km+ln) • Properties: • shifting g(x-x0) exp(-i 2fxx0)G(fx) • differentiation dg(x)/dx i 2fxG(fx) • convolution [f * g](x) [F G](fx) Super-Resolution

  9. Blurring examples • Increasing amounts of blur + Fourier transform Super-Resolution

  10. Sharpening • Unsharp mask (darkroom photography): • remove some low-frequency content y’ = y + s (y – g * y)spatial (blur, sharp) freq (blur,sharp) Super-Resolution

  11. Sharpening - result • Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2) Super-Resolution

  12. Deconvolution • Filter by inverse of blur • easiest to do in the Fourier domain • problem: high-frequency noise amplification Super-Resolution

  13. Bayesian modeling • Use prior model for image and noise • y = g * x + n, x is original, y is blurred • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2) • -log p(x|y) |y – g*x|2σn-2 + |x|2σx-2where the norm || is summed squares over all pixels Super-Resolution

  14. Parseval’s Theorem • Energy equivalence in spatial ↔ frequency domain • |x|2 = |F(x)|2 • -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2 • least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] Super-Resolution

  15. Wiener filtering • Optimal linear filter given noise and signal statistics • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur) • high frequencies: X(f) ≈ G(f) σn-2σx2 Y(f)attenuate by blur (gain) Super-Resolution

  16. Wiener filtering – white noise prior • Assume all frequencies equally likely • p(x) ~ N(0,σx2) • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • solution is too noisy in high frequencies Super-Resolution

  17. Wiener filtering – pink noise prior • Assume frequency falloff (“natural statistics”) • p(X(f)) ~ N(0,|f|-βσx2) • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2] • greater attenuation at high frequencies G(f) H(f) Super-Resolution

  18. Markov Random Field modeling • Use spatial neighborhood prior for image • -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm: • ρ(v) = v2: quadratic norm ↔ pink noise • ρ(v) = |v|: total variation (popular with maths) • ρ(v) = |v|β: natural statistics • ρ(v) = v2,|v|: Huber norm[Schultz, R.R.; Stevenson, IEEE TIP, 1996] i j Super-Resolution

  19. MRF estimation • Set up discrete energy (quadratic or non-) • -log p(x|y) σn-2|y – Gx|2 + ijCρ(xi-xj)where G is sparse convolution matrix • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) Super-Resolution

  20. Sampling a signal • sampling: • creating a discrete signal from a continuous signal • downsampling (decimation) • subsampling a discrete signal • upsampling • introducing zeros between samples • aliasing • two sampled signals that differ in their original form (many → one mapping) Super-Resolution

  21. Sampling interpolation Super-Resolution

  22. Nyquist sampling theorem • Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency • s = 2 / ns > 2 0 Super-Resolution

  23. Box filter (top hat) Super-Resolution

  24. Ideal low-pass filter Super-Resolution

  25. Simplified camera optics • Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian • Integrate = box filter • Sample = produce single digital sample • Noise = additive white noise Super-Resolution

  26. Aliasing • Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1) • shift each image in previous pipeline by 1 Super-Resolution

  27. Aliasing - less • Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2) • shift each image in previous pipeline by 1 Super-Resolution

  28. Multi-image super-resolution • Exploit aliasing to recover frequencies above Nyquist cutoff • kσn-2|yk – Gkx|2 + ijCρ(xi-xj)where Gk are sparse convolution matrices • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) • projection onto convex sets (POCS) Super-Resolution

  29. Multi-image super-resolution • Need: • accurate (sub-pixel) motion estimates(Wednesday’s lecture) • accurate models of blur (pre-filtering) • accurate photometry • no (or known) non-linear pre-processing(Bayer mosaics) • sufficient images and low-noise relative to amount of aliasing Super-Resolution

  30. Prior-based Super-Resolution • “Classical” non-Gaussian priors: • robust or natural statistics • maximum entropy (least blurry) • constant colors (black & white images) Super-Resolution

  31. Example-based Super-Resolution • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002 • learn the association between low-resolution patches and high-resolution patches • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence Super-Resolution

  32. Example-based Super-Resolution • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002 Super-Resolution

  33. References – “classic” • Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239. • Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011. • Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658. • Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834. • Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1. • Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001. Super-Resolution

  34. References – strong priors • Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2 • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002 • Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88. • Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8. Super-Resolution

More Related