super resolution l.
Download
Skip this Video
Download Presentation
Super-Resolution

Loading in 2 Seconds...

play fullscreen
1 / 34

Super-Resolution - PowerPoint PPT Presentation


  • 258 Views
  • Uploaded on

Super-Resolution. Digital Photography CSE558, Spring 2003 Richard Szeliski. Super-resolution. convolutions, blur, and de-blurring Bayesian methods Wiener filtering and Markov Random Fields sampling, aliasing, and interpolation multiple (shifted) images prior-based methods MRFs

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Super-Resolution' - clem


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
super resolution

Super-Resolution

Digital PhotographyCSE558, Spring 2003Richard Szeliski

super resolution2
Super-resolution
  • convolutions, blur, and de-blurring
  • Bayesian methods
    • Wiener filtering and Markov Random Fields
  • sampling, aliasing, and interpolation
  • multiple (shifted) images
  • prior-based methods
    • MRFs
    • learned models
    • domain-specific models (faces)- Gary

Super-Resolution

linear systems
Linear systems
  • Basic properties
  • homogeneity T[a X] = a T[X]
  • additivity T[X1+X2] = T[X1]+T[X2]
  • superposition T[aX1+bX2] = aT[X1]+bT[X2]
  • Linear system  superposition
  • Examples:
  • matrix operations (additions, multiplication)
  • convolutions

Super-Resolution

signals and linear operators
Signals and linear operators
  • Continuous I(x)
  • Discrete I[k] or Ik
  • Vector form I
  • Discrete linear operator y = A x
  • Continuous linear operator:
  • convolution integral
  • g(x) = sh(,x) f() d, h(,x): impulse response
  • g(x) = s h(-x) f() d= [f * h](x)shift invariant

Super-Resolution

2 d signals and convolutions
2-D signals and convolutions
  • Continuous I(x,y)
  • Discrete I[k,l] or Ik,l
  • 2-D convolutions (discrete)
  • g[k,l] = m,n f[m,n] h[k-m,l-n]
  • = m,n f[m,n] h1[k-m]h2[l-n] separable
  • Gaussian kernel is separable and radial
  • h(x,y) = (22)-1exp-(x2+y2)/2

Super-Resolution

convolution and blurring
Convolution and blurring

Super-Resolution

fourier transforms
Fourier transforms
  • Project onto a series of complex sinusoids
  • F[m,n] = kf[k,l] e-i 2(km+ln)
  • Properties:
  • shifting g(x-x0) exp(-i 2fxx0)G(fx)
  • differentiation dg(x)/dx i 2fxG(fx)
  • convolution [f * g](x) [F G](fx)

Super-Resolution

blurring examples
Blurring examples
  • Increasing amounts of blur + Fourier transform

Super-Resolution

sharpening
Sharpening
  • Unsharp mask (darkroom photography):
  • remove some low-frequency content y’ = y + s (y – g * y)spatial (blur, sharp) freq (blur,sharp)

Super-Resolution

sharpening result
Sharpening - result
  • Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2)

Super-Resolution

deconvolution
Deconvolution
  • Filter by inverse of blur
  • easiest to do in the Fourier domain
  • problem: high-frequency noise amplification

Super-Resolution

bayesian modeling
Bayesian modeling
  • Use prior model for image and noise
  • y = g * x + n, x is original, y is blurred
  • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2)
  • -log p(x|y) |y – g*x|2σn-2 + |x|2σx-2where the norm || is summed squares over all pixels

Super-Resolution

parseval s theorem
Parseval’s Theorem
  • Energy equivalence in spatial ↔ frequency domain
  • |x|2 = |F(x)|2
  • -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2
  • least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]

Super-Resolution

wiener filtering
Wiener filtering
  • Optimal linear filter given noise and signal statistics
  • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
  • low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur)
  • high frequencies: X(f) ≈ G(f) σn-2σx2 Y(f)attenuate by blur (gain)

Super-Resolution

wiener filtering white noise prior
Wiener filtering – white noise prior
  • Assume all frequencies equally likely
  • p(x) ~ N(0,σx2)
  • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
  • solution is too noisy in high frequencies

Super-Resolution

wiener filtering pink noise prior
Wiener filtering – pink noise prior
  • Assume frequency falloff (“natural statistics”)
  • p(X(f)) ~ N(0,|f|-βσx2)
  • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2]
  • greater attenuation at high frequencies G(f) H(f)

Super-Resolution

markov random field modeling
Markov Random Field modeling
  • Use spatial neighborhood prior for image
  • -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm:
    • ρ(v) = v2: quadratic norm ↔ pink noise
    • ρ(v) = |v|: total variation (popular with maths)
    • ρ(v) = |v|β: natural statistics
    • ρ(v) = v2,|v|: Huber norm[Schultz, R.R.; Stevenson, IEEE TIP, 1996]

i

j

Super-Resolution

mrf estimation
MRF estimation
  • Set up discrete energy (quadratic or non-)
  • -log p(x|y) σn-2|y – Gx|2 + ijCρ(xi-xj)where G is sparse convolution matrix
  • quadratic: solve sparse linear system
  • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)

Super-Resolution

sampling a signal
Sampling a signal
  • sampling:
    • creating a discrete signal from a continuous signal
  • downsampling (decimation)
    • subsampling a discrete signal
  • upsampling
    • introducing zeros between samples
  • aliasing
    • two sampled signals that differ in their original form (many → one mapping)

Super-Resolution

sampling
Sampling

interpolation

Super-Resolution

nyquist sampling theorem
Nyquist sampling theorem
  • Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency
  • s = 2 / ns > 2 0

Super-Resolution

box filter top hat
Box filter (top hat)

Super-Resolution

ideal low pass filter
Ideal low-pass filter

Super-Resolution

simplified camera optics
Simplified camera optics
  • Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian
  • Integrate = box filter
  • Sample = produce single digital sample
  • Noise = additive white noise

Super-Resolution

aliasing
Aliasing
  • Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1)
  • shift each image in previous pipeline by 1

Super-Resolution

aliasing less
Aliasing - less
  • Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2)
  • shift each image in previous pipeline by 1

Super-Resolution

multi image super resolution
Multi-image super-resolution
  • Exploit aliasing to recover frequencies above Nyquist cutoff
  • kσn-2|yk – Gkx|2 + ijCρ(xi-xj)where Gk are sparse convolution matrices
  • quadratic: solve sparse linear system
  • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …)
  • projection onto convex sets (POCS)

Super-Resolution

multi image super resolution29
Multi-image super-resolution
  • Need:
  • accurate (sub-pixel) motion estimates(Wednesday’s lecture)
  • accurate models of blur (pre-filtering)
  • accurate photometry
  • no (or known) non-linear pre-processing(Bayer mosaics)
  • sufficient images and low-noise relative to amount of aliasing

Super-Resolution

prior based super resolution
Prior-based Super-Resolution
  • “Classical” non-Gaussian priors:
  • robust or natural statistics
  • maximum entropy (least blurry)
  • constant colors (black & white images)

Super-Resolution

example based super resolution
Example-based Super-Resolution
  • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002
  • learn the association between low-resolution patches and high-resolution patches
  • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence

Super-Resolution

example based super resolution32
Example-based Super-Resolution
  • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002

Super-Resolution

references classic
References – “classic”
  • Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239.
  • Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011.
  • Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.
  • Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834.
  • Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1.
  • Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.

Super-Resolution

references strong priors
References – strong priors
  • Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2
  • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002
  • Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.
  • Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8.

Super-Resolution