1. Best Huber-MAP reconstruction. Error compared to known ground truth image. RMSE = 15.5 grey levels. Best registration-integrating reconstruction. Note improvement of black-white edge detail in some letters on the bottom line. RMSE = 14.8 grey levels. image y(1) Lighting changes with parameters λ. Blur by point-spread function. Corrupt with additive Gaussian noise. Warp with parameters . Decimate by zoom factor. Image x. image y(K) Inputs 1 & 10 xMAP using a Huber prior, assuming perfect knowledge of image registrations. The same problem, but with registrations perturbed by δ ~ N ( 0, 0.12 ) low-resolution pixels. xMAPusing a Gaussian image prior, giving softer edges. Input 16/16 ML1 ML2 abs(ML1-ML2 ) ground truth Input 1/16 Bayesian Image Super-resolution, Continued Lyndsey C. Pickup, David P. Capel, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford Introduction Key Idea: Integrate over the unknown registrations Synthetic Data Example The image registrations can be viewed as “nuisance parameters” to be integrated out of the super-resolution problem in a Bayesian setting. Goal: Multi-frame super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. The problem is made more difficult because the sub-pixel displacements -- as well as lighting changes and image blur parameters -- must be recovered from this noisy set of low-resolution input images. This alternative: accept registration uncertainty, and integrate over the resulting distribution to find super-resolution image directly from inputs. Most methods, including Tipping & Bishop’s Bayesian Image Super-resolution work, fix the registration first, then compute the high-resolution image. First and last of the 16 input images from the synthetic dataset. Generative model for super-resolution Assume small perturbations or corrections δ in the set of registration parameters: Take a Taylor series expansion of the error in terms δ: Now find –log(p(x|{y(k)})) by integrating over δ: Results on real data The goal is to reconstruct the high-resolution image x (right) from a set of noise inputs {y(k)} generated according to the model above. 10 input images, each with 4 registration parameters (two geometric, two photometric). Images were registered using a simple iterative intensity-based method, typically accurate to within 0.1 pixels. Outputs are at a zoom factor of 4. where H is a block diagonal sparse matrix, so for many covariance assumptions, S is correspondingly sparse also. Typically C can be assumed diagonal, and L can be solved using gradient descent optimization methods. Registration-Integrating (detail) Maximum A Posteriori Solution Huber-MAP: regularizes the smooth areas well, though gives slightly over-crisp edges. Tipping & Bishop: typical Gaussian over-smoothing, though weakening the prior would risk over-fitting to the noise. Registration-integrating: shows letters clearly but also gives good regularization on the constant white background areas. The regular Huber-MAP objective function for finding x is where G(x) is the set of image gradients at each pixel, and  is the Huber function, defined: Huber-MAP (detail) Advantages of Registration-Integrating There are a number of advantages to integrating over the unknown registrations rather than the high-resolution image in Bayesian image super-resolution: Tipping & Bishop (detail), extended to include photometric model. Integrating over high-resolution image 1 – Must have a Gaussian prior in order for the integral to be tractable. This is not a good model for natural image statistics and tends to blur out the image edges. 2 – The matrix in the log-determinant term of the objective function is large because the matrices tend to be NxN. 3 – The full image-integrating approach can only be run on very small input images (e.g. 9x9 pixels). Integrating over registrations 1a – Can use good image priors such as the Huber prior, which give much better results than Gaussian priors. 1b – A Gaussian prior for the registration offsets is realistic. 2 – The matrix in the log-determinant term of the objective function is small (4*K). 3 – Requires little more memory than the standard Huber-MAP algorithm, so can handle large images. Extra terms as an image prior The two terms of L involving S can be viewed as an extra prior over the high-resolution image, favouring images which are not acutely sensitive to tiny perturbations in the registrations. Such a perturbation is enough to change the noise pattern on the ML estimate drastically. Two vectors δ1 and δ2are drawn from N(0,I*0.042) and used to corrupt the registration on a synthetic dataset.