Download Presentation
## Super-Resolution

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Super-Resolution**Barak Zackay Yaron Kassner**Outline**• Introduction to Super-Resolution • Reconstruction Based Super Resolution • An Algorithm • Limits on Reconstruction Based Super Resolution • Example Based Super Resolution • Halucination • Example Based • Single Image Super Resolution • Summary**Definition of the Problem**• Super-resolution is the process of combining multiple low resolution images to form a higher resolution one. • No cheating! • Resulting image should represent reality better than all the input images.**Physical Properties**• Each camera suffers from some inherent optical issues: • Finite size of the aperture - generates blur, modeled by the Point-Spread-Function (PSF). • Noise**Mathematical Model**• Each pixel in the resulting image is given by: • Loi(m) – the i-th LR image in pixel m. • Ei(x) – total photon count from the direction x • PSFi – Point Spread Function**LR**HR HR HR Deresolution • Given HR image • Project to LR image • Each LR pixel is a linear combination of HR pixels**LR**LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR HR HR HR HR HR HR HR HR HR HR HR Reconstruction-based Super Resolution • Reconstruct hidden HR pixels out of known linear combinations.**Example-BasedSuper Resolution**• Use prior knowledge to reconstruct a HR image. Prior Knowledge of faces**Reconstruction Based Super Resolution**from Improving Resolution by Image Registration Michal Irani and Shmuel Peleg**Basic Idea**The HR image should create the LR images when deresoluted.**Notation**: The kth observed LR image. : The approximation to the HR image after n iterations. : The LR image obtained by applying the simulated imaging process to . : The point spread function of the imaging blur. : a HR pixel : a LR pixel influenced by x : The center of the receptive field of y.**Problem Formulation**Find a HR image , that gives .**Algorithm Overview**• Register the LR images. • Guess the HR image . • Iteration n: • Simulate the imaging process to create from . • Compare and . • Correct in the direction of the error. • output**LR**LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR LR HR HR HR HR HR HR HR HR HR HR HR Registration**Iteration**Take the current guess. Decrease its resolution to get Update each HR pixel x according to the error in all LR pixels (y) it influences.c is a constant normalizing factor. c is a constant normalizing factor. Yk,x is the group of all pixels y that are influenced by x. is a back-projection kernel applied on that represents the way the HR pixel x should be updated from y.**Wasach**One of three input images Initial guess (average of input images) Output**Debluring**Original Image Blurred Image Restored Image**Wasach**Blurred Image Initial Guess Restored Image**Limits on Reconstruction Based Methods**from Limits on Super-Resolution and How to Break Them Simon Baker and Takeo Kanade**Large Magnification Factor is Problematic**• Large magnification factor causes: • Overly smooth HR image • Fine details are not recovered • An explanation is needed.**LR**HR HR HR Evil Example • Suppose we want to increase the resolution by exactly M=2. • Lets look on a checkboard like scene, where each pixel is either white or black.**Information is Inherently Missing**• The resulting image would be grey independently from the offset of the LR image! • Conclusion: some information is inherently missing on our LR images!**When M is not an Integer**LR HR HR HR**Limits of Super-Resolution**• Size of LR images: N pixels. • Size of HR image: NM 2pixels. • Each HR pixel can be added noise of amplitude smaller than M 2which wont change the LR image! • Volume of possible HR solutions: O(M 2N) 1 • It can be shown that under practical considerations the effective magnification factor (M) is bounded by 1.6, no matter how many LR images are taken2. 1 Limits on Super-Resolution and How to Break Them, Simon Baker and Takeo Kanade 2 Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation, Zhouchen Lin, and Heung-Yeung Shum**Break**• Introduction to Super-Resolution • Reconstruction Based Super Resolution • An Algorithm • Limits on Reconstruction Based Super Resolution • Example Based Super Resolution • Halucination • Example Based • Single Image Super Resolution • Summary**Introduction to Example-Based Super Resolution**• Reconstruction constraints are not enough. • One has to use prior knowledge of the image to break the reconstruction limits. • The following algorithms will use priors learned from databases of example images.**Recogstruction or Hallucination**from Limits on Super-Resolution and How to Break Them Simon Baker and Takeo Kanade**General Idea**• Find a HR image Su that satisfies two kinds of constraints: • Reconstruction constraints: When projected to the LR dimensions, the image is similar to the observed input images. • Recognition constraints: The pixels of Su should resemble pixels from images in the DB that where found to have similar features to the observed LR images’ features.**MAP formulation**• To solve the problem, given the LR images, we need to find the HR image that maximizes • - Su: the HR image • - Lo: the LR images Reconstruction Constraints Recognition Constraints**Reconstruction Constraints**The probability of the LR images given the HR image can be computed from the distance between the deresoluted HR image and the LR images. : the noise variance PSF: Point Spread Function : The pixel in Lo that corresponds to pixel z in Su. m: a LR pixel index**Recognition: LR features**• We use “Parent Structures” to describe LR features.**Recognition: Choosing the Pixels from the DB**PS = Parent Structure F = Features – like First deriviative, or Laplacian**Formulation of Recognition Constraints**• Instead of estimating the probability of the HR image, Su, we estimate its probability given each pixel’s “recognition”. H0 – Horizontal derivative V0 – Vertical derivative. - Variance of the recognition errors. T - the training images. BI – best images for the pixels of the LR images. BP – best pixel indices in the best images for the pixels of the LR images. Ci,BP,BI – Class of all images that would have the Best corresponding Images BI, and the Best corresponding Pixels BP in the db. - The function that fits a LR pixel index to the corresponding HR pixel index. 2k – the ratio between the HR image scale and the LR image scale.**Maximization**• Note that the function we need to maximize is quadratic with the HR image’s pixels. • Do gradient descent.**Algorithm Summary**• Preliminary work: • Take a training set of images. • Build a DB that matches parent structures to HR pixels. • Compute the reconstruction constraints. • For each LR image: • For each HR pixel index: • Search for the corresponding parent structure in the DB. • Find the HR image that fits best both the reconstruction constraints and the HR pixels extracted from the database.**Example Based Super Resolution**William T. Freeman, Thouis R. Jones and Egon C. Pasztor**Algorithm Overview**• Construct a DB of matching LR-HR patches • Algorithmically find the most coherent patch assignment to generate a good image**Constructing the DB**• Given a DB of images • Make a table from LR patches to HR patches. • Each image in the DB is treated as follows: • Take each 7x7 patch from the image and deresolute into a 5x5 patch • Normalize the 5x5 patches to have the same mean and relative contrast. • Arrange the DB by the low frequencies of the LR patches