super resolution n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Super-Resolution PowerPoint Presentation
Download Presentation
Super-Resolution

Loading in 2 Seconds...

play fullscreen
1 / 77

Super-Resolution - PowerPoint PPT Presentation


  • 449 Views
  • Uploaded on

Super-Resolution. Barak Zackay Yaron Kassner. Outline. Introduction to Super-Resolution Reconstruction Based Super Resolution An Algorithm Limits on Reconstruction Based Super Resolution Example Based Super Resolution Halucination Example Based Single Image Super Resolution Summary.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Super-Resolution


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
super resolution

Super-Resolution

Barak Zackay

Yaron Kassner

outline
Outline
  • Introduction to Super-Resolution
  • Reconstruction Based Super Resolution
    • An Algorithm
    • Limits on Reconstruction Based Super Resolution
  • Example Based Super Resolution
    • Halucination
    • Example Based
    • Single Image Super Resolution
  • Summary
definition of the problem
Definition of the Problem
  • Super-resolution is the process of combining multiple low resolution images to form a higher resolution one.
  • No cheating!
    • Resulting image should represent reality better than all the input images.
physical properties
Physical Properties
  • Each camera suffers from some inherent optical issues:
    • Finite size of the aperture - generates blur, modeled by the Point-Spread-Function (PSF). 
    • Noise
mathematical model
Mathematical Model
  • Each pixel in the resulting image is given by:
  • Loi(m) – the i-th LR image in pixel m.
  • Ei(x) – total photon count from the direction x
  • PSFi – Point Spread Function
deresolution

LR

HR

HR

HR

Deresolution
  • Given HR image
  • Project to LR image
  • Each LR pixel is a linear combination of HR pixels
reconstruction based super resolution

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

HR

HR

HR

HR

HR

HR

HR

HR

HR

HR

HR

Reconstruction-based Super Resolution
  • Reconstruct hidden HR pixels out of known linear combinations.
example based super resolution
Example-BasedSuper Resolution
  • Use prior knowledge to reconstruct a HR image.

Prior Knowledge of faces

reconstruction based super resolution1

Reconstruction Based Super Resolution

from

Improving Resolution by Image Registration

Michal Irani and Shmuel Peleg

basic idea
Basic Idea

The HR image should create the LR images when deresoluted.

notation
Notation

: The kth observed LR image.

: The approximation to the HR image after n iterations.

: The LR image obtained by applying the simulated imaging process to .

: The point spread function of the imaging blur.

: a HR pixel

: a LR pixel influenced by x

: The center of the receptive field of y.

problem formulation
Problem Formulation

Find a HR image , that gives .

algorithm overview
Algorithm Overview
  • Register the LR images.
  • Guess the HR image .
  • Iteration n:
    • Simulate the imaging process to create from .
    • Compare and .
    • Correct in the direction of the error.
  • output
registration

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

LR

HR

HR

HR

HR

HR

HR

HR

HR

HR

HR

HR

Registration
iteration
Iteration

Take the current guess.

Decrease its resolution to get

Update each HR pixel x according to the error in all LR pixels (y) it influences.c is a constant normalizing factor.

c is a constant normalizing factor.

Yk,x is the group of all pixels y that are influenced by x.

is a back-projection kernel applied on that represents the way the HR pixel x should be updated from y.

wasach
Wasach

One of three input images

Initial guess (average of input images)

Output

debluring
Debluring

Original Image

Blurred Image

Restored Image

wasach1
Wasach

Blurred Image

Initial Guess

Restored Image

limits on reconstruction based methods

Limits on Reconstruction Based Methods

from

Limits on Super-Resolution and How to Break Them

Simon Baker and Takeo Kanade

large magnification factor is problematic
Large Magnification Factor is Problematic
  • Large magnification factor causes:
    • Overly smooth HR image
    • Fine details are not recovered
  • An explanation is needed.
evil example

LR

HR

HR

HR

Evil Example
  • Suppose we want to increase the resolution by exactly M=2.
  • Lets look on a checkboard like scene, where each pixel is either white or black.
information is inherently missing
Information is Inherently Missing
  • The resulting image would be grey independently from the offset of the LR image!
  • Conclusion: some information is inherently missing on our LR images!
limits of super resolution
Limits of Super-Resolution 
  • Size of LR images: N pixels.
  • Size of HR image: NM 2pixels.
  • Each HR pixel can be added noise of amplitude smaller than M 2which wont change the LR image!
  • Volume of possible HR solutions: O(M 2N) 1
  • It can be shown that under practical considerations the effective magnification factor (M) is bounded by 1.6, no matter how many LR images are taken2.

1 Limits on Super-Resolution and How to Break Them, Simon Baker and Takeo Kanade

2 Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation, Zhouchen Lin, and Heung-Yeung Shum

break
Break
  • Introduction to Super-Resolution
  • Reconstruction Based Super Resolution
    • An Algorithm
    • Limits on Reconstruction Based Super Resolution
  • Example Based Super Resolution
    • Halucination
    • Example Based
    • Single Image Super Resolution
  • Summary
introduction to example based super resolution
Introduction to Example-Based Super Resolution
  • Reconstruction constraints are not enough.
  • One has to use prior knowledge of the image to break the reconstruction limits.
  • The following algorithms will use priors learned from databases of example images.
recogstruction or hallucination

Recogstruction or Hallucination

from

Limits on Super-Resolution and How to Break Them

Simon Baker and Takeo Kanade

general idea
General Idea
  • Find a HR image Su that satisfies two kinds of constraints:
    • Reconstruction constraints: When projected to the LR dimensions, the image is similar to the observed input images.
    • Recognition constraints: The pixels of Su should resemble pixels from images in the DB that where found to have similar features to the observed LR images’ features.
map formulation
MAP formulation
  • To solve the problem, given the LR images, we need to find the HR image that maximizes
  • - Su: the HR image
  • - Lo: the LR images

Reconstruction Constraints

Recognition Constraints

reconstruction constraints
Reconstruction Constraints

The probability of the LR images given the HR image can be computed from the distance between the deresoluted HR image and the LR images.

: the noise variance

PSF: Point Spread Function

: The pixel in Lo that corresponds to pixel z in Su.

m: a LR pixel index

recognition lr features
Recognition: LR features
  • We use “Parent Structures” to describe LR features.
recognition choosing the pixels from the db
Recognition: Choosing the Pixels from the DB

PS = Parent Structure

F = Features – like First deriviative, or Laplacian

formulation of recognition constraints
Formulation of Recognition Constraints
  • Instead of estimating the probability of the HR image, Su, we estimate its probability given each pixel’s “recognition”.

H0 – Horizontal derivative

V0 – Vertical derivative.

- Variance of the recognition errors.

T - the training images.

BI – best images for the pixels of the LR images.

BP – best pixel indices in the best images for the pixels of the LR images.

Ci,BP,BI – Class of all images that would have the Best corresponding Images BI,

and the Best corresponding Pixels BP in the db.

- The function that fits a LR pixel index to the corresponding HR pixel index.

2k – the ratio between the HR image scale and the LR image scale.

maximization
Maximization
  • Note that the function we need to maximize is quadratic with the HR image’s pixels.
  • Do gradient descent.
algorithm summary
Algorithm Summary
  • Preliminary work:
    • Take a training set of images.
    • Build a DB that matches parent structures to HR pixels.
  • Compute the reconstruction constraints.
  • For each LR image:
    • For each HR pixel index:
      • Search for the corresponding parent structure in the DB.
  • Find the HR image that fits best both the reconstruction constraints and the HR pixels extracted from the database.
example based super resolution2

Example Based Super Resolution

William T. Freeman, Thouis R. Jones and Egon C. Pasztor

algorithm overview1
Algorithm Overview
  • Construct a DB of matching LR-HR patches
  • Algorithmically find the most coherent patch assignment to generate a good image
constructing the db
Constructing the DB
  • Given a DB of images
  • Make a table from LR patches to HR patches.
  • Each image in the DB is treated as follows:
    • Take each 7x7 patch from the image and deresolute into a 5x5 patch
    • Normalize the 5x5 patches to have the same mean and relative contrast.
    • Arrange the DB by the low frequencies of the LR patches
local patch matching
Local Patch Matching
  • Match a LR patch to a HR patch from the DB, using low frequencies.
  • Get an estimation to the unknown (high) frequencies, based on the match.
  • Remaining problem: match between neighboring overlapping patches.
global patch matching
Global Patch Matching
  • Run over patches from left to right and from top to bottom
  • Match each patch its nearest neighbor in the DB using the predetermined patches as additional constraints.
wasach4
Wasach

Cubic-spline

Super-resolution

True high-resolution image

super resolution from a single image

Super Resolution From a Single Image

Daniel Glaser, Shai Bagon and Michal Irani

employing cross scale patch redundancy
Employing Cross-scale Patch Redundancy
  • Build a cascade of decreasing resolution images from the LR image.
  • For each patch in the LR image, search for its Nearest Neighbour in the even lower resolution image.
  • Take the found neighbour’s parent in the original LR image and copy it to be the HR image.
wasach6
Wasach

Bicubic interpolation

Unified single-image SR (x3)

Ground truth image

wasach7
Wasach

Bicubic interpolation

Unified single-image SR (x3)

summary
Summary
  • We have presented two basic approaches for super resolution:
    • Reconstruction-based – which simply tries to reverse the imaging process
    • Example-based – which uses example images to reconstruct the original image.
  • We have shown that there are limits to reconstruction based methods, which are independent of the number of LR images we use.
  • We have presented an algorithm that combines both approaches to achieve SR from a single image.