**Super-Resolution Through Neighbor Embedding** Hong Chang, Dit-Yan Yeung and Yimin Xiong Presented By: Ashish Parulekar, Ritendra Datta, Shiva Kasiviswanathan and Siddharth Pal

**Contents** • Introduction • What is Super resolution ? • Multiframe superresolution. • Single frame superresolution. • Problem Statement. • Review on Manifold Learning • Experimental Setup • Results • Comments

**same resolution** higher resolution Definition of Resolution • Resolution ability to resolve/distinguish two objects • w.r.t image capture devices, resolution is determined by number of sensing elements in the two dimensions • Higher resolution Larger image size but, larger image size higher resolution

**superresolution algorithm** high resolution image LR frames Superresolution • Any algorithm/method which is capable of producing an image with a resolution greater than that of the input • Typically, input is a sequence of low resolution (LR) images (also referred to as frames) • LR frames displaced from each other • Have common region of interest

**Super Resolution Algorithms** • Multi-frame superresolution • Single-frame superresolution

**Multi-frame superresolution** • Introduced by Tsai and Huang • Most common approach to superresolution • Exploits information in several LR frames/images to generate a HR image • Kim, Bose and Valenzuela • Recursive wavenumber domain approach • Basic model employed

**Multi-frame superresolution** • Ur and Gross • Ensemble of spatially shifted observations • Papoulis and Brown generalized sampling theorem • Irani and Peleg – Iterative Back Projection • Initial guess of HR image • Compute LR image estimates using imaging process • Back-project the error to improve HR image • S. Lertrattanapanich • Delaunay triangulation of registered images • Surface approximation

**Multi-frame superresolution** • Nguyen and Milanfar • Wavelet superresolution • Approximate given points by a sufficiently dense dyadic set • Initially, ignore detail coefficients and obtain regularized least squares solution for approximation coefficients • Use difference between data and approximation to solve for detail coefficients • Reconstruct on HR grid using coefficients

**SGWSR 1-D Method** • M. Chappalli and N. K. Bose (2004) • HR reconstruction restricted to LR frames on semi-regular grids • Semi-regular grid tensor product of 1-D arbitrarily irregular grids • LR frames displaced only by translations • approximates far-field imaging, ex: aerial photography, satellite imaging • Based on 1-D lifting – splitting, prediction and updating • Applied on rows and subsequently on columns • Simultaneous noise reduction using hard and soft thresholding of wavelet coefficients

**+** odd sj - + sj+1 Pj Uj Uj Pj m u l t i p l e x e r + even dj - + forward transform inverse transform Lifting to construct SGW sj+1

**Result of SGWSR 1-D Method** • Results DT bilinear interpolation PSNR: 26.4996 DT bicubic interpolation PSNR: 25.6810 sample LR frame SGWSR soft thresholding PSNR: 28.5462 original

**Single-frame superresolution** • Spatial boundedness & Bandlimitedness – not simultaneously possible • exploited to reconstruct higher frequencies • LSI interpolation cannot generate new information • nonlinear and shift-varying interpolation is used • Examples • Directional filters, Adaptive interpolation • Edge-preserving interpolation, perceptual edge enhancement • Learning-based methods • Lack of information – major limitation

**Problem Formulation** Training Xsi Training Ysi ? Testing Xt Testing Yt

**Manifold Definition Revisited** A manifold is a topological space which is locally Euclidean. Represents a very useful and challenging unsupervised learning problem. In general, any object which is nearly "flat" on small scales is a manifold.

**Manifold Learning** • Discover low dimensional structures (smooth manifold) for data in high dimension. • Linear Approaches • Principal component analysis. • Multi dimensional scaling. • Non Linear Approaches • Local Linear Embedding • ISOMAP • Laplacian Eigenmap.

**Nonlinear Approaches- ISOMAP** • Construct neighbourhood graph G. • For each pair of points in G, compute, shortest path distances - geodesic distances. • Construct k-dimensional coordinate vectors Geodesic: Shortest curve along the manifold connecting two points

**ISOMAP algorithm Pros/Cons** Advantages: • Nonlinear • Globally optimal • Guarantee asymptotically to recover the true dimensionality Drawback: • May not be stable, dependent on topology of data • As N increases, pair wise distances provide better approximations to geodesics, but cost more computation

**Local Linear Embedding (a.k.a LLE)** • LLE is based on simple geometric intuitions. • Suppose the data consist of N real-valued vectors Xi, each of dimensionality D. • Each data point and its neighbors expected to lie on or close to a locally linear patch of the manifold.

**Steps of locally linearembedding:** • Reconstruction errors are measured by the cost function • έ(W) =

**Steps in LLE algorithm** • Assign neighbors to each data point • Compute the weights Wij that best linearly reconstruct the data point from its neighbors, solving the constrained least-squares problem. • Compute the low-dimensional embedding vectors best reconstructed by Wij.

**The final step** • Each high-dimensional observation is mapped to a low-dimensional vector representing global internal co ordinates.

**Fit locally, Think Globally** From Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul

**Neighbor embedding method** 1) For each patch xqtin image Xt: a) Find the set Nqof K nearest neighbors in Xs. b) Compute the reconstruction weights of the neighbors that minimize the error of reconstructing xqt.

**Neighbor embedding methodContd….** c) Compute the high-resolution embedding yqt using the appropriate high-resolution features of the K nearest neighbors and the reconstruction weights. 2) Construct the target high-resolution image Yt by enforcing local compatibility and smoothness constraints between adjacent patches obtained in step 1(c).

**Equations to fit** Minimize local reconstruction error Gram Matrix

**Solution to Constrained Least Squares Problem** Efficient way 1)Rather than inverting G matrix is to solve the equations Gqwq=1 2)Normailze the weights

**Final step** Final equation in the reconstruction of high resolution image And finally, Step 2 of the algorithm is achieved by averaging the feature values in overlapped regions between adjacent patches

**Problem Formulation** Training Xsi Training Ysi ? Testing Xt Testing Yt

**Intuition** • Patches of the image lie on a manifold Training Xsi Low dimensional Manifold High dimensional Manifold Training Ysi

**Algorithm** • Get feature vectors for each low resolution training patch. • For each test patch feature vector find K nearest neighboring feature vectors of training patches. • Find optimum weights to express each test patch vector as a weighted sum of its K nearest neighbor vectors. • Use these weights for reconstruction of that test patch in high resolution.

**Experimental Setup** • Target magnification : 3X • Transform to YIQ space • Feature Selection • First derivative of luminance • Second derivative of luminance • Reconstruction of Y and transfer of I and Q from lower dimension.

**Results** Experiments with images from the paper. Intuition: Authors’ test set may be biased !

**Results** Training Xsi Training Ysi Testing Xt Testing Yt

**Results** Training Xsi Training Ysi Testing Xt Testing Yt

**Results** Training Xsi Training Ysi Testing Xt Testing Yt

**Results** Training Ysi Training Xsi Testing Xt Testing Yt

**Results** K=1 K=2 K=3 K=4 k=5 k=6

** Results** As expected, good super-resolution images generated. Now for the real test – our images !

**Results – their method our data** Our Test Image 1/3X Obtained from scaling down Ground Truth Image

**Results – their method our data** High-resolution training images : (A) Similar (B) Dissimilar images Corresponding low-resolution training image

**Results – their method our data** We test the results using RMS Error using the following formula: Error = √ (1/n) ∑ || Pground-truth – Pgenerated || This essentially indicates the average Luminance deviations between corresponding pixels in the Ground-truth and Generated Images. Remember, the Chrominance components I and Q were copied into the generated image without change.

**Results – Similar training Image ** K=1 K=2 K=3 K=4 K=5 K=6

**Results – Different training Image** K=1 K=2 K=3 K=4 K=5 K=6

**Results – The Graph !**

**Comments on results** • Best results when using • Different images: K = 4 or 5 (confirming to what is stated in the paper). • Same images: K =1 (why ?))

**Comments** • Method worked well with our test data. • Care should be taken about image patches which do NOT form a manifold. • Limitations on magnification. • Size/ Overlap in the patch. • Many extensions are possible: • Diagonal gradients for LR (increased dimensionality) • Using Lumina as feature vector in LR • Can be extended to Video shot SR • Multiple training images

**Thank You**