super resolution n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Super-Resolution PowerPoint Presentation
Download Presentation
Super-Resolution

Loading in 2 Seconds...

play fullscreen
1 / 74

Super-Resolution - PowerPoint PPT Presentation


  • 199 Views
  • Uploaded on

Super-Resolution. Dr. Yossi Rubner yossi@rubner.co.il. Many slides from Miki Elad - Technion. Example - Video. 53 images, ratio 1:4. Example – Surveillance. 40 images ratio 1:4. Example – Enhance Mosaics. Super-Resolution - Agenda. The basic idea Image formation process

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Super-Resolution


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    1. Super-Resolution Dr. Yossi Rubner yossi@rubner.co.il Many slides from Miki Elad - Technion

    2. Example - Video 53 images, ratio 1:4

    3. Example – Surveillance 40 images ratio 1:4

    4. Example – Enhance Mosaics

    5. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

    6. For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible. D D Intuition

    7. 2D 2D Intuition Due to our limited camera resolution, we sample using an insufficient 2D grid

    8. 2D 2D Intuition However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:

    9. Intuition Similarly, by shifting down we get a third image: 2D 2D

    10. Intuition And finally, by shifting down and to the right we get the fourth image: 2D 2D

    11. Intuition It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

    12. Rotation/Scale/Disp. What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

    13. Rotation/Scale/Disp. There is no sampling theorem covering this case

    14. A Small Example 3:1 scale-up in each axis using 9 images, with pure global translation between them

    15. Complicated motion perspective, local motion, … Blur sampling is not a point operation Spatially variant blur Temporally variant blur Noise Changes in the scene Further Complications

    16. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

    17. Geometric transformation Optical Blur Scene Sampling Noise Image Formation LR HR Can we write these steps as linear operators?

    18. Geometric transformation Geometric Transformation • Any appropriate motion model • Every frame has different transformation • Usually found by a separate registration algorithm Scene

    19. = Geometric Transformation Can be modeled as a linear operation

    20. Geometric transformation Optical Blur Optical Blur • Due to the lens PSF and pixel integration • Usually

    21. H H PSF * PIXEL =

    22. = Optical Blur Can be modeled as a linear operation

    23. Optical Blur Sampling Sampling • Pixel operation consists of area integration followed by decimation • D is the decimation only • Usually

    24. o 1 = o Decimation Can be modeled as a linear operation

    25. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

    26. Super-Resolution - Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H D D H V V Low- Resolution Exposures 1 1 1 1 N X Additive Noise Y N F N N N

    27. Simplified Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H H D D V V Low- Resolution Exposures 1 1 N X Additive Noise Y N F N

    28. The Super-Resolution Problem • Given Yk – The measured images (noisy, blurry, down-sampled ..) H – The blur can be extracted from the camera characteristics D – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation n– The noise can be extracted from the camera / image • Recover X – HR image

    29. =[10M×1] =[10M×16M] =[16M×1] The Model as One Equation r = resolution factor MXM = size of the frames N = number of frames r = resolution factor = 4 MXM = size of the frames = 1000X1000 N = number of frames = 10 Linear algebra notation is intended only to develop algorithm

    30. Smoothness constraint regularization SR - Solutions • Maximum Likelihood (ML): Often ill posed problem! • Maximum Aposteriori Probability (MAP)

    31. ML Reconstruction (LS) Minimize: Thus, require:

    32. Back projection Simulated error All the above operations can be interpreted as operations performed on images. There is no actual need to use the Matrix-Vector notations as shown here. LS - Iterative Solution • Steepest descent

    33. LS - Iterative Solution • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT -

    34. LR + noise X4 Least squares Example HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

    35. Robust Reconstruction • Cases of measurements outlier: • Some of the images are irrelevant • Error in motion estimation • Error in the blur function • General model mismatch

    36. Minimize: Robust Reconstruction

    37. Robust Reconstruction • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign -

    38. Least squares LR + noise X4 Robust Reconstruction Example - Outliers HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

    39. L2 norm based L1 norm based Example – Registration Error 20 images, ratio 1:4

    40. MAP Reconstruction • Regularization term: • Tikhonov cost function • Total variation • Bilateral filter

    41. Minimize: Robust Estimation + Regularization

    42. Robust Estimation + Regularization For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign - For l,m=-P..P - - horizontal shift l vertical shift m horizontal shift -l vertical shift -m sign From Farisu at al. IEEE trans. On Image Processing, 04

    43. Example • 8 frames • Resolution factor of 4 From Farisu at al. IEEE trans. On Image Processing, 04

    44. Example Images from Vigilant Ltd.

    45. Handling Color in SR Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr. Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images). In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

    46. SR for Full Color 20 images, ratio 1:4

    47. SR+Demosaicing 20 images, ratio 1:4 Mosaiced input Mosaicing and then SR Combined treatment

    48. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

    49. Special Case – Translational Motion • In this case H and F commute: • SR is decomposed into 2 steps • Find blur HR image from LR images  non-iterative • Deconvolve the result using H  iterative