 Download Presentation Super-Resolution

# Super-Resolution - PowerPoint PPT Presentation

Super-Resolution. Dr. Yossi Rubner yossi@rubner.co.il. Many slides from Miki Elad - Technion. Example - Video. 53 images, ratio 1:4. Example – Surveillance. 40 images ratio 1:4. Example – Enhance Mosaics. Super-Resolution - Agenda. The basic idea Image formation process I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation ## Super-Resolution

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
1. Super-Resolution Dr. Yossi Rubner yossi@rubner.co.il Many slides from Miki Elad - Technion

2. Example - Video 53 images, ratio 1:4

3. Example – Surveillance 40 images ratio 1:4

4. Example – Enhance Mosaics

5. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

6. For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible. D D Intuition

7. 2D 2D Intuition Due to our limited camera resolution, we sample using an insufficient 2D grid

8. 2D 2D Intuition However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:

9. Intuition Similarly, by shifting down we get a third image: 2D 2D

10. Intuition And finally, by shifting down and to the right we get the fourth image: 2D 2D

11. Intuition It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

12. Rotation/Scale/Disp. What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

13. Rotation/Scale/Disp. There is no sampling theorem covering this case

14. A Small Example 3:1 scale-up in each axis using 9 images, with pure global translation between them

15. Complicated motion perspective, local motion, … Blur sampling is not a point operation Spatially variant blur Temporally variant blur Noise Changes in the scene Further Complications

16. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

17. Geometric transformation Optical Blur Scene Sampling Noise Image Formation LR HR Can we write these steps as linear operators?

18. Geometric transformation Geometric Transformation • Any appropriate motion model • Every frame has different transformation • Usually found by a separate registration algorithm Scene

19. = Geometric Transformation Can be modeled as a linear operation

20. Geometric transformation Optical Blur Optical Blur • Due to the lens PSF and pixel integration • Usually

21. H H PSF * PIXEL =

22. = Optical Blur Can be modeled as a linear operation

23. Optical Blur Sampling Sampling • Pixel operation consists of area integration followed by decimation • D is the decimation only • Usually

24. o 1 = o Decimation Can be modeled as a linear operation

25. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

26. Super-Resolution - Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H D D H V V Low- Resolution Exposures 1 1 1 1 N X Additive Noise Y N F N N N

27. Simplified Model Geometric warp Blur Decimation Y High- Resolution Image 1 F =I H H D D V V Low- Resolution Exposures 1 1 N X Additive Noise Y N F N

28. The Super-Resolution Problem • Given Yk – The measured images (noisy, blurry, down-sampled ..) H – The blur can be extracted from the camera characteristics D – The decimation is dictated by the required resolution ratio Fk – The warp can be estimated using motion estimation n– The noise can be extracted from the camera / image • Recover X – HR image

29. =[10M×1] =[10M×16M] =[16M×1] The Model as One Equation r = resolution factor MXM = size of the frames N = number of frames r = resolution factor = 4 MXM = size of the frames = 1000X1000 N = number of frames = 10 Linear algebra notation is intended only to develop algorithm

30. Smoothness constraint regularization SR - Solutions • Maximum Likelihood (ML): Often ill posed problem! • Maximum Aposteriori Probability (MAP)

31. ML Reconstruction (LS) Minimize: Thus, require:

32. Back projection Simulated error All the above operations can be interpreted as operations performed on images. There is no actual need to use the Matrix-Vector notations as shown here. LS - Iterative Solution • Steepest descent

33. LS - Iterative Solution • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT -

34. LR + noise X4 Least squares Example HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

35. Robust Reconstruction • Cases of measurements outlier: • Some of the images are irrelevant • Error in motion estimation • Error in the blur function • General model mismatch

36. Minimize: Robust Reconstruction

37. Robust Reconstruction • Steepest descent For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign -

38. Least squares LR + noise X4 Robust Reconstruction Example - Outliers HR image Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

39. L2 norm based L1 norm based Example – Registration Error 20 images, ratio 1:4

40. MAP Reconstruction • Regularization term: • Tikhonov cost function • Total variation • Bilateral filter

41. Minimize: Robust Estimation + Regularization

42. Robust Estimation + Regularization For k=1..N - inverse geometry wrap geometry wrap convolve with H down sample up sample convolve with HT sign - For l,m=-P..P - - horizontal shift l vertical shift m horizontal shift -l vertical shift -m sign From Farisu at al. IEEE trans. On Image Processing, 04

43. Example • 8 frames • Resolution factor of 4 From Farisu at al. IEEE trans. On Image Processing, 04

44. Example Images from Vigilant Ltd.

45. Handling Color in SR Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr. Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images). In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

46. SR for Full Color 20 images, ratio 1:4

47. SR+Demosaicing 20 images, ratio 1:4 Mosaiced input Mosaicing and then SR Combined treatment

48. Super-Resolution - Agenda • The basic idea • Image formation process • Formulation and solution • Special cases and related problems • Limitations of Super-Resolution • SR in time

49. Special Case – Translational Motion • In this case H and F commute: • SR is decomposed into 2 steps • Find blur HR image from LR images  non-iterative • Deconvolve the result using H  iterative