1 / 48

Video Frames Interpolation Using Adaptive Warping

Video Frames Interpolation Using Adaptive Warping. Ying Chen Lou Major Advisor: M.J.T. Smith Co-advisor: Edward Delp Nov. 15, 2010. Outline. Background Generic motion model Video spatial interpolation Video compression Video frame rate up-conversion Summary and future work. Motivation.

reeves
Download Presentation

Video Frames Interpolation Using Adaptive Warping

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Video Frames Interpolation Using Adaptive Warping Ying Chen Lou Major Advisor: M.J.T. Smith Co-advisor: Edward Delp Nov. 15, 2010

  2. Outline • Background • Generic motion model • Video spatial interpolation • Video compression • Video frame rate up-conversion • Summary and future work

  3. Motivation • Spatial interpolation • Conversion from SDTV to HDTV • Zooming of region of interest (ROI) • Surveillance/forensics • Medical imaging • Satellite imaging • Temporal interpolation (Frame rate up-conversion) • 3:2 pull down 24Hz -> 30Hz • Avoid flicker and blurring on LCD • Loss of frames in transmission Frame Rate Up-Converter

  4. Challenges and Goal • Core: motion estimation • To derive a generic motion model which can be used for different applications • Motion needs to be accurate • Ill-posed problem (aperture problem) • Suitable for different types of motion • Translational (panning) • Zoom in/out • Rotational • Low to moderate computational intensive Local window used for ME

  5. Block Matching Optical Flow Illustrative ExampleBlock Matching vs. Optical Flow Block Matching Optical Flow Motion

  6. Motion Estimation Method (Warping) I[n1, n2, k] = I[n1 + d1[n1, n2, k], n2 + d2[n1, n2, k], k + δk] OFE where

  7. Warping Method (cont’d) • Assumes that the pixel displacement functions within some region R of an image I[n1,n2,k] can be written as: • and are vectors composed of the displacement parameters to be estimated • The bilinear displacement parameter are computed by minimizing the mean squared error function (MSEF)

  8. Quad-tree Divisions Quad-tree Uniform Makes the algorithm adaptive and more efficient

  9. Application I Spatial Interpolation Motivation • Characteristics of lecture videos • Large static backgrounds • Little to medium motion of foreground • Possible to store several high resolution frames and retrieve them later • Simplest scenario • periodically transmit one high resolution frame and the remaining frames are in the form of low resolution • Extend the proposed method to other types of video

  10. Proposed method I: Full-band Warping (FWWA) • Interpolation • Bilinear • Adaptive Synthesis • Filter Banks • Warping • Block matching-based and optical flow-based • Quadtree splitting

  11. High Motion Breakdown

  12. Robustness Issue • Address the robustness issue • Obtain reliable motion vectors • Maintain the sharpness • Challenges • No corresponding pixels in the reference frame • Ambiguity of improvement in sharpness and distortion objectively

  13. Proposed method IIThe Composite Algorithm • Incorporate advanced spatial interpolation algorithms • Bidirectional warping • Forward and backward warping • To solve “no corresponding pixels in the reference frame” • Hierarchical motion structure • To solve “ambiguity of improvement in sharpness and geometric distortion ”

  14. Hierarchical Motion Structure ZF3 – forward warped frame ZF3’ – downsampled forward warped frame ZB3 – backward warped frame ZB3’ – downsampled backward warped frame X3 – original MxN low resolution frame

  15. Experiment Setup • Assessment of the composite algorithm • Compare with bilinear, bicubic, NEDI, VA, and the full-band warping algorithm (FWWA) • Investigate the impact of different spatial interpolation methods on the composite warping algorithm • Incorporate different spatial interpolation methods in the first step • The remaining stages are the same • 3 sets of video sequences, CIF, 50 frames • Low motion (‘talking head’ lecture video ) • High detail • High motion

  16. Assessment of the Composite Algorithm • Every 5th frame as a high resolution reference frame • Decimate frames to get low resolution frames • Use a 21-tap lowpass filter • Downsample factor 2 • Use original frames as ground truth • Competing methods • Bilinear, Bicubic • simple image interpolation methods • New-edge Directed Interpolation (NEDI) • an advanced image interpolation method • A super resolution method proposed by Patrick Vandewalle (VA) 1 • Full band warping (FWWA) [1]P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain approach to registration of aliased images with application to superresolution”, EURASIP Journal on Applied Signal Processing, 2006

  17. Objective Results

  18. Talking Head Video Original frame Bilinear interpolation Comp FWWA

  19. High Detail Video Original frame Bilinear interpolation Comp FWWA

  20. High Motion Video Original frame Bilinear interpolation Comp FWWA

  21. (a) (b) (c) • (c) (d) • (a) Bilinear (b) NEDI (c) VA (d) Bicubic + Comp

  22. Demo

  23. Different Spatial Interpolation

  24. Conclusion • The composite methods achieved good spatial interpolation results • Accommodate complex motion • Outperform competing methods subjectively and objectively • Improvement comes mostly from the warping process • A combination of bicubic interpolation and warping results in best overall performance • Complexity not too high • Subjective and objective results are satisfactory • Particularly perform well for lecture videos and high detail videos

  25. Application II Video Compression H.264/AVC • Retain edges but remove texture at low bitrates

  26. Goal and Proposed Method • Goal • Propose a coding method which keeps the high frequency components • Achieve as high visual quality as possible • Maintain integrating of H.264/AVC coder which is well engineered • Be robust • Three assumptions • Smaller resolution requires fewer bits • Sequences with low motion don’t need full resolution coding • Key frames more bits • Proposed method • Adaptive warping • Spatio-temporal

  27. Overall System encoder decoder

  28. Experiment Setup • New algorithm compared against H.264 in the following setup • H.264 codec • Every Nth frame as an I frame, the rest coded as P or B frames • Proposed method • Every Nth frame used as the reference frame • Other frames are decimated (LL subband) and coded • Total bit rate is sum of full resolution reference frame and the quarter resolution LL subbands • Bit rates are the same in both cases

  29. Result (1)

  30. Subjective Result (a) H.264 (b) proposed method 3rd frame for Salesman sequence @ 80kbits/s

  31. Conclusion • The proposed method achieve better visual quality at low bitrates • The gap decreases as the bitrates increase • At high bitrates, H.264 has more bits to spend on high frequency components and thus achieves better quality • The nature of the proposed method works better for sequences with more details • Room remained to be improved • Explore tradeoffs in spatio-temporal decimation rates • More frequently for video with large motion and less often for video with small motion • For long lecture video, we can choose full coverage of reference frame and no anymore later

  32. Application III Video Frame Rate Up-Conversion • Overview of FRUC • No motion vectors • Frame repetition, frame average • Use motion vectors • Use motion vectors from the decoder directly • Advantage: Low complexity • Disadvantage: Not true motion • Perform motion estimation again • Advantage: true motion • Disadvantage: High computational complexity Frame Rate Up-Converter

  33. Goal and Proposed Method • Goal (1) True motion vectors (2) Relative low complexity • Challenges (1) Highly accurate MVs (2) Low percentage of MV re-estimation (3) Occlusion (4) Blocking artifacts • Approach • Decoded video sequences from the decoder • Additional information from the decoder • System diagram Previous Reconstructed frame X1 Interpolated Frame MV Reliability Check MV Re-estimation Motion Compensated Interpolation Small Block Merging Current residual frame, reconstructed frame X2, and its MVs

  34. MV reliability check • Categoried into 3 groups

  35. Small Block Merging • Avoid broken edges and want to maintain object structure

  36. MV Re-estimation • Key in the system • Accurate MVs are required for FRUC • Low complexity • A combination of Optical Flow-based and Block Matching-based motion estimation (Warping method)

  37. Motion Compensated Interpolation N=2, k=1

  38. Occlusion • Uni-directional interpolation N=2, k=1

  39. Overlapped Boundary Motion Compensation (OBMC) • Goal: • To reduce the blocking artifacts. • Selectively perform OBMC to reduce the computational complexity • (BAD > T)

  40. Experiment Setup • JM 11.0 • GOP: IPP…P, 15th I frame, fixed QP • Code odd frame and skip every other frame • 15fps • Transform 8x8=2 mode on • Search range = 16 • Standard CIF Video sequences used: • Akiyo, News, Salesman, Foreman, Carphone • Flower Garden, Tempete • Football, Table Tennis • Test against DMCFI, correlation-based motion selection1 [1] Ai-Mei Huang and T. Nguyen, “Correlation-based motion vector processing for motion compensated interpolation”, ICIP 2008

  41. Visual Result (1) 384kb/s (b) DMCFI 20.54dB (a) Orig (c ) Correlation-based 20.48dB (d) Proposed 24.19dB

  42. Visual Result (2) 512kb/s (b) DMCFI 20.01dB (a) Orig (c ) Correlation-based 20.17dB (d) Proposed 20.13dB

  43. Conclusion • Proposed a FRUC method that combines optical flow and block matching-based motion estimation • Reduced computational complexity • Reduced blocking artifacts • Achieve better visual quality for low motion video sequences and perform on par with other methods for high motion video sequences

  44. Summary • Provide a generic framework to achieve • Spatial enlargement of video frames • Video compression • Frame rate up-conversion • Achiever higher objective and subjective results • Improve the robustness by using FW, BW and hierarchical motion structure

  45. Future Work • Continue to refine the model • Apply to higher resolution video • Incorporate Subjective Video Quality Analysis • Reference frame recycling • Adaptively select the position of high quality reference frame

  46. Q & A

  47. Application I Spatial InterpolationRelated Work • Frame restoration • Frame interpolation • Bilinear, bicubic, spline, … • Adaptive Synthesis Filter Banks • New edge-directed interpolation • Superresolution (SR)

  48. Different Spatial Interpolation

More Related