New Techniques for Next Generation Video Coding

1. New Techniques for Next Generation Video Coding Li Liu, Robert Cohen, Huifang Sun, Anthony Vetro, Xinhua Zhuang BMSB 2010

2. Outline • Introduction • Existing New Techniques • Weighted Prediction (WP) • Localized Weighted Prediction for Video Coding • Second Order Prediction on H.264/AVC • Proposed Second-Order Prediction for Inter Coding • Proposed Reduced Resolution Update for Intra Coding • Experimental Results • Conclusion

3. Introduction • Increasing popularity of high definition TV, video delivery on mobile devices , and other multimedia applications creates new demands for video coding standards. • Both MPEG and VCEG launched their next-generation video coding project, which potentially could be either an extension of H.264/AVC or a brand new standard. • In January 2010, MPEG and VCEG have established a Joint Collaborative Team on Video Coding (JCT-VC) to develop the proposed High Efficiency Video Coding (HEVC) standard.

4. Existing New Techniques • To provide a software platform to gather and evaluate these new techniques, a Key Technique Area (KTA) platform was developed based on JM11. • Intra Prediction : BIP, MDDT • Inter prediction : increasing resolution to 1/8-pel , MB size to 64x64 • Quantization : RDOQ, AQMS • Transform : 16x16 transform • In-loop Filter : QALF, BALF • Internal bit-depth increase : 12 bits of internal bit depth for 8-bit source

5. Weighted Prediction (WP) • An early form of second-order prediction. • For scenes with temporal brightness variations (illumination changes, fade-in/out effects, camera flashes) • Multiplicative weighting factor a and an additive weighting offset b are used to enhance motion compensation : • at time t

6. Localized Weighted Prediction for Video Coding[1] • Lighting conditions may vary not only between frames but also within a frame, to handle local lighting variation. • Assume the spatial variance of the intensity in a region is small , represent the brightness variation only using a weighting offset b. • : pixels in the source picture • : pixels in the reference frame • Assume the correlation between the neighboring samples and the current block is high. • : reconstructed neighboring samples of [1] Peng Yin, Alexis Michael Tourapis , Jill Boyce, “Localized Weighted Prediction for Video Coding,” IEEE Circuits and Systems, 2005.

7. Localized Weighted Prediction for Video Coding

8. Localized Weighted Prediction for Video Coding • For image without brightness change, the proposed method can also reduce coding efficiency. • LWP Adaption • Calculate weighting factor • Comparing the distance between the current picture and its closet reference picture • = • If , LWP is not used • Otherwise, use LWP to code current picture

9. Localized Weighted Prediction for Video Coding • Implement: • Step1: decide if LWP should be used for current slice. If not, perform normal coding as H.264 does, otherwise, go to step2. • Step2 : for each MB, first calculate the mean of the reconstructed neighboring pixel of the current MB. Then perform ME and mode decision using LWP.

10. Second Order Prediction on H.264/AVC[2] • The predicted blocks generated by MCP will result in low coding efficiency when the video containing complex movements such as shape transforming, rotation or fading. • Weighed prediction in H.264/AVC is presented to deal with the fading sequences with global illumination change between frames. • Utilizes only temporal correlation but no spatial correlation. • Can’t handle motion like shape transforming and rotation. • This paper proposes a Second Order Prediction (SOP) to exploit remaining signal correlation after MCP. [2] ShangwenLi, Sijia Chen, Jianpeng Wang and Lu Yu, “Second Order Prediction on H.264/AVC,” Picture Coding Symposium, 2009.

11. Second Order Prediction on H.264/AVC • Slight rotation • Visible residual • Residual exhibit high spatial correlation ※ All-black blocks indicate the MBs applying P-skip mode in the bit-stream

12. Second Order Prediction on H.264/AVC • Residual Subjective-Textured MBs (RST MBs) : MBs with relatively large residuals. More than twice

13. Second Order Prediction on H.264/AVC • Apply intra-prediction of H.264/AVC to residuals of inter-prediction. • The reconstructed pixel values of an SOP MB are derived as follow : • Reconstructed pixel-value = Motion-compensated prediction (first-predictor) + Prediction of first order residuals (second-predictor) + Second order residuals (need to be coded) • It seems straightforward to use the previously reconstructed first-order residuals of the neighboring blocks as reference for the current block.

14. Second Order Prediction on H.264/AVC • The discontinuity caused by motion between blocks will prohibit the efficient utilization of the remaining correlation of the first-order residuals.

15. Second Order Prediction on H.264/AVC • Reference generation : • Get reconstructed pixel values in the current frame, and reconstructed pixel values in the temporal reference frame • is integer within [-1, 2*(n-1)] when = -1 • is integer within [-1, (n-1)] when =-1 • is the motion vector of the current block. • Get the reference first-order residual RFR :

16. Second Order Prediction on H.264/AVC • SOP may take 4x4 or 8x8 block as its second prediction unit. • Nine 4x4 intra prediction modes of AVC Baseline profile • Nine 8x8 intra prediction modes of AVC High profile • Transform of the second prediction residuals takes the same block size as the second prediction. The block size might be chosen adaptively based on rate-distortion criterion.

17. Second Order Prediction on H.264/AVC • Coding of the additional side information of SOP • Indicator of SOP • An SOP flag to indicate the usage of SOP at MB level. • Mode indicator of the second prediction mode • A second prediction mode is calculated for each MB, and the coding procedure is the same as that of 4x4 or 8x8 intra-prediction modes encoding in H.264/AVC. • The decision of whether an MB will be coded in SOP mode follow the rate-distortion criterion.

18. Experimental Results Stable improvement at most 0.41dB gain

19. Proposed Second-Order Prediction for Inter Coding • It is not efficient for blocks whose size is smaller than 8x8, as too much side information needs to be coded. • Partition size larger than 8x8 will be divided into multiple 8x8 sub-blocks, each with its own second-order prediction mode.

20. Proposed Reduced Resolution Update for Intra Coding • Reduced resolution update (RRU) is a technique that aims to save coding bits by resize image/prediction residuals to a reduced spatial resolution. • At low bit rates, it’s known that down-sampling an image to a low resolution, then compressing the lower resolution, and interpolating the result to the original resolution can improve the overall PSNR.

21. Proposed Reduced Resolution Update for Intra Coding JPEG Reduced resolution • Blocks of 8x8 pixels • Allocate too few bits (4 bits per block on average) • Only DC coefficients are coded • Blocking artifacts

22. Proposed Reduced Resolution Update for Intra Coding

23. Proposed Reduced Resolution Update for Intra Coding • Modified the framework of H.264/AVC so that residual after intra prediction can be optionally down-sampled before the transform and quantization steps. • For instance : a 16x16 block can be down-sampled by a factor of 2 so that only an 8x8 block needs to be encoded • Decoder shall up-sample the down-sampled residual to reconstruct full resolution picture. • The choice of RRU should be considered under RDO.

24. Experimental Results • Second-Order Prediction for Inter Coding • Gains are not significant • The optimal motion vector position may be different from first-order motion vector. Perform motion vector search for each individual second-order prediction mode. • Increase computational complexity.

25. Experimental Results • Proposed Reduced Resolution Update for Intra Coding

26. Experimental Results • RRU improves the coding efficiency for medium content complexity. • H.264/AVC is efficient for flat areas. • RRU may bright too much loss for areas with high frequency content.

27. Experimental Results • RRU works well for 16x16 blocks, the contribution of RRU to overall intra coding shall depend on the percentage 16x16 block size is used over 4x4 and 8x8 modes.

28. Conclusion • Both the new techniques listed and our experiments on second-order prediction and RRU prove that there is still room for performance improvement of current coding standard. • The Call for Evidence for HVC provided results that averaged a 15-25% gain in coding efficiency.