Wei Zhu, Xiang Tian , Fan Zhou and Yaowu Chen IEEE TCE, 2010

1. Wei Zhu, Xiang Tian, Fan Zhou and YaowuChen IEEE TCE, 2010 Fast Inter Mode Decision Based on Textural Segmentation and Correlations for Multiview Video Coding

2. Outline • Introduction • Observation and Analysis • Proposed Fast Inter Mode Decision • Experimental Results • Conclusion

3. Introduction • Joint Video Team(JVT) has developed a Joint Multiview Video Model(JMVM) which is based on H.264/AVC to compress multiple video data efficiently. • Fig.1 shows an illustration of prediction structure employed in JMVM(Hierarchical B prediction structure). Primary view Auxiliary view DE ME + ME

4. Introduction • MVC uses Rate-Distortion Optimization(RDO) technique to select the optimal coding mode which is time consuming. • The coding structure in MVC are more complex than H.264/AVC, especially for auxiliary views employing inter-view prediction. • The prediction directions could be decided to reduce the computational complexity of auxiliary views. • The correlations between views can be exploited to further speed up the mode decision of MVC.

5. Observation and Analysis • Use JMVM4.0 with the full mode decision to analyze modes features. • Analysis • Skip mode occupies the largest proportion among all MB modes •  a large proportion background •  no inter estimation, faster •  determine to be Skip mode as • fast as possible • Inter8x8 mode occupies the smallest proportion among Inter modes •  consume 50% encoding time •  reduce the estimation of • Inter8x8

6. Observation and Analysis • Inter_DE mode only occupy a small proportion, but the computation time consumed by disparity estimation for inter-view prediction occupies more than 50% of encoding time.  reduce disparity estimation as much as possible • Intra mode occupies a small proportion among all MB modes, and computation is also small.  implement it before Inter mode decision, the results can be used for the optimization of Inter mode decisions

7. Observation and Analysis • Target: exploit the correlation between Intra mode and Inter mode, and the correlation between textural complexities and RD costs. • Analysis • Both Intra mode and Inter mode have higher RD costs in higher texture region • Both Intra mode and Inter mode have lower RD costs in lower texture region • Inter/Intra mode RD costs are sensitive to textural regions •  Intra mode RD costs can be used • for the segmentation of textural • region Inter Intra ※Blocks with higher values of RD costs are darker and lower values are lighter.

8. Proposed Fast Inter Mode Decision • Reference views’ Intra/Skip RD costs info are transferred to the current view. • Current frame is segmented into three textural regions. • Do Intra, Skip, and Inter16x16 modes. Analysis • Intra : get its RD cost to obtain the textural region type of MB • Skip : get its RD cost or the early decision of Skip mode • Inter16x16 : get results of ME and DE, and use these results to decide if to do DE/Inter8x8 or not

9. Proposed Fast Inter Mode Decision A. Segmentation of Textural Regions • Reference views’ Intra mode RD costs are used to segment textural regions. • High textural area  Intra4x4 mode • Low textural area  Intra16x16 mode • Intra mode is classified into INTRA4 and INTRA16 • INTRA4 : Intra4x4, Intra8x8 and PCM modes  high texture • INTRA16 : Intra16x16  low texture

10. Proposed Fast Inter Mode Decision • Sort RD costs of INTRA4 and INTRA16 from low to high INTRA4 mode RD cost are in a relative high level INTRA16 mode RD cost are in a relative low level • The variation of RD costs with the same Intra mode among views are consistent • The distribution of all Intra mode RD costs can be divided into three segments  the low, the high, and the in-between values

11. Proposed Fast Inter Mode Decision • Calculate thresholds of the textural segmentation: • Use neighboring views Intra RD costs • k : index of neighboring reference views • Obtain segmentation threshold • For each MB in current frame, obtain its textural region type • n : index of MB

12. Proposed Fast Inter Mode Decision • Distribution of textural regions in Race1 sequence is shown in Fig.6 High textural region Median textural region Low textural region • Distributions of textural regions are varying with different QP, it is more suitable for Inter mode decision

13. Proposed Fast Inter Mode Decision B. Early decision of Skip Mode • Idea : Considering that RD costs have different sensitivities to textural complexities, so the RD cost threshold should be calculated for different textural regions. • Numbers of Skip mode selected by full mode decision: • The numbers are similar between frames with the same QP

14. Proposed Fast Inter Mode Decision

15. Proposed Fast Inter Mode Decision always above  high texture • Average Skip mode RD cost curves: Values are similar at same time instance  low texture

16. Proposed Fast Inter Mode Decision • Skip mode RD cost includes only distortion part. It is more relevant to textural complexity of MB. • The relationship between and : • Early decision of Skip mode: • if Skip mode has minimum RD cost among Intra, Skip, Inter16x16. • Its RD cost is less than its corresponding threshold ()according to textural region type. • Not used for anchor frames

17. Proposed Fast Inter Mode Decision C. Selection of Disparity Estimation • The correlation of selected inter-view prediction between Inter16x16 mode and all Inter modes was analyzed: • Enabling of disparity estimation for other Inter modes: Inter16x16 All Inter mode ※ and 1.0 1.1 1.2 Choose Inter-view prediction Choose temporal prediction • Prediction results of INTER16x16 mode can be utilized to guide the selection of DE for other Inter modes

18. Proposed Fast Inter Mode Decision D. Reduction of Inter8x8 Mode Estimation • A distribution of Inter8x8 mode selected by the full mode decision: • The proportions of Inter8x8 mode in different textural regions: • Most of Inter8x8 mode appear in regions with more textural complexity • There are few Inter8x8 mode in backgrounds with no motion activity • Most of Inter8x8 mode are in the high textural region, very few in low textural region

19. Proposed Fast Inter Mode Decision • Enabling of Inter8x8 mode estimation: ※is the motion vector of MB

20. Experiment Results [1] L. Q. Shen, Z. Liu, S. Liu, Z. Y. Zhang, and P. An, “Selective disparity estimation and variable size motion estimation based on motion homogeneity for multi-view coding,” IEEE Trans. Broadcasting, vol. 55, no. 4, pp. 761-766, Dec. 2009.

21. Experiment Results • For the early decision of Skip mode, parameter was determined by experimental analysis: ※

22. Experiment Results • Compare to the full mode decision:

23. Experiment Results • Compare to Shen’s algorithm

24. Experiment Results • Compare to Shen’s algorithm

25. Conclusion • A fast Inter mode decision is proposed to reduce the computational complexity of MVC. • Skip mode is early decided, the disparity estimation is selected, and the Inter8×8 mode estimation is reduced. • As compared to the full mode decision about 55~70% time saving with negligible loss of coding efficiency. • The proposed algorithm could be combined with fast motion search and disparity search algorithms to further save the computation time.