Applying 3-D Methods to Video for Compression

1. Applying 3-D Methods to Video for Compression Salih Burak Gokturk Anne Margot Fernandez Aaron March 13, 2002 EE 392J Project Presentation

2. OUTLINE • MOTIVATION • PREVIOUS WORK • ALGORITHMS • RESULTS • CONCLUSION

3. BACKGROUND • Most common video compression scheme • Blockwise transform (i.e. 2D-DCT) for spatial compression • Blockwise Motion Estimation and Compensation for temporal redundancy • Used in MPEG • Motion Compensated Predictive Coding is better than Image based coding since it uses a good prediction from the previous frame.

4. OBJECTIVE Exploit Temporal Redundancies MORE! • Why? • How? • Use 3-D Transformations and Methods.

5. 3-D Transform Methods • Good for ideal case – smooth motion, global motion • Bad for non-uniform motion • Need large number of frame stores

6. PREVIOUS WORK • 3-D DCT for Video • Bauer and Sayood, Chan and Sui, Servais and de Jager • 3-D Wavelets for Video – Pearlman et al. , Taubman et al., • Motion Compensated Coding

7. ALGORITHMS IN OUR EXPERIMENTS • 2D DCT • 3D DCT • (Block) Motion Compensated Coding • 3D (Volume) Motion Compensated Coding • Other Transformation as a prediction scheme (PCA, Wavelet)

8. 2-D versus 3-D DCT • 2-D Blocks versus 3-D Blocks • 2-D DCT versus 3-D DCT • When would 3-D DCT not work? Markov-1 chain P(Xn=Xn-1) = p Xn uniform [0, 255]

9. 3D Volume Motion Compensated Coding depth Current stack of frames Previous stack of frames For each volume block… Divide stack into volume blocks Find 3D motion vectors Calculate residual 3D DCT on residual Q

10. 3D Volume Motion Compensated Coding • Finding the motion vectors • Full Search • Measurement criterion • MAD • standard deviation of residual • When it does not work • No smooth motion (acceleration present) • Cases when 2D MC does not work (occlusion, illumination, non-uniform motion) • Block size (depth) along time axis is too small or too big • Centroid Effect : Center of the block usually has a better prediction than edges of the block.

11. Initial Prediction Results 2D MC Prediction 3D MC Prediction

12. [ ] U=[ ] … 2 – Each Block is represented by a linear combination of principal components: … 1 + 2 + 3 + = For each volume block… Divide each stack into volume blocks Find PCA Coefs Calculate residual 3D DCT on residual Q 3D Principal Component Analysis Current stack of frames 1 – Apply PCA on a large collection of blocks = A = U  VT …

13. Experimental Setup • Techniques • 2D DCT by 8x8 Blocks • 3D DCT by 8x8x8 Blocks • Block Motion Compensated Coding (8x8 and 16x16 followed by 2-D DCT) • Volume Motion Compensated Coding (8x8x4, 8x8x6 and 8x8x8) followed by 3-D DCT. • Principal Component Analysis (use principal components of 8x8x8 blocks as prediction followed by 3-D DCT) • Bit Rate vs PSNR Comparison • DCT coefficients were quantized using a uniform quantizer • PSNR was computed over the same support region. • Bit rate was calculated by finding the theoretical entropy of the quantized DCT coefficients and motion vectors.

14. Miss Am

15. 0.14 bit/pixel Miss Am 2D-DCT 2D-MC 3D-DCT 3D-MC 3D-PCA

16. Salesman – 2D-MC vs 3D-MC 2D-MC 3D-MC 0.11 bit/pixel

17. Salesman – 3D-DCT vs 3D-MC 3D-DCT 3D-MC 0.16 bit/pixel

18. 2D-MC 3D-MC Foreman 3D-DCT 0.56 bit/pixel

19. Bus

20. CONCLUSION • There is redundancy across temporal domain. • There is a lot of room for improvement if we use 3-D compression methods! • 2-D motion compensated coding is good for movies with A LOT of motion. • When movements are smooth, 3-D methods will perform better than 2-D motion compensated coding. FUTURE WORK • Find Better Residual Coding Schemes for 3D MC. (That deals with centroid effect better) • Other 3D Methods ?