1 / 48

Roadmap

Roadmap. Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Coding Motion Compensated Predictive (MCP) Coding Object-based and scalable video coding* Motion segmentation, scalability issues. Introduction to Video Coding.

velvet
Download Presentation

Roadmap

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) Coding • Motion Compensated Predictive (MCP) Coding • Object-based and scalable video coding* • Motion segmentation, scalability issues EE569 Digital Video Processing

  2. Introduction to Video Coding • Lossless vs. lossy data compression • Source entropy H(X) • Rate-Distortion function R(D) or D(R) • Probabilistic modeling is at the heart of data compression • What is P(X) for video source X? • Is video coding more difficult than image coding? EE569 Digital Video Processing

  3. Shannon’s Picture For Gaussian source N(0,2) Distortion Coder A Coder B Rate (bps) For video source, no one knows the limit (bound) EE569 Digital Video Processing

  4. Distortion Measures • Objective • Mean Square Error (MSE) • Peak Signal-to-Noise-Ratio (PSNR) • Measure the fidelity to original video • Subjective • Human Vision System (HVS) based • Emphasize visual quality rather than fidelity • We only discuss objective measures in this course, but subjective video quality assessment is an open and important topic EE569 Digital Video Processing

  5. Video Coding Applications EE569 Digital Video Processing

  6. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Object-based and scalable video coding* • Motion segmentation, scalability issues EE569 Digital Video Processing

  7. A Tour of JPEG Coding Standard Key Components • Transform -8×8 DCT -boundary padding • Quantization -uniform quantization -DC/AC coefficients • Coding -Zigzag scan -run length/Huffman coding EE569 Digital Video Processing

  8. JPEG Baseline Coder Tour Example EE569 Digital Video Processing

  9. Step 1: Transform • DC level shifting -128 • 2D DCT DCT EE569 Digital Video Processing

  10. Step 2: Quantization Why increase from top-left to bottom-right? Q-table Q EE569 Digital Video Processing

  11. Step 3: Entropy Coding Zigzag Scan (20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,0,0,0, 0,1,1,0,1,EOB) End Of the Block: All following coefficients are zero Zigzag Scan EE569 Digital Video Processing

  12. Roadmap • Introduction • Intra-frame coding • Review of JPEG • Inter-frame coding • Conditional Replenishment (CR) • Motion Compensated Prediction (MCP) • Object-based and scalable video coding* • Motion segmentation, scalability issues EE569 Digital Video Processing

  13. Conditional Replenishment • Based on motion detection rather than motion estimation • Partition the current frame into “still areas” and “moving areas” • Replenishment is applied to moving regions only • Repetition is applied to still regions • Need to transmit the location of moving areas as well as new (replenishment) information • No motion vectors transmitted EE569 Digital Video Processing

  14. Conditional Replenishment EE569 Digital Video Processing

  15. Motion Detection EE569 Digital Video Processing

  16. From Replenishment to Prediction • Replenishment can be viewed as a degenerated case of prediction • Only zero motion vector is considered • Discard the history • A more powerful approach of exploiting temporal dependency is prediction • Locate the best match from the previous frame • Use the history to predict the current EE569 Digital Video Processing

  17. Differential Pulse Coded Modulation ^ xn-1 ^ ^ ^ yn yn yn xn xn _ Q + ^ ^ xn-1 xn D + D ^ xn-1 Decoder Encoder Xn,yn: unquantized samples and prediction residues ^ ^ Xn,yn: decoded samples and quantized prediction residues EE569 Digital Video Processing

  18. Motion-Compensated Predictive Coding EE569 Digital Video Processing

  19. A Closer Look EE569 Digital Video Processing

  20. Key Components • Motion Estimation/Compensation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE569 Digital Video Processing

  21. Block-based Motion Model • Block size • Fixed vs. variable • Motion accuracy • Integer-pel vs. fractional-pel • Number of hypothesis • Overlapped Block Motion Compensation (OBMC) • Multi-frame prediction EE569 Digital Video Processing

  22. Quadtree Representation of Motion Field with Variable Blocksize Sullivan, G.J.; Baker, R.L., "Rate-distortion optimized motion compensation for video compression using fixed or variable size blocks," GLOBECOM '91. pp.85-90 vol.1, 2-5 Dec 1991 EE569 Digital Video Processing

  23. Example counted bits using a VLC table EE569 Digital Video Processing

  24. Fractional-pel BMA • Recall the tradeoff between spending bits on motion and spending bits on MCP residues • Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence. • The gain quickly saturates as motion accuracy refines EE569 Digital Video Processing

  25. Example 8-by-8 block, integer-pel, var(e)=220.8 8-by-8 block, half-pel, var(e)=123.8 MCP residue comparison for the first two frames of Mobile sequence EE569 Digital Video Processing

  26. Fractional-pel MCP Girod, B., "Motion-compensating prediction with fractional-pel accuracy," IEEE Trans. onCommunications, vol.41, no.4, pp.604-612, Apr 1993 EE569 Digital Video Processing

  27. Multi-Hypothesis MCP • Using one block from one reference frame represents a single-hypothesis MCP • It is possible to formulate multiple hypothesis by considering • Overlapped blocks • More than one reference frame • Why multi-hypothesis? • The benefit of reducing variance of MCP residues outweighs the increased overhead on motion EE569 Digital Video Processing

  28. Example: B-frame fn-1 fn fn+1 EE569 Digital Video Processing

  29. Generalized B-frame fn-2 fn-1 fn+2 fn fn+1 EE569 Digital Video Processing

  30. Overlapped Block Motion Compensation (OBMC)   EE569 Digital Video Processing

  31. Overlapped Block Motion Compensation (OBMC) • Conventional block motion compensation • One best matching block is found from a reference frame • The current block is predicted by the best matching block • OBMC • Each pixel in the current block is predicted by a weighted average of several corresponding pixels in the reference frame • The corresponding pixels are determined by the MVs of the current as well as adjacent MBs • The weights for each corresponding pixel depends on the expected accuracy of the associated MV EE569 Digital Video Processing

  32. OBMC Using 4 Neighboring MBs Should be inversely proportional to the distance between x and the center of EE569 Digital Video Processing

  33. Optimal Weighting Design* • Convert to an optimization problem: • Minimize • Subject to • Optimal weighting functions: EE569 Digital Video Processing

  34. Multi-Hypothesis MCP EE569 Digital Video Processing

  35. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE569 Digital Video Processing

  36. Motion Vector Coding • 2D lossless DPCM • Spatially (temporally) adjacent motion vectors are correlated • Use causal neighbors to predict the current one • Code Motion Vector Difference (MVD) instead of MVs • Entropy coding techniques • Variable length codes (VLC) • Arithmetic coding EE569 Digital Video Processing

  37. MVD Example MV1 MV2 MV MV3 Due to smoothness of MV field, MVD usually has a smaller variance than MV EE569 Digital Video Processing

  38. VLC Example MVx/MVy symbol codeword 0 1 1 010 1 2 -1 011 3 00100 4 2 -2 00101 5 3 00110 6 Exponential Golomb Codes: 0…01x…x m m-1 EE569 Digital Video Processing

  39. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE569 Digital Video Processing

  40. MCP Residue Coding Transform Quantization Coding Conceptually similar to JPEG Transform: unitary transform Quantization: Deadzone quantization Coding: Run-length coding EE569 Digital Video Processing

  41. Transform Unitary matrix: A is real, A-1=AT Unitary transform: A is unitary, Y=AXAT Examples 8-by-8 DCT 4-by-4 integer transform EE569 Digital Video Processing

  42. Deadzone Quantization deadzone 2       0 codewords EE569 Digital Video Processing

  43. Key Components • Motion Estimation • At the heart of MCP-based coding • Coding of Motion Vectors (overhead) • Lossless: errors in MV are catastrophic • Coding of MCP residues • Lossy: distortion is controlled by the quantization step-size • Rate-Distortion optimization EE569 Digital Video Processing

  44. Constrained Optimization Min f(x,y) subject to g(x,y)=c

  45. Lagrangian Multiplier Method Motion estimation Mode selection QUANT: a user-specified parameter controlling quantization stepsize EE569 Digital Video Processing

  46. Example: Rate-Distortion Optimized BMA Distortion alone Rate and Distortion counted bits using a VLC table EE569 Digital Video Processing

  47. Experimental Results Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized motion compensation for video compression using fixed or variable size blocks”, Globecom’1991 EE569 Digital Video Processing

  48. Summary • How does MCP coding work? • The predictive model captures the slow-varying trend of the samples {fn} • The modeling of prediction residues {en} is easier than that of original samples {fn} • Fundamental weakness • Quantization error will propagate unless the memory of predictor is refreshed • Not suitable for scalable coding applications EE569 Digital Video Processing

More Related