Roadmap
Download
1 / 50

Roadmap - PowerPoint PPT Presentation


  • 144 Views
  • Uploaded on

Roadmap. Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Motion Compensated Prediction (MCP) Object-based and scalable video coding* Motion segmentation, scalability issues. Introduction to Video Coding.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Roadmap' - gannon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Roadmap
Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Object-based and scalable video coding*

    • Motion segmentation, scalability issues

EE591f Digital Video Processing


Introduction to video coding
Introduction to Video Coding

  • Lossless vs. lossy data compression

    • Source entropy H(X)

    • Rate-Distortion function R(D) or D(R)

  • Probabilistic modeling is at the heart of data compression

    • What is P(X) for video source X?

    • Modeling moving pictures is more difficult than modeling still images due to temporal dependency

EE591f Digital Video Processing


Shannon s picture
Shannon’s Picture

Distortion

Coder A

Coder B

Rate (bps)

Which coder wins, A or B?

EE591f Digital Video Processing


Distortion measures
Distortion Measures

  • Objective

    • Mean Square Error (MSE)

    • Peak Signal-to-Noise-Ratio (PSNR)

    • Measure the fidelity to original video

  • Subjective

    • Human Vision System (HVS) based

    • Emphasize visual quality rather than fidelity

  • We only discuss objective measures in this course

EE591f Digital Video Processing


Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


A Tour of JPEG Coding Standard

Key Components

  • Transform

-8×8 DCT

-boundary padding

  • Quantization

-uniform quantization

-DC/AC coefficients

  • Coding

-Zigzag scan

-run length/Huffman coding

EE591f Digital Video Processing


JPEG Baseline Coder

Tour Example

EE591f Digital Video Processing


Step 1: Transform

• DC level shifting

-128

• 2D DCT

DCT

EE591f Digital Video Processing


Step 2: Quantization

Why increase

from top-left to

bottom-right?

Q-table

Q

EE591f Digital Video Processing


Step 3: Entropy Coding

Zigzag Scan

(20,5,-3,-1,-2,-3,1,1,-1,-1,

0,0,1,2,3,-2,1,1,0,0,0,0,0,

0,1,1,0,1,EOB)

End Of the Block:

All following coefficients

are zero

Zigzag Scan

EE591f Digital Video Processing


Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


Conditional replenishment
Conditional Replenishment

  • Based on motion detection rather than motion estimation

  • Partition the current frame into “still areas” and “moving areas”

    • Replenishment is applied to moving regions only

    • Repetition is applied to still regions

  • Need to transmit the location of moving areas as well as new (replenishment) information

    • No motion vectors transmitted

EE591f Digital Video Processing


Conditional replenishment1
Conditional Replenishment

EE591f Digital Video Processing


Motion detection
Motion Detection

EE591f Digital Video Processing


From replenishment to prediction
From Replenishment to Prediction

  • Replenishment can be viewed as a degenerated case of prediction

    • Only zero motion vector is considered

    • Discard the history

  • A more powerful approach of exploiting temporal dependency is prediction

    • Locate the best match from the previous frame

    • Use the history to predict the current

EE591f Digital Video Processing


Differential pulse coded modulation
Differential Pulse Coded Modulation

^

xn-1

^

^

^

yn

yn

yn

xn

xn

_

Q

+

^

^

xn-1

xn

D

+

D

^

xn-1

Decoder

Encoder

Xn,yn: unquantized samples and prediction residues

^

^

Xn,yn: decoded samples and quantized prediction residues

EE591f Digital Video Processing


Motion compensated predictive coding
Motion-Compensated Predictive Coding

EE591f Digital Video Processing


A closer look
A Closer Look

EE591f Digital Video Processing


Key components
Key Components

  • Motion Estimation/Compensation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Block based motion model
Block-based Motion Model

  • Block size

    • Fixed vs. variable

  • Motion accuracy

    • Integer-pel vs. fractional-pel

  • Number of hypothesis

    • Overlapped Block Motion Compensation (OBMC)

    • Multi-frame prediction

EE591f Digital Video Processing


Quadtree representation of motion field with variable blocksize
Quadtree Representation of Motion Field with Variable Blocksize

EE591f Digital Video Processing


Rate distortion optimized bma
Rate-Distortion Optimized BMA

Distortion alone

Rate and Distortion

counted bits using a VLC table

EE591f Digital Video Processing


Experimental results
Experimental Results

Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized

motion compensation for video compression using fixed or

variable size blocks”, Globecom’1991

EE591f Digital Video Processing


Fractional pel bma
Fractional-pel BMA

  • Recall the tradeoff between spending bits on motion and spending bits on MCP residues

  • Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence.

  • The gain quickly saturates as motion accuracy refines

EE591f Digital Video Processing


Example
Example

8-by-8 block, integer-pel, var(e)=220.8

8-by-8 block, half-pel, var(e)=123.8

MCP residue comparison for the first two frames of Mobile sequence

EE591f Digital Video Processing


Fractional pel mcp
Fractional-pel MCP

EE591f Digital Video Processing


Multi hypothesis mcp
Multi-Hypothesis MCP

  • Using one block from one reference frame represents a single-hypothesis MCP

  • It is possible to formulate multiple hypothesis by considering

    • Overlapped blocks

    • More than one reference frame

  • Why multi-hypothesis?

    • The benefit of reducing variance of MCP residues outweighs the increased overhead on motion

EE591f Digital Video Processing


Example b frame
Example: B-frame

fn-1

fn

fn+1

EE591f Digital Video Processing


Generalized B-frame

fn-2

fn-1

fn+2

fn

fn+1

EE591f Digital Video Processing


Multi hypothesis mcp1
Multi-Hypothesis MCP

EE591f Digital Video Processing


Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Motion vector coding
Motion Vector Coding

  • 2D lossless DPCM

    • Spatially (temporally) adjacent motion vectors are correlated

    • Use causal neighbors to predict the current one

    • Code Motion Vector Difference (MVD) instead of MVs

  • Entropy coding techniques

    • Variable length codes (VLC)

    • Arithmetic coding

EE591f Digital Video Processing


Mvd example
MVD Example

MV1

MV2

MV

MV3

Due to smoothness of MV field, MVD usually has

a smaller variance than MV

EE591f Digital Video Processing


Vlc example
VLC Example

MVx/MVy

symbol

codeword

0

1

1

010

1

2

-1

011

3

00100

4

2

-2

00101

5

3

00110

6

Exponential Golomb Codes: 0…01x…x

m

m

EE591f Digital Video Processing


Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Mcp residue coding
MCP Residue Coding

Transform

Quantization

Coding

Conceptually similar to JPEG

Transform: unitary transform

Quantization: Deadzone quantization

Coding: Run-length coding

EE591f Digital Video Processing


Transform
Transform

Unitary matrix: A is real, A-1=AT

Unitary transform: A is unitary, Y=AXAT

Examples

8-by-8 DCT

4-by-4 integer transform

EE591f Digital Video Processing


Deadzone quantization
Deadzone Quantization

deadzone

2

0

codewords

EE591f Digital Video Processing


Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Lagrangian multiplier method
Lagrangian Multiplier Method

Motion estimation

Mode selection

QUANT: a user-specified parameter controlling

quantization stepsize

EE591f Digital Video Processing


Summary
Summary

  • How does MCP coding work?

    • The predictive model captures the slow-varying trend of the samples {fn}

    • The modeling of prediction residues {en} is easier than that of original samples {fn}

  • Fundamental weakness

    • Quantization error will propagate unless the memory of predictor is refreshed

    • Not suitable for scalable coding applications

EE591f Digital Video Processing


Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


Scalable vs multicast
Scalable vs. Multicast

  • What is scalable coding?

foreman.yuv

foreman.yuv

foreman128k.cod

foreman.cod

foreman256k.cod

foreman512k.cod

foreman1024k.cod

128

256

512

1024

Multicast

Scalable coding

EE591f Digital Video Processing


Spatial scalability
Spatial scalability

EE591f Digital Video Processing


Temporal scalability

Frame 0,1,2,3,4,5,…

Frame 0,4,8,12,…

Frame 0,2,4,6,8,…

7.5Hz

15Hz

30Hz

EE591f Digital Video Processing


SNR (Rate) scalability

PSNRavg=40dB

PSNRavg=30dB

PSNRavg=35dB

PSNRi: PSNR of frame i

EE591f Digital Video Processing


Scalability via bit plane coding
Scalability via Bit-Plane Coding

sign bit

A=(a0+a12+a222+ … … +a727)

Least Significant Bit

(LSB)

Most Significant Bit

(MSB)

Example

A=129  sign=+,a0a1a2 …a7=10000001

sign=-, a0a1a2 …a7=00110011

A=-(4+8+64+128)=-204

EE591f Digital Video Processing


Why dpcm bad for scalability
Why DPCM Bad for Scalability?

Frame number

3

1

2

Base layer

Ibase

P

P

P

Enhancement

Layer 1

Ienh1

P

P

P

Enhancement

Layer 2

Ienh2

P

P

P

suffer from drifting problem

suffer from coding efficiency loss

EE591f Digital Video Processing


3d wavelet subband coding
3D Wavelet/Subband Coding

y

t

x

2D spatial WT+1D temporal WT

EE591f Digital Video Processing


Motion adaptive 3d wavelet transform
Motion-Adaptive 3D Wavelet Transform

Recall Haar transform

lifting-based implementation

Motion-adaptive Haar transform

W,W-1: forward and backward motion vector

EE591f Digital Video Processing


ad