Roadmap
This presentation is the property of its rightful owner.
Sponsored Links
1 / 50

Roadmap PowerPoint PPT Presentation


  • 109 Views
  • Uploaded on
  • Presentation posted in: General

Roadmap. Introduction Intra-frame coding Review of JPEG Inter-frame coding Conditional Replenishment (CR) Motion Compensated Prediction (MCP) Object-based and scalable video coding* Motion segmentation, scalability issues. Introduction to Video Coding.

Download Presentation

Roadmap

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Roadmap

Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Object-based and scalable video coding*

    • Motion segmentation, scalability issues

EE591f Digital Video Processing


Introduction to video coding

Introduction to Video Coding

  • Lossless vs. lossy data compression

    • Source entropy H(X)

    • Rate-Distortion function R(D) or D(R)

  • Probabilistic modeling is at the heart of data compression

    • What is P(X) for video source X?

    • Modeling moving pictures is more difficult than modeling still images due to temporal dependency

EE591f Digital Video Processing


Shannon s picture

Shannon’s Picture

Distortion

Coder A

Coder B

Rate (bps)

Which coder wins, A or B?

EE591f Digital Video Processing


Distortion measures

Distortion Measures

  • Objective

    • Mean Square Error (MSE)

    • Peak Signal-to-Noise-Ratio (PSNR)

    • Measure the fidelity to original video

  • Subjective

    • Human Vision System (HVS) based

    • Emphasize visual quality rather than fidelity

  • We only discuss objective measures in this course

EE591f Digital Video Processing


Roadmap

Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


Roadmap

A Tour of JPEG Coding Standard

Key Components

  • Transform

-8×8 DCT

-boundary padding

  • Quantization

-uniform quantization

-DC/AC coefficients

  • Coding

-Zigzag scan

-run length/Huffman coding

EE591f Digital Video Processing


Roadmap

JPEG Baseline Coder

Tour Example

EE591f Digital Video Processing


Roadmap

Step 1: Transform

• DC level shifting

-128

• 2D DCT

DCT

EE591f Digital Video Processing


Roadmap

Step 2: Quantization

Why increase

from top-left to

bottom-right?

Q-table

Q

EE591f Digital Video Processing


Roadmap

Step 3: Entropy Coding

Zigzag Scan

(20,5,-3,-1,-2,-3,1,1,-1,-1,

0,0,1,2,3,-2,1,1,0,0,0,0,0,

0,1,1,0,1,EOB)

End Of the Block:

All following coefficients

are zero

Zigzag Scan

EE591f Digital Video Processing


Roadmap

Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


Conditional replenishment

Conditional Replenishment

  • Based on motion detection rather than motion estimation

  • Partition the current frame into “still areas” and “moving areas”

    • Replenishment is applied to moving regions only

    • Repetition is applied to still regions

  • Need to transmit the location of moving areas as well as new (replenishment) information

    • No motion vectors transmitted

EE591f Digital Video Processing


Conditional replenishment1

Conditional Replenishment

EE591f Digital Video Processing


Motion detection

Motion Detection

EE591f Digital Video Processing


From replenishment to prediction

From Replenishment to Prediction

  • Replenishment can be viewed as a degenerated case of prediction

    • Only zero motion vector is considered

    • Discard the history

  • A more powerful approach of exploiting temporal dependency is prediction

    • Locate the best match from the previous frame

    • Use the history to predict the current

EE591f Digital Video Processing


Differential pulse coded modulation

Differential Pulse Coded Modulation

^

xn-1

^

^

^

yn

yn

yn

xn

xn

_

Q

+

^

^

xn-1

xn

D

+

D

^

xn-1

Decoder

Encoder

Xn,yn: unquantized samples and prediction residues

^

^

Xn,yn: decoded samples and quantized prediction residues

EE591f Digital Video Processing


Motion compensated predictive coding

Motion-Compensated Predictive Coding

EE591f Digital Video Processing


A closer look

A Closer Look

EE591f Digital Video Processing


Key components

Key Components

  • Motion Estimation/Compensation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Block based motion model

Block-based Motion Model

  • Block size

    • Fixed vs. variable

  • Motion accuracy

    • Integer-pel vs. fractional-pel

  • Number of hypothesis

    • Overlapped Block Motion Compensation (OBMC)

    • Multi-frame prediction

EE591f Digital Video Processing


Quadtree representation of motion field with variable blocksize

Quadtree Representation of Motion Field with Variable Blocksize

EE591f Digital Video Processing


Rate distortion optimized bma

Rate-Distortion Optimized BMA

Distortion alone

Rate and Distortion

counted bits using a VLC table

EE591f Digital Video Processing


Experimental results

Experimental Results

Cited from G. Sullivan and L. Baker, “Rate-Distortion optimized

motion compensation for video compression using fixed or

variable size blocks”, Globecom’1991

EE591f Digital Video Processing


Fractional pel bma

Fractional-pel BMA

  • Recall the tradeoff between spending bits on motion and spending bits on MCP residues

  • Intuitively speaking, going from integer-pel to fractional-pel is good for it dramatically reduces the variance of MCP residues for some video sequence.

  • The gain quickly saturates as motion accuracy refines

EE591f Digital Video Processing


Example

Example

8-by-8 block, integer-pel, var(e)=220.8

8-by-8 block, half-pel, var(e)=123.8

MCP residue comparison for the first two frames of Mobile sequence

EE591f Digital Video Processing


Fractional pel mcp

Fractional-pel MCP

EE591f Digital Video Processing


Multi hypothesis mcp

Multi-Hypothesis MCP

  • Using one block from one reference frame represents a single-hypothesis MCP

  • It is possible to formulate multiple hypothesis by considering

    • Overlapped blocks

    • More than one reference frame

  • Why multi-hypothesis?

    • The benefit of reducing variance of MCP residues outweighs the increased overhead on motion

EE591f Digital Video Processing


Example b frame

Example: B-frame

fn-1

fn

fn+1

EE591f Digital Video Processing


Roadmap

Generalized B-frame

fn-2

fn-1

fn+2

fn

fn+1

EE591f Digital Video Processing


Multi hypothesis mcp1

Multi-Hypothesis MCP

EE591f Digital Video Processing


Roadmap

Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Motion vector coding

Motion Vector Coding

  • 2D lossless DPCM

    • Spatially (temporally) adjacent motion vectors are correlated

    • Use causal neighbors to predict the current one

    • Code Motion Vector Difference (MVD) instead of MVs

  • Entropy coding techniques

    • Variable length codes (VLC)

    • Arithmetic coding

EE591f Digital Video Processing


Mvd example

MVD Example

MV1

MV2

MV

MV3

Due to smoothness of MV field, MVD usually has

a smaller variance than MV

EE591f Digital Video Processing


Vlc example

VLC Example

MVx/MVy

symbol

codeword

0

1

1

010

1

2

-1

011

3

00100

4

2

-2

00101

5

3

00110

6

Exponential Golomb Codes: 0…01x…x

m

m

EE591f Digital Video Processing


Roadmap

Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Mcp residue coding

MCP Residue Coding

Transform

Quantization

Coding

Conceptually similar to JPEG

Transform: unitary transform

Quantization: Deadzone quantization

Coding: Run-length coding

EE591f Digital Video Processing


Transform

Transform

Unitary matrix: A is real, A-1=AT

Unitary transform: A is unitary, Y=AXAT

Examples

8-by-8 DCT

4-by-4 integer transform

EE591f Digital Video Processing


Deadzone quantization

Deadzone Quantization

deadzone

2

0

codewords

EE591f Digital Video Processing


Roadmap

Key Components

  • Motion Estimation

    • At the heart of MCP-based coding

  • Coding of Motion Vectors (overhead)

    • Lossless: errors in MV are catastrophic

  • Coding of MCP residues

    • Lossy: distortion is controlled by the quantization step-size

  • Rate-Distortion optimization

EE591f Digital Video Processing


Lagrangian multiplier method

Lagrangian Multiplier Method

Motion estimation

Mode selection

QUANT: a user-specified parameter controlling

quantization stepsize

EE591f Digital Video Processing


Summary

Summary

  • How does MCP coding work?

    • The predictive model captures the slow-varying trend of the samples {fn}

    • The modeling of prediction residues {en} is easier than that of original samples {fn}

  • Fundamental weakness

    • Quantization error will propagate unless the memory of predictor is refreshed

    • Not suitable for scalable coding applications

EE591f Digital Video Processing


Roadmap

Roadmap

  • Introduction

  • Intra-frame coding

    • Review of JPEG

  • Inter-frame coding

    • Conditional Replenishment (CR)

    • Motion Compensated Prediction (MCP)

  • Scalable video coding

    • 3D subband/wavelet coding and recent trend

EE591f Digital Video Processing


Scalable vs multicast

Scalable vs. Multicast

  • What is scalable coding?

foreman.yuv

foreman.yuv

foreman128k.cod

foreman.cod

foreman256k.cod

foreman512k.cod

foreman1024k.cod

128

256

512

1024

Multicast

Scalable coding

EE591f Digital Video Processing


Spatial scalability

Spatial scalability

EE591f Digital Video Processing


Roadmap

Temporal scalability

Frame 0,1,2,3,4,5,…

Frame 0,4,8,12,…

Frame 0,2,4,6,8,…

7.5Hz

15Hz

30Hz

EE591f Digital Video Processing


Roadmap

SNR (Rate) scalability

PSNRavg=40dB

PSNRavg=30dB

PSNRavg=35dB

PSNRi: PSNR of frame i

EE591f Digital Video Processing


Scalability via bit plane coding

Scalability via Bit-Plane Coding

sign bit

A=(a0+a12+a222+ … … +a727)

Least Significant Bit

(LSB)

Most Significant Bit

(MSB)

Example

A=129  sign=+,a0a1a2 …a7=10000001

sign=-, a0a1a2 …a7=00110011

A=-(4+8+64+128)=-204

EE591f Digital Video Processing


Why dpcm bad for scalability

Why DPCM Bad for Scalability?

Frame number

3

1

2

Base layer

Ibase

P

P

P

Enhancement

Layer 1

Ienh1

P

P

P

Enhancement

Layer 2

Ienh2

P

P

P

suffer from drifting problem

suffer from coding efficiency loss

EE591f Digital Video Processing


3d wavelet subband coding

3D Wavelet/Subband Coding

y

t

x

2D spatial WT+1D temporal WT

EE591f Digital Video Processing


Motion adaptive 3d wavelet transform

Motion-Adaptive 3D Wavelet Transform

Recall Haar transform

lifting-based implementation

Motion-adaptive Haar transform

W,W-1: forward and backward motion vector

EE591f Digital Video Processing


  • Login