Distributed video system realized on mobile device with efficient feedback channel
Download
1 / 75

Distributed Video System realized on mobile device with efficient Feedback channel - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Distributed Video System realized on mobile device with efficient Feedback channel. 分散 式 影像編碼在手機上的實現與有效率的回饋通道. Chen, chun-yuan 陳群元 Advisor: Prof. Wu, Ja -Ling 吳家麟 教授 2012/7/9. Outline. Motivation and introduction DVC architecture overview DVC architecture with mobile device

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Distributed Video System realized on mobile device with efficient Feedback channel' - tasya


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Distributed video system realized on mobile device with efficient feedback channel

Distributed Video System realized on mobile device with efficient Feedback channel

分散式影像編碼在手機上的實現與有效率的回饋通道

Chen, chun-yuan陳群元

Advisor: Prof. Wu, Ja-Ling 吳家麟 教授

2012/7/9


Outline
Outline efficient Feedback channel

  • Motivation and introduction

  • DVC architecture overview

  • DVCarchitecture with mobiledevice

  • Proposed algorithm for efficient feedback channel

  • Experimental Results

  • Conclusion

  • Future work


Motivation
Motivation efficient Feedback channel

  • Video communication is essential in Mobile device

  • Decrease the coding time and battery consuming on mobile.


Introduction
Introduction efficient Feedback channel

  • For Mobile device video codec

    • Conventional video coding (ex:MPEG-4 H.264)

      • Heavy weight encoder, light weight decoder.

    • Distributed Video Coding

      • Light weight encoder, heavy weight decoder.


Conventional video codec
Conventional video codec efficient Feedback channel

Decoding

Encoding


Dvc to h 264 transcoder
DVC to H.264 Transcoder efficient Feedback channel

Clouding server:

DVC to H.264 transcoder

LDPCA

decoding is the bottleneck of the overall codec system.

Feedback

channel

Encoded

sequence

Parity bits

DVC encoder

H.264 decoder

Tse-Chung Su, Yun-Chung Shen , and Ja-Ling Wu , “Real-time decoding for LDPC based distributed video coding

,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011.


Distributed video system realized on mobile device with efficient feedback channel

1.Encoder

without an actual

feedback channel !

2.Decoder

Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu. 2011. Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University


Time complexity analysis without feedback channel
Time complexity analysis without feedback channel efficient Feedback channel

  • DVC Decoding without Feedback channel

    • Side-Information generation

    • LDPCA

Total decoding time: 17.10 sec

31%

Tse-Chung Su, Yun-Chung Shen , and Ja-Ling Wu , “Real-time decoding for LDPC based distributed video coding

,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2011.


Time complexity analysis with feedback channel
Time complexity analysis efficient Feedback channelwith feedback channel

  • DVC Decoding

    • Side-Information generation

    • Send-Request at Feedback channel

    • LDPCA

Total decoding time: 446.83 sec

1%

96%

Bottle neck!

T. V. Lakshman and U. Madhow, “The performance of TCP/IP for networks with high bandwidth-delay products and random loss, IEEE/ACM Trans. Networking, vol. 6, pp. 336–350, June 1997.


Outline1
Outline efficient Feedback channel

  • Motivation and introduction

  • DVC architecture Overview

  • DVCarchitecture with mobiledevice

  • Proposed algorithm for efficient feedback channel

  • Experimental Results

  • Conclusion

  • Future work


Distributed video codec
Distributed Video Codec efficient Feedback channel

Decoded

WZ Frames

Key

Frame

Key

Frame

WZ Frames

CRC-8

Mode selection

CRC check

CRC

IDCT&

Deblocking filter

DCT

Feedback channel

LDPCA Encoder

LDPCA Decoder

WZ bitstream

level

Quantizer

SI Reconstruction

Buffer

Bitplanes

WZ

Frame

WZ

Frame

WZ

Frame

Correlation Noise Modeling

DCT

SI generation&

refinement

GOP size 4

Conventional

Intraframe

Encoder

Conventional

Intraframe

Decoder

Key Frames

Decoded

Key Frames


Ldpc decoding
LDPC decoding efficient Feedback channel

Side Information

(real number)

+  0

-  1

decode output

hard decision

a25 b25 c25 d25 e25 f25 g25

a1 b1 c1 d1 e1 f1 g1

a b c d e f g

3

4

7

1

5

6

2

Horizontal processing

Vertical processing

b

c

a

1

0

1

syndrome bits

Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm.IEEE Trans. Inform. Theory


Ldpc accumulate ldpca codes
LDPC Accumulate (LDPCA) codes efficient Feedback channel

Rate adaptivity

D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006


Distributed video system realized on mobile device with efficient feedback channel

65 LDPC codes efficient Feedback channel


Outline2
Outline efficient Feedback channel

  • Motivation and introduction

  • DVC architecture overview

  • DVCarchitecture with mobiledevice

  • Proposed algorithm for efficient feedback channel

  • Experimental Results

  • Conclusion

  • Future work


Distributed video codec with mobile
Distributed Video Codec with mobile efficient Feedback channel

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Decoder

Key Frames

Decoded

Key Frames

Android

java coding


Distributed video codec with mobile1
Distributed Video Codec with mobile efficient Feedback channel

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Decoder

Key Frames

Decoded

Key Frames


Key frame encoding on mobile
Key frame encoding on Mobile efficient Feedback channel

  • Previous ver. use JM9.5

  • Transcoder ref x264 on PC

Keyframe

encoding

Cross compiler

x264

static

library

x264

lib


Distributed video codec with mobile2
Distributed Video Codec with mobile efficient Feedback channel

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Decoder

Key Frames

Decoded

Key Frames


Wz coding with ndk
WZ coding with NDK efficient Feedback channel

Jni

Makefile

Native

code

DVC encoder

C/C++ code

with NDK rule


Distributed video codec with mobile3
Distributed Video Codec with mobile efficient Feedback channel

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Decoder

Key Frames

Decoded

Key Frames


Network connection
Network connection efficient Feedback channel

  • Create network connection to connect remote server.

    • TCP socketin C

Clouding server:

DVC to H.264 transcoder


Ack at feedback channel
ACK at Feedback channel efficient Feedback channel

TCP socket

syndromes

syndromes

syndromes

syndromes

Send syndrome

Request


Decoding complexity on feedback channel
Decoding complexity on efficient Feedback channelFeedback channel

  • Packet header occupy large part of network bandwidth.

    • 48 bits syndromes vs. 24 bytes header

  • Network latency in communication at Feedback channel

    • About 20 thousands times requests for 10 sec sequence.


Outline3
Outline efficient Feedback channel

  • Motivation and introduction

  • DVCarchitecture with mobiledevice

  • DVC architecture overview

  • Proposed algorithm for efficient feedback channel

  • Experimental Results

  • Conclusion

  • Future work


Efficient feedback channel
Efficient feedback channel efficient Feedback channel

  • we propose two methods to decrease time consuming at feedback channel.

    • Estimate the syndrome size per bitplane

    • Estimate the syndrome size per WZ frame


Number of requests per bitplane per band
Number of requests per efficient Feedback channelbitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number


Number of requests per bitplane per band1
Number of requests per efficient Feedback channelbitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number


Number of requests per bitplane per band2
Number of requests per efficient Feedback channelbitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number


Efficient feedback channel1
Efficient feedback channel efficient Feedback channel

  • we propose two methods to decrease time consuming at feedback channel.

    • Estimate the syndrome size per bitplane

    • Estimate the syndrome size per WZ frame


A verage number of requests per bitplane
A efficient Feedback channelverage number of requests per bitplane

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number


Estimate the syndromes size per wz frame
Estimate the syndromes size per WZ frame efficient Feedback channel

  • Estimate syndroms size by correspond WZ frame in previous GOP

  • Ex. GOP 4

WZ

frame

WZ

frame

WZ

frame

WZ

frame

Key

frame

Key

frame

WZ

frame

WZ

frame

Key

frame

Second GOP

First GOP


Outline4
Outline efficient Feedback channel

  • Motivation and introduction

  • DVC architecture overview

  • DVCarchitecture with mobiledevice

  • Proposed algorithm for efficient feedback channel

  • Experimental Results

  • Conclusion

  • Future work


Test condition
Test condition efficient Feedback channel

  • 12 CPU, 24 processor

    • Intel(R) Xeon(R) CPU X5650 @ 2.67GHz

  • GPU: Tesla M2050

  • Mobile device: HTC sensation


  • Test materials
    Test materials efficient Feedback channel

    Motion: High Low

    Soccer

    Foreman

    Coastguard

    Hall Monitor

    • Test sequences :

    • QCIF, 15Hz, all frames

    • GOP Size 2, 4 and 8

    • Only luminance component is used


    Decoding time
    Decoding time efficient Feedback channel


    Request amount
    Request amount efficient Feedback channel


    Speed up ratio
    Speed up ratio efficient Feedback channel


    Bitrate
    Bitrate efficient Feedback channel


    Rd foreman
    RD(Foreman) efficient Feedback channel


    Rd hall
    RD(hall) efficient Feedback channel


    Outline5
    Outline efficient Feedback channel

    • Motivation and introduction

    • DVC architecture overview

    • DVCarchitecture with mobiledevice

    • Proposed algorithm for efficient feedback channel

    • Experimental Results

    • Conclusion

    • Future work


    C onclusion
    C efficient Feedback channelonclusion

    • We build a realistic DVC system based on mobile and cloud.

    • Propose algorithm for Feedback channel which the previous work doesn’t discuss or improve.

    • Transcode the decoded sequence to mobile codec.


    Outline6
    Outline efficient Feedback channel

    • Motivation and introduction

    • DVC architecture overview

    • DVCarchitecture with mobiledevice

    • Proposed algorithm for efficient feedback channel

    • Experimental Results

    • Conclusion

    • Future work


    Future work
    Future Work efficient Feedback channel

    • Take a more precise estimation of syndrome size.

    • Use motion information to estimate the syndrome size.

    • Build a reliable DVC codec which is loss tolerant.


    Distributed video system realized on mobile device with efficient feedback channel

    Thank You efficient Feedback channel


    Streaming
    streaming efficient Feedback channel

    Key

    Frame

    Key

    Frame

    Key

    Frame

    Key

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    GOP size 4

    GOP size 4


    Distributed video system realized on mobile device with efficient feedback channel

    Key efficient Feedback channel

    Frame

    Key

    Frame

    Key

    Frame

    Key

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    WZ

    Frame

    GOP size 4

    GOP size 4


    To do
    To do efficient Feedback channel

    • Real time decoder

    • Efficient intra coding

    • Print MSG on Mobile


    Dvc decoding time
    DVC decoding time efficient Feedback channel

    • Side Info Creation: 0.85 sec

    • Overcomplete Transform: 2.35 sec

    • Correlation Noise Modeling: 2.33 sec

    • Conditional Bit Prob Compute: 0.60 sec

    • Update Side Info: 7.14 sec

    • Ldpca Decode: 123.34 sec

    • Motion Learning: 1.79 sec

    • Others: 1.80 sec


    Distributed video codec1
    Distributed Video Codec efficient Feedback channel

    Decoded

    WZ Frames

    WZ Frames

    CRC-8

    Mode selection

    CRC check

    CRC

    IDCT&

    Deblocking filter

    DCT

    Feedback channel

    LDPCA Encoder

    LDPCA Decoder

    WZ bitstream

    level

    Quantizer

    SI Reconstruction

    Buffer

    Bitplanes

    Correlation Noise Modeling

    DCT

    SI generation&

    refinement

    Conventional

    Intraframe

    Encoder

    Conventional

    Intraframe

    Encoder

    Key Frames

    Decoded

    Key Frames


    Distributed video codec with mobile4
    Distributed Video Codec with mobile efficient Feedback channel

    Decoded

    WZ Frames

    CRC-8

    Decoded

    Video sequence

    With specific form

    Transcode

    level

    Quantizer

    Bitplanes

    CRC check

    CRC

    Video

    deliver

    IDCT&

    Deblocking filter

    Feedback channel

    DCT

    LDPCA Encoder

    LDPCA Decoder

    WZ bitstream

    SI Reconstruction

    Buffer

    Mode selection

    Correlation Noise Modeling

    DCT

    WZ Frames

    SI generation&

    refinement

    Conventional

    Intraframe

    Encoder

    Conventional

    Intraframe

    Encoder

    Key Frames

    Decoded

    Key Frames


    Distributed video system realized on mobile device with efficient feedback channel

    Video recode efficient Feedback channel

    With

    mobile camera


    Ack at feedback channel1
    ACK at Feedback channel efficient Feedback channel

    LDPCA Encoder

    &CRC

    LDPCA Decoder

    &CRC

    Syndromes

    for one bitplane

    Send syndrome


    Ack at feedback channel2
    ACK at Feedback channel efficient Feedback channel

    LDPCA Encoder

    &CRC

    LDPCA Decoder

    &CRC

    Syndromes

    for one bitplane

    Decoding one bitplane

    with received syndromes

    And chack CRC

    Request


    Ack at feedback channel3
    ACK at Feedback channel efficient Feedback channel

    LDPCA Encoder

    &CRC

    LDPCA Decoder

    &CRC

    Syndromes

    for one bitplane

    Send syndrome


    Contribution
    Contribution efficient Feedback channel

    • Realize DVC encoding part on Mobile device

    • Implement Feedback channel

    • Implement encoder’s component with x264

    • Propose an efficient Feedback channel

    • Transcode the decoded sequence to mobile codec


    Outline7
    Outline efficient Feedback channel

    • Motivation and introduction

    • DVCarchitecture with mobiledevice

    • Propose an efficient feedback channel

    • Transcoding

    • Experiment Result

    • Conclusion

    • Future work


    Quantization
    Quantization efficient Feedback channel

    Q8

    Q5

    2^5

    2^4

    2^3

    2^2

    2^7

    2^6

    2^5

    2^4

    2^6

    2^5

    2^4

    2^3

    2^4

    2^3

    2^2

    2^2

    2^3

    2^2

    2^2

    2^5

    2^4

    2^3

    2^2

    2^2

    2^2

    2^4

    2^3

    2^2

    63bits for one block

    37bits for one block


    Distributed video codec2
    Distributed Video Codec efficient Feedback channel

    Decoded

    WZ Frames

    WZ Frames

    CRC-8

    Mode selection

    CRC check

    CRC

    IDCT&

    Deblocking filter

    DCT

    Feedback channel

    LDPCA Encoder

    LDPCA Decoder

    WZ bitstream

    level

    Quantizer

    SI Reconstruction

    Buffer

    Bitplanes

    Correlation Noise Modeling

    DCT

    SI generation&

    refinement

    Conventional

    Intraframe

    Encoder

    Conventional

    Intraframe

    Encoder

    Key Frames

    Decoded

    Key Frames


    Distributed video codec with mobile5
    Distributed Video Codec with mobile efficient Feedback channel

    Decoded

    WZ Frames

    CRC-8

    Decoded

    Video sequence

    With specific form

    Transcode

    level

    Quantizer

    Bitplanes

    CRC check

    CRC

    Video

    deliver

    IDCT&

    Deblocking filter

    Feedback channel

    DCT

    LDPCA Encoder

    LDPCA Decoder

    WZ bitstream

    SI Reconstruction

    Buffer

    Mode selection

    Correlation Noise Modeling

    DCT

    WZ Frames

    SI generation&

    refinement

    Conventional

    Intraframe

    Encoder

    Conventional

    Intraframe

    Encoder

    Key Frames

    Decoded

    Key Frames


    Distributed video codec with mobile6
    Distributed Video Codec with mobile efficient Feedback channel

    Decoded

    WZ Frames

    CRC-8

    Decoded

    Video sequence

    With specific form

    Transcode

    level

    Quantizer

    Bitplanes

    CRC check

    CRC

    Video

    deliver

    IDCT&

    Deblocking filter

    Feedback channel

    DCT

    LDPCA Encoder

    LDPCA Decoder

    WZ bitstream

    SI Reconstruction

    Buffer

    Mode selection

    Correlation Noise Modeling

    DCT

    WZ Frames

    SI generation&

    refinement

    Conventional

    Intraframe

    Encoder

    Conventional

    Intraframe

    Encoder

    Key Frames

    Decoded

    Key Frames


    Dvc encoder realization
    DVC Encoder Realization efficient Feedback channel

    • Implement DVC encoder on Mobile

      • Media recording

      • Key frame encoding

      • WZ frame encoding

    • Feedback channel transmission

      • Network connection between mobile and server

      • Transcode the result sequence


    Formula
    formula efficient Feedback channel

    • (ESbt)WZn=(Sbt)WZ(n-GOPsize)

      • WZn means the WZ frame’s index

      • n-GOPsize means the correspond WZ frame in the previous GOP.

      • bt is the bitplane index.

      • (ESbt)WZn means the estimated syndromes bit for the bt’thbitplane of WZn frame.

      • (Sbt)WZ(n-GOPsize) means the syndromes bit for the bt’thbitplane of WZ(n-GOPsize)frame


    Formula1
    formula efficient Feedback channel

    • (ESacnbt)WZn= ( Sdcbt-1 )WZn

      • WZn means the n’th WZ frame.

      • acn means the n’th AC band.

      • bt means the bitplane index of this AC band.

      • Which ESacnbtmean the estimated syndrome size for the bt’thbitplane in n’th AC band.

      • dc means the DC band.

      • Sdcbt-1means the syndromes bits for bt-1’th bitplane in DC band.


    Time complexity
    Time complexity efficient Feedback channel

    95%

    75%


    Distributed video system realized on mobile device with efficient feedback channel

    bottleneck component of decoding process is always the efficient Feedback channel

    LDPCA decoding

    Yu-Shan Pai, Han-Ping Cheng, Yun-Chung Shen,andJa-Ling Wu , “Fast decoding for LDPC based distributed video coding,” MM '11 Proceedings of the 19th ACM international conference on Multimedia,, 2010.


    Transcode to mobile codec
    Transcode to mobile codec efficient Feedback channel

    • Because DVC only encode luminance part in YUV, so we got only Y for output.

    • Our mobile device can only read mp4 and 3gp.

    Y

    G

    R

    B

    Frame 1

    Frame 2

    Frame 3

    Frame4

    Frame 5

    Frame 6

    Frame 7

    U

    V

    .mp4

    Frame n


    Correlation with ldpca
    Correlation with LDPCA efficient Feedback channel

    Residual

    SI quality


    Distributed video system realized on mobile device with efficient feedback channel
    統計關係圖 efficient Feedback channel