distributed video system realized on mobile device with efficient feedback channel
Download
Skip this Video
Download Presentation
Distributed Video System realized on mobile device with efficient Feedback channel

Loading in 2 Seconds...

play fullscreen
1 / 73

Distributed Video System realized on mobile device with efficient Feedback channel - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Distributed Video System realized on mobile device with efficient Feedback channel. 分散 式 影像編碼在手機上的實現與有效率的回饋通道. Chen, chun-yuan 陳群元 Advisor: Prof. Wu, Ja -Ling 吳家麟 教授 2012/6/28. Outline. Motivation and introduction DVC architecture overview DVC architecture with mobile device

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Distributed Video System realized on mobile device with efficient Feedback channel' - dirk


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
distributed video system realized on mobile device with efficient feedback channel

Distributed Video System realized on mobile device with efficient Feedback channel

分散式影像編碼在手機上的實現與有效率的回饋通道

Chen, chun-yuan陳群元

Advisor: Prof. Wu, Ja-Ling 吳家麟 教授

2012/6/28

outline
Outline
  • Motivation and introduction
  • DVC architecture overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Experiment Result
  • Conclusion
  • Future work
motivation
Motivation
  • Video communication is essential in Mobile device
  • Decrease the coding time and battery consuming on mobile.
introduction
Introduction
  • For Mobile device video codec
    • Conventional video coding (ex:MPEG-4 H.264)
      • Heavy weight encoder, light weight decoder.
    • Distributed Video Coding
      • Light weight encoder, heavy weight decoder.
conventional video codec
Conventional video codec

Decoding

Encoding

dvc to h 264 transcoder
DVC to H.264 Transcoder

Clouding server:

DVC to H.264 transcoder

Feedback channel

Parity bits

Encoded sequence

H.264 decoder

DVC encoder

slide7

DVC codec baseline:

    • DISPAC+DVC transcoder

1.Encoder

no feedback channel !

2.Decoder

Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu. 2011. Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University

time complexity analysis without feedback channel
Time complexity analysis without feedback channel
  • DVC Decoding without Feedback channel
    • Side-Information generation
    • LDPCA

Total decoding time: 17.10 sec

31%

time complexity analysis with feedback channel
Time complexity analysis with feedback channel
  • DVC Decoding
    • Side-Information generation
    • Send-Request at Feedback channel
    • LDPCA

Total decoding time: 446.83 sec

1%

96%

Bottle neck!

outline1
Outline
  • Motivation and introduction
  • DVC architecture Overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Experiment Result
  • Conclusion
  • Future work
distributed video codec
Distributed Video Codec

Decoded

WZ Frames

Key

Frame

Key

Frame

WZ Frames

CRC-8

Mode selection

CRC check

CRC

IDCT&

Deblocking filter

DCT

Feedback channel

LDPCA Encoder

LDPCA Decoder

WZ bitstream

level

Quantizer

SI Reconstruction

Buffer

Bitplanes

WZ

Frame

WZ

Frame

WZ

Frame

Correlation Noise Modeling

DCT

SI generation&

refinement

GOP size 4

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

ldpc decoding sum product algorithm message passing
LDPC decodingSum-Product Algorithm(Message Passing)

Side Information

(real number)

+  0

-  1

decode output

hard decision

a25 b25 c25 d25 e25 f25 g25

a1 b1 c1 d1 e1 f1 g1

a b c d e f g

4

6

7

1

2

3

5

Horizontal

processing

Vertical processing

0

1

1

From DVC encoder

(syndrome bits)

Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm.IEEE Trans. Inform. Theory

ldpc accumulate ldpca codes
LDPC Accumulate (LDPCA) codes

Rate adaptivity

D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006

outline2
Outline
  • Motivation and introduction
  • DVC architecture overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Experiment Result
  • Conclusion
  • Future work
distributed video codec with mobile
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

distributed video codec with mobile1
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

key frame encoding on mobile
Key frame encoding on Mobile
  • Previous ver. use JM9.5
  • Transcoder ref x264 on PC

Keyframe

encoding

Cross compiler

x264

static

library

x264

lib

distributed video codec with mobile2
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

wz coding with ndk
WZ coding with NDK

Jni

Makefile

Native

code

DVC encoder

C/C++ code

with NDK rule

distributed video codec with mobile3
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

transcode to mobile codec
Transcode to mobile codec
  • Because DVC only encode luminance part in YUV, so we got only Y for output.
  • Our mobile device can only read mp4 and 3gp.

Y

G

R

B

Frame 1

Frame 2

Frame 3

Frame4

Frame 5

Frame 6

Frame 7

U

V

.mp4

Frame n

distributed video codec with mobile4
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

network connection
Network connection
  • Create network connection to connect remote server.
    • TCP socketin C

Clouding server:

DVC to H.264 transcoder

ack at feedback channel
ACK at Feedback channel

TCP socket

syndromes

syndromes

syndromes

syndromes

Send syndrome

Request

decoding complexity on feedback channel
Decoding complexity onFeedback channel
  • Packet header occupy large part of network bandwidth.
    • 48 bits syndromes vs. 24 bytes header
  • Network latency in communication at Feedback channel
    • About 20 thousands times requests for 10 sec sequence.
outline3
Outline
  • Motivation and introduction
  • DVCarchitecture with mobiledevice
  • DVC architecture overview
  • Propose an efficient feedback channel
  • Experiment Result
  • Conclusion
  • Future work
correlation with ldpca
Correlation with LDPCA

Residual

SI quality

efficient feedback channel
Efficient feedback channel
  • we propose two methods to decrease time consuming at feedback channel.
    • Estimate the syndromes size per WZ frame
    • Estimate the syndromes size per bitplane
a verage number of requests per bitplane
Average number of requests per bitplane

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number

estimate the syndromes size per wz frame
Estimate the syndromes size per WZ frame
  • Estimate syndromes size by correspond WZ frame in previous GOP
  • Ex. GOP 4

WZ

frame

WZ

frame

WZ

frame

WZ

frame

Key

frame

Key

frame

WZ

frame

WZ

frame

Key

frame

Second GOP

First GOP

efficient feedback channel1
Efficient feedback channel
  • we propose two methods to decrease time consuming at feedback channel.
    • Estimate the syndromes size per WZ frame
    • Estimate the syndromes size per bitplane
number of requests per bitplane per band
Number of requests per bitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number

number of requests per bitplane per band1
Number of requests per bitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number

number of requests per bitplane per band2
Number of requests per bitplane per band

AC11

AC13

AC9

AC10

AC12

AC14

AC6

AC7

AC8

AC1

AC2

AC4

AC5

DC

AC3

Bitplane Number

outline4
Outline
  • Motivation and introduction
  • DVC architecture overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Transcoding
  • Experiment Result
  • Conclusion
  • Future work
test condition
Test condition
  • 12 CPU, 24 processor
      • Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
  • GPU: Tesla M2050
  • Mobile device: HTC sensation
test materials
Test materials

Motion: High Low

Soccer

Foreman

Coastguard

Hall Monitor

  • Test sequences :
  • QCIF, 15Hz, all frames
  • GOP Size 2, 4 and 8
  • Only luminance component is used
outline5
Outline
  • Motivation and introduction
  • DVC architecture overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Transcoding
  • Experiment Result
  • Conclusion
  • Future work
c onclusion
Conclusion
  • We combine DISPAC and transcoder, then port it’s encoder on mobile.
  • Implement and propose Feedback channel which the previous work doesn’t discuss or improve.
  • Record video with mobile device’s camera.
  • Transcode the decoded sequence to mobile codec.
outline6
Outline
  • Motivation and introduction
  • DVC architecture overview
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Experiment Result
  • Conclusion
  • Future work
future work
Future Work
  • Take a more precise estimation of syndrome size.
  • Use motion information to estimate the syndrome size.
  • Build a reliable DVC codec which is loss tolerant.
to do
To do
  • Real time decoder
  • Efficient intra coding
  • Print MSG on Mobile
dvc decoding time
DVC decoding time
  • Side Info Creation: 0.85 sec
  • Overcomplete Transform: 2.35 sec
  • Correlation Noise Modeling: 2.33 sec
  • Conditional Bit Prob Compute: 0.60 sec
  • Update Side Info: 7.14 sec
  • Ldpca Decode: 123.34 sec
  • Motion Learning: 1.79 sec
  • Others: 1.80 sec
distributed video codec1
Distributed Video Codec

Decoded

WZ Frames

WZ Frames

CRC-8

Mode selection

CRC check

CRC

IDCT&

Deblocking filter

DCT

Feedback channel

LDPCA Encoder

LDPCA Decoder

WZ bitstream

level

Quantizer

SI Reconstruction

Buffer

Bitplanes

Correlation Noise Modeling

DCT

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

distributed video codec with mobile5
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

slide56

Video recode

With

mobile camera

ack at feedback channel1
ACK at Feedback channel

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes

for one bitplane

Send syndrome

ack at feedback channel2
ACK at Feedback channel

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes

for one bitplane

Decoding one bitplane

with received syndromes

And chack CRC

Request

ack at feedback channel3
ACK at Feedback channel

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes

for one bitplane

Send syndrome

contribution
Contribution
  • Realize DVC encoding part on Mobile device
  • Implement Feedback channel
  • Implement encoder’s component with x264
  • Propose an efficient Feedback channel
  • Transcode the decoded sequence to mobile codec
outline7
Outline
  • Motivation and introduction
  • DVCarchitecture with mobiledevice
  • Propose an efficient feedback channel
  • Transcoding
  • Experiment Result
  • Conclusion
  • Future work
quantization
Quantization

Q8

Q5

2^5

2^4

2^3

2^2

2^7

2^6

2^5

2^4

2^6

2^5

2^4

2^3

2^4

2^3

2^2

2^2

2^3

2^2

2^2

2^5

2^4

2^3

2^2

2^2

2^2

2^4

2^3

2^2

63bits for one block

37bits for one block

distributed video codec2
Distributed Video Codec

Decoded

WZ Frames

WZ Frames

CRC-8

Mode selection

CRC check

CRC

IDCT&

Deblocking filter

DCT

Feedback channel

LDPCA Encoder

LDPCA Decoder

WZ bitstream

level

Quantizer

SI Reconstruction

Buffer

Bitplanes

Correlation Noise Modeling

DCT

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

distributed video codec with mobile6
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

distributed video codec with mobile7
Distributed Video Codec with mobile

Decoded

WZ Frames

CRC-8

Decoded

Video sequence

With specific form

Transcode

level

Quantizer

Bitplanes

CRC check

CRC

Video

deliver

IDCT&

Deblocking filter

Feedback channel

DCT

LDPCA Encoder

LDPCA Decoder

WZ bitstream

SI Reconstruction

Buffer

Mode selection

Correlation Noise Modeling

DCT

WZ Frames

SI generation&

refinement

Conventional

Intraframe

Encoder

Conventional

Intraframe

Encoder

Key Frames

Decoded

Key Frames

dvc encoder realization
DVC Encoder Realization
  • Implement DVC encoder on Mobile
    • Media recording
    • Key frame encoding
    • WZ frame encoding
  • Feedback channel transmission
    • Network connection between mobile and server
    • Transcode the result sequence
formula
formula
  • (ESbt)WZn=(Sbt)WZ(n-GOPsize)
    • WZn means the WZ frame’s index
    • n-GOPsize means the correspond WZ frame in the previous GOP.
    • bt is the bitplane index.
    • (ESbt)WZn means the estimated syndromes bit for the bt’thbitplane of WZn frame.
    • (Sbt)WZ(n-GOPsize) means the syndromes bit for the bt’thbitplane of WZ(n-GOPsize)frame
formula1
formula
  • (ESacnbt)WZn= ( Sdcbt-1 )WZn
    • WZn means the n’th WZ frame.
    • acn means the n’th AC band.
    • bt means the bitplane index of this AC band.
    • Which ESacnbtmean the estimated syndrome size for the bt’thbitplane in n’th AC band.
    • dc means the DC band.
    • Sdcbt-1means the syndromes bits for bt-1’th bitplane in DC band.
ad