1 / 66

使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術

Progressive Side Information Refinement with Non-Local Means Denoising in Distributed Video Coding. Wang , Pin-Hsiang 王品翔 Advisor: Prof. Wu, Ja -Ling 吳家麟 教授 2011/10/13. 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術. CMLab , CSIE, NTU. Outline. Introduction and Motivation DVC Architecture Overview

marsha
Download Presentation

使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Progressive Side Information Refinement with Non-Local Means Denoising in Distributed Video Coding Wang, Pin-Hsiang 王品翔Advisor: Prof. Wu, Ja-Ling 吳家麟 教授 2011/10/13 使用於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術 CMLab, CSIE, NTU

  2. Outline • Introduction and Motivation • DVC Architecture Overview • Proposed Side Information Refinement Framework • Experimental Result • Conclusions and Future Work CMLab, CSIE, NTU

  3. Emerging application • Requiring low complexity and power-efficient encoder… Video surveillance mobile video conference Wireless sensor network mobile cameras phone CMLab, CSIE, NTU

  4. Emerging application • Requiring low complexity and power-efficient encoder… • Conventional video coding (e.g. H.264/AVC, MPEG-2) -Inherent high complexity encoder, low complexity decoder • Distributed video coding (DVC) - New video coding paradigm shifts complexity from encoder to decoder CMLab, CSIE, NTU

  5. Application of DVC • Make the Clients slimmer & thinner Cloud Computational Resource H.264 encoded bitstream DVC to H.264 Transcoder DVC encoded bitstream H.264 decoder (Low Complexity) DVC encoder (Low Complexity) CMLab, CSIE, NTU

  6. Encoder X Joint Encoder Joint Decoder Joint Decoder Distributed Video Coding • Slepian-Wolf Theorem (1973, Lossless coding) • Wyner-Ziv Theorem (1976, Lossy coding) Encoder Y Conventional video coding paradigm Statistical dependency is not exploited Statistical dependency Slepian-Wolf theorem CMLab, CSIE, NTU

  7. Distributed Video Coding Side Information Estimation side information (SI) • Wyner-Ziv Theorem (1976, Lossy coding) Joint Decoder Encoder X Parity bits ChannelEncoder Source Encoder Source Encoder ChannelDecoder Quantizer Quality Corrupted version of X Statistical dependency is not exploited Correlation is exploited at the decoder side Virtual channel Encoder Y Source Encoder Source Encoder Quantizer • DVC is also called Wyner-Ziv video coding (WZVC) CMLab, CSIE, NTU

  8. Motivation Encoder-side Past reference frame Future reference frame F( t ) Source X Decoder-side Side Information Estimation F( t ) F( t-1 ) F( t+1 ) CMLab, CSIE, NTU

  9. Motivation Encoder-side Past reference frame Future reference frame F( t ) Source X Decoder-side Frame interpolation (Decoder-side ME) F( t ) F( t-1 ) F( t+1 ) CMLab, CSIE, NTU

  10. Motivation Encoder-side Past reference frame Future reference frame F( t ) Source X Decoder-side Side information F( t-1 ) F( t ) F( t+1 ) CMLab, CSIE, NTU

  11. Motivation Encoder-side Past reference frame Future reference frame Most reported WZ codecs have a poor RD performance for high motion and large GOP size sequences F( t ) Limitation Source X Decoder-side Side information F( t-1 ) F( t ) F( t+1 ) CMLab, CSIE, NTU

  12. NLM-SIR framework for DVC • Non-Local Means Side Information Refinement framework (NLM-SIR) for DVC is proposed • Improve the SI quality to better rate-distortion (RD) performance of WZVC • Overcoming some of the limitations about current SI estimation methods in WZVC CMLab, CSIE, NTU

  13. Outline • Introduction and Motivation • DVC Architecture Overview • Proposed Side Information Refinement Framework • Experimental Result • Conclusions and Future Work CMLab, CSIE, NTU

  14. DVC Architecture Overview • Reference codec : • DISCOVER codec (Distributed coding for video services) • X. Artigas et al., “The DISCOVER codec: architecture, techniques and evaluation”, PCS, 2007 • Feedback channel based transform domain WZ codec CMLab, CSIE, NTU

  15. DISCOVER Codec Architecture WZ Encoder WZ Decoder Key Frame Key Frame Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Key Frame WZ Frame WZ Frame WZ Frame WZ Frame Key Frame Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P GOP size 2 GOP size 4 XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  16. S31 S21 S11 S12 S22 S32 S26 S36 S16 S17 S27 S37 S33 S23 S13 S15 S25 S35 S18 S28 S38 S213 S313 S113 S24 S34 S14 S29 S19 S39 S312 S212 S112 S114 S214 S314 S110 S210 S310 S111 S211 S311 S215 S115 S315 S216 S316 S116 Block1 Block3 Block2 Quantization • DCT coefficients bands … DC band • DCT coefficient band b1 : { S11, S21, S31, …SN1 } • DCT coefficient band b2 : { S12, S22, S32, …SN2 } AC bands … • DCT coefficient band b16 : { S116, S216, S316, …SN16 }

  17. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  18. Bit plane Extraction • For each DCT coefficient band… Bit planes of DC band: MSB 0 0 1 0 0 0 0 0 0 1 Bit plane 1: Bit plane 2: Independently Channel Encode (LDPCA) Bit plane 3: Bit plane 4: 0 0 0 0 0 1 1 1 1 0 Bit plane 5: Zig-zag order LSB 4 30 0 1 6 7 1 6 5 3 7 7

  19. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  20. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel Forward motion estimation LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  21. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Bidirectional motion estimation & compensation Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  22. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  23. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Laplacian distribution Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  24. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel Iterative decoding (band by band, bitplane by bitplane) LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Conditional bit probabilities DCT YDCT DCT XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  25. Reconstruction • Boundary reconstruction method : Quantization Interval Case3 Case1 Case2 : Side Information : Reconstructed value CMLab, CSIE, NTU

  26. DISCOVER Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Poor RD performance for high motion andlarge GOP size sequences Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input DCT YDCT DCT Room for improvement XDCT XWZ WZ Frames Frame Buffer Side Information Creation YWZ X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder X’P XK Decoded Key Frames Key Frames Ref. X. Artigas et al., PCS, 2007

  27. Outline • Introduction and Motivation • DVC Architecture Overview • Proposed Side Information Refinement Framework • Experimental Result • Conclusions and Future Work CMLab, CSIE, NTU

  28. NLM-SIR framework • Observed model : Original XWZframe (at the encoder) • Reduce noise to achieve better SI quality • Denoising problem Correlation noise Nbetween Original XWZframe and side information Side information YWZ(at the decoder)

  29. NLM-SIR framework Iterative decoding (band by band) Iterative decoding • New information about original WZ frame is not exploited • Progressive available during decoding • Not available at the time the initial SI was estimated

  30. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  31. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder DC band decoding CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation Initial SI (YWZ) is always used to decoding the DC band X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  32. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder AC bands decoding CRC-8 CRC Gen CRC Check Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  33. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Other decoding iterations Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Refined Side Information Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement Partially decoded WZ frame DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation NLM-SIR X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  34. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Other decoding iterations Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Decoded coefficients DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ Copy from Initial SI coefficients NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  35. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Other decoding iterations Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  36. Candidate Block Selection • Selection of the SI blocks which are worthwhile of refining • Initial guess by the side information creation process has basically failed (1) Noise Computation : Noise indicator (2) Block Selection for Refinement: Fine SI blocks : Noisy SI blocks :

  37. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Other decoding iterations Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  38. Non-Local Means • Images possess repeating elements and patches

  39. Non-Local Means Refinement • NLM algorithm : • Similarity measurement Partially decoded frame PWZ CMLab, CSIE, NTU

  40. Non-Local Means Refinement • Weight assignment : Normalized term : Smoothing parameter h : Partially decoded frame PWZ

  41. Non-Local Means Refinement • Weight assignment : • , Central weight : Assigned the same value as the maximum of the other weights observed in the searching window Partially decoded frame PWZ

  42. Non-Local Means Refinement • Searching for more similar patches… Past decoded frame Partially decoded frame Future decoded frame

  43. Non-Local Means Refinement • Take into account of the temporal similar patches Past decoded frame Partially decoded frame Future decoded frame

  44. Parameter Setting of NLM • Neighborhood size (Patch size) : • Search window size : Past decoded frame Partially decoded frame Future decoded frame

  45. Parameter Setting of NLM • Smoothing parameter : • First iteration : • Afterward iterations : Correlation noise X’B Motion compensated residual frame X’F MC Residue frame CMLab, CSIE, NTU

  46. Proposed Codec Architecture WZ Encoder WZ Decoder X(u,v) Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Other decoding iterations Feedback Channel Y ’(u,v) LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Y(u,v) Decoded WZ Frames WZ Bitstream Probability DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Update Noise Distribution Model : Refined SI coefficient Y’WZ NLM Refinement DCT XDCT Coefficient Value XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  47. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 unrefined SI refined SI CRC Gen CRC Check refined SI Other decoding iterations Feedback Channel LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream Quantization bin DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input q‧Wk (q+1)‧Wk Failed refinement detection Y’WZ NLM Refinement DCT XDCT XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  48. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel Progressive refinement LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT SI quality of 13th frame in Foreman Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT PSNR, dB XWZ WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames # of decoded bands X’P XK Key Frames

  49. Proposed Codec Architecture WZ Encoder WZ Decoder Slepian-Wolf Encoder Slepian-Wolf Decoder CRC-8 CRC Gen CRC Check Feedback Channel Quality gains LDPCA Decoder LDPCA Encoder Buffer Reconstruction IDCT Decoded WZ Frames WZ Bitstream DCT YDCT Uniform Quantizer Correlation Noise Modeling Bitplanes Soft Input Y’WZ NLM Refinement DCT XDCT XWZ Bitrate savings WZ Frames PWZ Candidate Block Selection YWZ Frame Buffer Side Information Creation NLM-SIR X’F H.264/AVC Intra Encoder H.264/AVC Intra Decoder Decoded Key Frames X’P XK Key Frames

  50. Outline • Introduction and Motivation • DVC Architecture Overview • Proposed Side Information Refinement Framework • Experimental Result • Conclusions and Future Work CMLab, CSIE, NTU

More Related