Multimedia Compression (2)

1. Multimedia Compression (2) Mei-Chen Yeh 03/23/2009

2. Review • Entropy • Entropy coding • Huffman coding • Arithmetic coding Lossless!

3. Outline • Revisiting Information theory • Lossy compression • Quantization • Transform coding

4. Encoder Decoder source s=r Lossless s≠r Lossy Performance measures Distortion reconstruction 1 Rate 2 • Lossless coding • Rate • Lossy coding • Rate • Distortion

5. Lossy compression • Goals • Minimize the rate • Keep the distortion small • Tradeoffs between the best of both worlds • Rate-distortion theory • Provides theoretical bounds for lossy compression

6. Distortion (1) • Suppose we send a symbol xi, receive yi • d(xi, yi) ≥ 0 • d(xi, yi) = 0 for xi= yi • Average distortion • D = ∑x∑yp(xi, yi)d(xi, yi) • Distortion measurement?

7. Distortion (2) decibels (分貝) • User feedbacks • Subjective and may be biased • Some popular criteria: • Mean square error (mse) • Average of the absolute difference • Signal-to-noise ratio (SNR) • Peak-signal-to-noise ratio (PSNR)

8. Information theory revisited (1) • Information: reduction in uncertainty #1: predict the outcome of a coin flip #2: predict the outcome of a die roll Next #1: You observe the outcome of a coin flip #2: You observe the outcome of a die roll Which has more uncertainty? #2 #2 Which provides more information?

9. Information theory revisited (2) • Entropy • The average self-information • The average amount of information provided per symbol • The uncertainty an observer has before seeing the symbol • The average number of bits needed to communicate each symbol

10. Conditional entropy • Example: two random variables X and Y X : College major Y : Likes “XBOX” Yes No 0 H(Y|X=CS) =? 1 H(Y|X=Math) =?

11. Conditional entropy • The conditional entropy H(Y|X) H(Y|X) = 0.5H(Y|X=Math)+0.25H(Y|X=CS)+0.25H(Y|X=H.) H(Y|X) = 0.5*1 + 0.25*0 + 0.25*0 = 0.5

12. Joint probability conditional probability

13. Conditional entropy H(X) = 1.5 H(Y) = 1 H(Y|X) = 0.5 H(X|Y) = 1 ≤ • H(X|Y) • The amount of uncertainty remaining about X, given we know the value Y • H(X|Y) vs. H(X)

15. Average mutual information The amount of information that X and Y convey about each other! Mutual information Average mutual information

16. Average mutual information • Properties • 0 ≤ I(X; Y) = I(Y; X) • I(X; Y) ≤ H(X) • I(Y; X) ≤ H(Y)

17. Lossy compression Lower the bit rate R by allowing some acceptable distortion D of the signal

18. Types of lossy data compression Given R, minimize D Given D, minimize R

19. Rate distortion theory Calculates the minimum transmission bit-rate R for a required reconstruction quality The results do not depend on a specific coding method.

20. Rate distortion function With the assumption of statistical independence between distortion and reconstructed signal Represents trade-offs between distortion and rate Definition: The lowest rate at which the source can be encoded while keeping the distortion less than or equal to D* Shannon lower bound

21. Example: Rate distortion function for the Gaussian source No compression system exists that performs outside the gray area! • A zero mean Gaussian pdf with variance σx2 • Distortion: the MSE measure • D = E[(X-Y)2] • The rate distortion function is:

22. For more information: Khalid Sayood. Introduction to Data Compression, 3rd edition, Morgan Kaufmann, 2005.

23. Outline • Revisiting Information theory • Lossy compression • Quantization • Transform coding

24. Quantization • A practical lossy compression technique • The process of representing a large–possibly infinite–set of values with a much smaller set

25. Example Use -10, -9, …,0, 1, …, 10 (21 values) to represent real numbers (infinite values) 2.47 => 2 3.1415926 => 3 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 Loss of information! The reconstruction value 3 could be 2.95, 3.16, 3.05, …

26. Scalar quantization • Inputs and outputs are scalars • Design of a scalar quantizer • Construct intervals (Encoder) • Assign codewords to intervals (Encoder) • Select reconstruction values (Decoder) -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

27. Scalar quantization (cont.) • Input-output map Output Input -∞ ∞ v Decision boundaries Reconstruction levels

28. Example: Quantization for image compression 64 196 8-bit per pixel [0 255] 1-bit per pixel {0, 128, 255} 2-bit per pixel {0, 64, 128, 196, 255} 3-bit per pixel (8 intervals)

29. Quantization problem (1) • Fixed-length coding Given an input pdf fx(x) and the number of levels M in the quantizer, find the decision boundaries {bi} and the reconstruction levels {yi} so as to minimize the distortion.

30. Quantization problem (2) • Variable-length coding Given a distortion constraint find the decision boundaries {bi} and the reconstruction levels {yi}, andbinary codesthat minimize the rate while satisfying the constraint.

31. Vector Quantization SLOW FAST K L How to generate the codebook? Operate on blocks of data The VQ procedure:

32. Why VQ? Samples may be correlated! Example: height and weight of individuals

33. 2-d Vector Quantization Quantization rule Quantization regions The quantization regions are no longer restricted to be rectangles!

34. Codebook Design 3 2.5 2 1.5 y 1 0.5 0 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 x The Linde-Buzo-Gray Algorithm (also known as k-means)

35. LBG Algorithm Training set

37. Problems of LBG No guarantee that the procedure will converge to the optimal solution! Sensitive to initial points Empty-cell

38. Converge to a local optimum

39. The Empty Cell Problem No update in an empty region End up with an output point that is never used

40. More… • LBG has problems when clusters have different • Sizes • Densities • Non-globular shapes

41. Differing Sizes LBG (3 Clusters) Original Points

42. Different Densities LBG (3 Clusters) Original Points

43. Non-globular Shapes Original Points LBG (2 Clusters)

44. Use of LBG for Image Compression Use 4x4 blocks of pixels Codebook size 16 Codebook size 64 Codebook size 256 Codebook size 1024

45. Outline • Revisiting Information theory • Lossy compression • Quantization • Transform coding and the baseline JPEG

46. To compact most of the information into a few elements! Slide credit: Bernd Girod

47. One more example

48. Transform coding Transform Quantization Binary coding • Three steps • Divide a data sequence into blocks of size N and transform each block using a reversible mapping • Quantize the transformed sequence • Encode the quantized values

49. Transforms of interest • Data-dependent • Discrete Karhunen Lòeve transform (KLT) • Data-independent • Discrete cosine transform (DCT) • Sub-band coding • Wavelet transform

50. KLT • Also known as Principal Component Analysis (PCA), or the Hotelling transform • Transforms correlated variables into uncorrelated variables • Basis vectors are eigenvectors of the covariance matrix of the input signal • Achieves optimum energy concentration • Disadvantages • Dependent on signal statistics • Not separable