Chapter 4
Download
1 / 15

Chapter 4 - PowerPoint PPT Presentation


  • 114 Views
  • Updated On :

Chapter 4. Variable–Length and Huffman Codes. Unique Decodability. We must always be able to determine where one code word ends and the next one begins. Counterexample: Suppose: s 1 = 0; s 2 = 1; s 3 = 11; s 4 = 00 0011 = s 4 s 3 or s 1 s 1 s 3

Related searches for Chapter 4

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Chapter 4' - ivria


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Chapter 4 l.jpg

Chapter 4

Variable–Length and

Huffman Codes


Unique decodability l.jpg

Unique Decodability

We must always be able to determine where one code word ends and the next one begins. Counterexample:

Suppose: s1 = 0; s2 = 1; s3 = 11; s4 = 00

0011 = s4s3 or s1s1s3

Unique decodability means that any two distinct sequences of symbols (of possibly differing lengths) result in distinct code words.

4.1, 2


Instantaneous codes l.jpg
Instantaneous Codes

1

1

1

s1 = 0

s2 = 10

s3 = 110

s4 = 111

s4

No code word is the prefix of another. By reading a continuous sequence of code words, one can instantaneously determine the end of each code word.

Consider the reverse: s1 = 0; s2 = 01; s3 = 011; s4 = 111

0111……111 is uniquely decodable, but the first symbol cannot be decoded without reading all the way to the end.

0

0

0

decoding tree

s1

s2

s3

4.3


Constructing instantaneous codes l.jpg
Constructing Instantaneous Codes

comma code: s1 = 0 s2 = 10 s3 = 110 s4 = 1110 s5 = 1111

modification: s1 = 00 s2 = 01 s3 =10 s4 = 110 s5 = 111

Decoding tree

0

1

0

1

0

1

s1 = 00

s2 = 01

s3 = 10

0

1

Notice that every code word is located on the leaves

s4 = 110

s5 = 111

4.4


Kraft inequality l.jpg
Kraft Inequality

Theorem: There exists an instantaneous code for S where each symbol s S is encoded in radix r with length |s| if and only if

Proof: () By induction on the height (maximal length path) of the decoding tree, max{|s|: s  S}. For simplicity, pick r = 2 (the binary case). By IH, the leaves of T0, T1 satisfy the Kraft inequality.

Basis: n = 1

Induction: n > 1

Prefixing one symbol at top of tree increases all the lengths by one, so

0

1

0

1

or

0,1

s1

s1

s2

T0

T1

<n

<n

Could use n = 0 here!

4.5


Slide6 l.jpg

Induction: n > 1

Basis: n = 1

0

≤ r 1

……

Same argument for radix r:

0

≤ r1

……

T0

T≤r-1

s1

…………

s≤r

at mostr

at mostr subtrees

IH

so adding at most r of these together gives ≤ 1 

Inequality in the binary case implies that not all internal nodes have degree 2, but if a node has degree 1, then clearly that edge can be removed by contraction.

4.5


Kraft inequality7 l.jpg

Kraft Inequality ()

Construct a code via decoding trees. Number the symbols s1, …, sq so that l1 ≤ … ≤ lq and assume K ≤ 1.

Greedy method: proceed left-to-right, systematically assigning leaves to code words, so that you never pass through or land on a previous one. The only way this method could fail is if it runs out of nodes (tree is over-full), but that would mean K > 1.

Exs:r = 2 1, 3, 3, 3r= 2 1, 2, 3, 3r= 2 1, 2, 2, 3

0

1

0

1

0

1

1

0

1

0

1

0

0

1

0

0

1

½ + ¼ + ¼ + ⅛ > 1

not used

½ + ⅛ + ⅛ + ⅛ < 1

½ + ¼ + ⅛ + ⅛ = 1

4.5


Shortened block codes l.jpg

Shortened Block Codes

0

s1

1

s1

0

0

s2

1

With exactly 2m symbols, we can form a set of code words each of length m:b1 …… bmbi {0,1}. This is a complete binary decoding tree of depth m. With < 2m symbols, we can chop off branches to get modified (shortened) block codes.

1

0

s2

0

1

0

s3

1

1

s3

0

s4

0

1

s4

1

s5

s5

Ex 2

Ex 1

4.6


Mcmillan inequality l.jpg
McMillan Inequality

Idea: Uniquely decodable codes satisfy the same bounds as instantaneous codes.

Theorem: Suppose we have a uniquely decodable code in radix r of lengths of l1 ≤ … ≤ lq . Then their Kraft sum is ≤ 1.

Use a multinomial expansion to see that Nk = the number of ways nl‘s can add up to k, which is the same as the number of different ways n symbols can form a coded message of length k. Because of uniqueness, this must be ≤ rk, the number of codewords.

Conclusion: WLOG we can use only instantaneous codes.

4.7


Average code length l.jpg
Average code length

Our goal is to minimize the average coded length.

If pn > pm then ln ≤ lm. For if pm < pn with lm < ln, then interchanging the encodings for sm and snwe get

So we can assume that if p1 ≥ … ≥ pq then l1 ≤ … ≤ lq, because if pi = pi+1 with li > li+1, we can just switch si and si+1.

new

old

>

4.8


Slide11 l.jpg

Start with S = {s1, …, sq} the source alphabet. And consider B = {0, 1} as our code alphabet (binary). First, observe that lq1 = lq, since the code is instantaneous, s<q cannot be a prefix of sq, so dropping the last symbol from sq (if lq> lq1) won’t hurt. Huffman algorithm: So, we can combine sq1 and sq into a “combo-symbol” (sq1+sq) with probability (pq1+pq) and get a code for the reduced alphabet.

For q = 1, assign s1 = ε . For q > 1, let sq-1 = (sq-1+sq) 0 and sq = (sq-1+sq) 1

Example:

N. B. the case for q = 1 does not produce a valid code.

4.8


Huffman is always of shortest average length l.jpg

Huffman

 Lavg

We know

l1≤ … ≤ lq

trying to show

Huffman is always of shortest average length

Alternative

 L

Example: p1 = 0.7; p2 = p3 = p4 = 0.1

Compare Lavg = 1.5 to log2 q = 2.

Base Case: For q = 2, no shorter code exists.

Induction Step: For q > 2 take anyinstantaneous code for s1, …, sq with minimal average length.

0

1

Assume p1 ≥ … ≥ pq

s1

s2

4.8


Slide13 l.jpg

total height = lq

reduced code

combined symbol

sq1+sq

0

1

Claim that lq1 = lq = lq1, q+ 1 because

So its reduced code will always satisfy:

By IH, L′avg ≤ L′. But more importantly the reduced Huffman code shares the same properties so it also satisfies the same equation L′avg + (pq1 + pq) = Lavg, henceLavg ≤ L.

4.8


Code extensions l.jpg

Code Extensions

Take p1 = ⅔ and p2 = ⅓ Huffman code gives

s1 = 0 s2 = 1 Lavg = 1

Square the symbol alphabet to get:

S2 : s1,1 = s1s1; s1,2 = s1s2; s2,1 = s2s1; s2,2 = s2s2;

p1,1 = 4⁄9p1,2 = 2⁄9p2,1 = 2⁄9p2,2 = 1⁄9

Apply Huffman to S2:

s1,1 = 1; s1,2 = 01; s2,1 = 000; s2,2 = 001

But we are sending two symbols at a time!

4.10


Huffman codes in radix r l.jpg
Huffman Codes in radix r

At each stage down, we merge the last (least probable) r states into 1, reducing the # of states by r 1. Since we end with one state, we must begin with no. of states  1 mod (r  1) . We pad out states with probability 0 to get this. Example: r = 4; k = 3

pads

4.11


ad