Language-Model Based Text-Compression

1. Language-Model Based Text-Compression James Connor Antoine El Daher

2. Compressing with Structure • Compression • Huffman • Arithmetic • Lempel Ziv (LV78 LV77) • Most popular compression tools based on LV77 • Exploiting structure • Our goal: incorporate prior knowledge about the structure of the input sequence

3. Perplexity and Entropy • Compression ratio is bounded by the Entropy of the sequence to be compressed: • A low-perplexity language model is also a low-entropy distribution:

4. Character N-grams • Represent text as an nth order markov chain of characters • Maintain counts of n-grams • Build a library of huffman tables based on these counts

5. Compressing the file • Training • For each bigram in the training set, we keep a map of all the words that can follow it, along with their probabilities. • E.g. “to have”  (“seen”, 0.1), (“been”, 0.1), (UNK, 0.1), etc. • Then for each bigram, we build a Huffman tree.

6. Compressing the File • Compressing: • We go through the input file, using the Huffman trees from the training set to code each word based on the two preceding words. • If the trigram is unknown, we code the UNK token, the revert to a unigram model (also coded using Huffman). • If the unigram is unknown, we use a character level Huffman (trained on the training set) to code it. • Decompression works similarily; we mimic the same behavior

7. Extensions • We have a sliding context window, so that whenever we are compressing a file, words that are seen there have their counts incremented when they enter the window (and decremented when they leave); this allows us to make better use of the local context in terms of trigrams/bigrams, and give more representative weights.

8. Results • Competitive with Gzip