Text Operations: Preprocessing and Compression

1. Text Operations:Preprocessing and Compression

2. Introduction • Document preprocessing • to improve the precision of documents retrieved • lexical analysis, stopwords elimination, stemming, index term selection, thesauri • build a thesaurus • Text compression • to improve the efficiency of the retrieval process • statistical methods vs. dictionary methods • inverted file compression

3. Document Preprocessing • Lexical analysis of the text • digits, hyphens, punctuation marks, the case of letters • Elimination of stopwords • filtering out the useless words for retrieval purposes • Stemming • dealing with the syntactic variations of query terms • Index terms selection • determining the terms to be used as index terms • Thesauri • the expansion of the original query with related term

4. structure Full text Index terms The Process of Preprocessing Accents spacing Noun groups Manual indexing Docs stopwords stemming structure

5. Lexical Analysis of the Text • Four particular cases Numbers • usually not good index terms because of their vagueness • need some advanced lexical analysis procedure • ex) 510B.C. , 4105-1201-2310-2213, 2000/2/12, …. Hyphens • breaking up hyphenated words might be useful • ex) state-of-the-art  state of the art (Good!) • but, B-49  B 49 (???) • need to adopt a general rule and to specify exceptions on a case by case basis

6. Lexical Analysis of the Text • Punctuation marks • removed entirely • ex) 510B.C 􀃆 510BC • if the query contains ‘510B.C’, removal of the dot both in query term and in the documents will not affect retrieval performance • require the preparation of a list of exceptions • ex) val.id 􀃆 valid (???) • The case of letters • converts all the text to either lower or upper case • part of the semantics might be lost • Northwestern University 􀃆 northwestern university (???)

7. Elimination of Stopwords • Basic concept • filtering out words with very low discrimination values • ex) a, the, this, that, where, when, …. • Advantage • reduce the size of the indexing structure considerably • Disadvantage • might reduce recall as well • ex) to be or not to be

8. Stemming • What is the “stem”? • the portion of a word which is left after the removal of its affixes (i.e., prefixes and suffixes) • ex) ‘connect’ is the stem for the variants ‘connected’, ‘connecting’, ‘connection’, ‘connections’ • Effect of stemming • reduce variants of the same root to a common concept • reduce the size of the indexing structure • controversy about the benefits of stemming

9. Index Term Selection • Index terms selection • not all words are equally significant for representing the semantics of a document Manual selection • selection of index terms is usually done by specialist Automatic selection of index terms • most of the semantics is carried by the noun words • clustering nouns which appear nearby in the text into a single indexing component (or concept) • ex) computer science

10. Thesauri • What is the “thesaurus”? • list of important words in a given domain of knowledge • a set of related words derived from a synonymity relationship • a controlled vocabularyfor the indexing and searching • Main purposes • provide a standard vocabulary for indexing and searching • assist users with locating terms for proper query formulation • provide classified hierarchies that allow the broadening and narrowing of the current query request

11. Thesauri • Thesaurus index terms • denote a concept which is the basic semantic unit • can be individual words, groups of words, or phrases • ex) building, teaching, ballistic missiles, body temperature • frequently, it is necessary to complement a thesaurus entry with a definition or an explanation • ex) seal (marine animals), seal (documents) • Thesaurus term relationships • mostly composed of synonyms and near-synonyms • BT (Broader Term), NT (Narrower Term), RT (Related Term)

12. Text Operations:Coding / Compression Methods

13. Text Compression • Motivation • finding ways to represent the text in fewer bits • reducing costs associated with space requirements, I/O overhead, communication delays • obstacle: need for IR systems to access text randomly • to access a given word in some forms of compressed text, the entire text must be decoded from the beginning until the desired word is reached • Two strategies • statistical methods • dictionary methods

14. Statistical Methods • Basic concepts • Modeling: a probability is estimated for each symbol • Coding: a code is assigned to each symbol based on the model • shorter codes are assigned to the most likely symbols • Relationship between probabilities and codes • Source code theorem (by Claude Shannon) : • a symbol that occurs with probability p should be assigned a code of length log2 (1/p) bits

15. Statistical Methods • Compression models • adaptive model: progressively learn about the statistical distribution as the compression process goes on • decompression of a file has to start from its beginning • static model: assume an average distribution for all input texts • poor compression ratios when data deviates from initial distribution assumptions • semi-static model: learn a distribution in a first pass, compress the data in a second pass by using a fixed code derived from the distribution learned • information on the data distribution must be stored

16. Statistical Methods • Word-based compression model • take words instead of characters as symbols. • Reasons to use this model in an IR context • much better compression rates • words carry a lot of meaning in natural languages and their distribution is much more related to the semantic structure of the text than is the distribution of individual letters • words are the atoms on which most IR systems are built • word frequencies are useful in answering queries involving combinations of words • the best strategy is to start with the least frequent words first

17. Statistical Methods • Coding • the task of obtaining the representation of a symbol based on a probability distribution given by a model • main goal: assign short codes to likely symbols and long codes to unlikely ones • Two statistical coding strategies • Huffman coding • a variable-length encoding in bits for each symbol • relatively fast, allows random access • Arithmetic coding • use an interval of real numbers between 0-1 • much slower, does not allow random access

18. Huffman Coding • Building a Huffman tree • for each symbol of the alphabet, create a node containing the symbol and its probability • the two nodes with the smallest probabilities become children of a newly created parent node • the parent node is associated a probability equal to the sum of the probabilities of the two chosen children • the operation is repeated, ignoring nodes that are already children, until there is only one node

19. Author’s Huffman Coding • Example: “for each rose, a rose is a rose” 0 1 0 1 rose a 0 1 0 1 0 1 each “, “ for is for each rose, a rose is a rose 0110 0100 1 0101 00 1 0111 00 1

20. Better Huffman Coding? • Example: “for each rose, a rose is a rose” 0 1 0 1 0 1 rose a 0 1 0 1 each “, “ for is for each rose, a rose is a rose 010 000 11 001 10 11 011 10 11

21. 0 1 0 1 rose 0 1 a 0 1 0 1 each “, “ for is Author’s Canonical Huffman Coding • Height of left tree is never shorter than right tree. • S: ordered sequence of pairs (xi, yi) for each level in tree where xi = # symbols yi = numerical value of first symbol S = ((1, 1) (1, 1) (0, ∞) (4, 0) for each rose, a rose is a rose 0010 0000 1 0001 00 1 0011 00 1

22. Byte-Oriented Huffman Coding • Tree has branching factor of 256 • Ensure no empty nodes in higher levels of tree # of bottom level elements = 1 + ((v – 256) mod 255) • Characteristics • Decompression is faster than for plain Huffman coding. • Compression ratios are better than for Ziv-Lempel family of codings. • Allows direct searching on compressed text

23. Dictionary Methods • Basic concepts • replacing groups of consecutive symbols with a pointer to an entry in a dictionary • the pointer representations are references to entries in a dictionary composed of a list of symbols that are expected to occur frequently • pointers to the dictionary entries are chosen so that they need less space than the phrase they replace • modeling and coding does not exist • there are no explicit probabilities associated to phrases

24. Dictionary Methods • Static dictionary methods • selected pairs of letters are replaced with codewords • ex) Digram coding • at each step the next two characters are inspected and verified if they correspond to a digram in the dictionary • if so, they are coded together and the coding position is shifted by two characters; otherwise, the single character is represented by its normal code and the position is shifted by one character • main problem • the dictionary might be suitable for one text, but unsuitable for another

25. Dictionary Methods • Adaptive dictionary methods • Ziv-Lempel • placing strings of characters with a reference to a previous occurrence of the string • if the pointer to an earlier occurrence of a string is stored in fewer bits than the string it replaces, compression is achieved

26. Ziv-Lempel Code • Characteristics • identifying each text segment the first time it appears and then simply pointing back to this first occurrence rather than repeating the segment • an adaptive model of coding, with increasingly long text segments encoded as the text is scanned • require less space than the repeated text segments • higher compression than the Huffman codes • codes of roughly 4 bits per character

27. Ziv-Lempel Code • LZ77 – Gzip encoding • the code consists of a set of triples <a,b,c> • a: identifies how far back in the decoded text to look for the upcoming text segment • b: tells how many characters to copy for the upcoming segment • c: a new character to add to complete the next segment • ex) <8,2,r>, <0,0,p>

28. Ziv-Lempel Code • An example (decoding) <0,0,p> p <0,0,e> pe <0,0,t> pet <2,1,r> peter <0,0,_> peter_ <6,1,i> peter_pi <8,2,r> peter_piper <6,3,c> peter_piper_pic <0,0,k> peter_piper_pick <7,1,d> peter_piper_picked ………..

29. Dictionary Methods • Adaptive dictionary methods • Disadvantages over the Huffman method • no random access: does not allow decoding to start in the middle of a compression file • Dictionary schemes are popular for their speed and low memory use, but statistical methods are more common in an IR environment

30. Comparing Text Compression Techniques