Indexing and Searching

1. Indexing and Searching Modern Information Retrieval by R. Baeza-Yates and B. Ribeiro-Neto Chapter 8

2. Outline • Inverted Files • Other Indices for Text • Sequential Searching • Pattern Matching • Compression

3. Inverted Files • And inverted file (or inverted index) is a word-oriented mechanism for indexing a text collection in order to speed up the searching task. • Structure：vocabulary and occurrences • Block addressing • The text is divided in blocks, and the occurrences point to the blocks • Full inverted indices：exact occurrences

6. Inverted Files • The search algorithm on an inverted index • Vocabulary search • Retrieval of occurrences • Manipulation of occurrences • Construction (split the index into two files) • Posting file：the lists of occurrences are stored contiguously • The vocabulary is stored in lexicographical order and points to its list.

8. Inverted Files • For Large texts • Partial index • Merging two indices consists of merging the sorted vocabularies.

10. Other Indices for Text • Suffix Trees • Suffix Arrays • Signature Files

11. Suffix Trees and Suffix Arrays • Each position in the text is considered as a text suffix • Index points are selected form the text, which point to the beginning of the text positions which will be retrievable

13. Suffix arrays • The main drawbacks of Suffix Array are its costlyconstruction process. • Allow binary searches done by comparing the contents of each pointer. • Supra-indices (for large suffix array)

16. Construction of Suffix Arrays for Large Texts

17. Signature Files • Word-oriented index structures base on hashing • Maps words to bit masks of B bits • Divides the text in blocks of b words each • The mask is obtained by bitwise ORing the signatures of all the words in the text block. • Hash the query to a bit mask W • If W & Bi = W, the text block may contain the word

19. Sequential Searching • Brute Force • Knuth-Morris-Pratt • Boyer-Moore Family • Shift-Or • Suffix Automaton • Backward DAWG matching (BDM) • BNDM

20. Knuth-Morris-Pratt

21. Boyer-Moore Family

22. Shift-Or

23. Suffix Automaton

25. Pattern Matching • Searching allowing errors • Dynamic Programming • Automaton • Regular Expressions and Extended patterns • Pattern Matching Using Indices • Inverted files • Suffix Trees and Suffix Arrays

26. Dynamic Programming

27. Automaton

28. Regular Expressions

29. Pattern Matching Using Indices • Inverted Files • The types of queries such as suffix or substring queries, searching allowing errors and regular expressions, are solved by a sequential search • The restriction is to find approximate matches or regular expressions that span many word.

30. Pattern Matching Using Indices • Suffix Trees • Suffix trees are able to perform complex searches • Word, prefix, suffix, substring, and Range queries • Regular expressions • Unrestricted approximate string matching • Useful in specific areas • Find the longest substring • Find the most common substring of a fixed size

31. Pattern Matching Using Indices • Suffix Arrays • Some patterns can be searched directly in the suffix array without simulation the suffix tree • Word, prefix, suffix, subword search and range search

32. Compression • Compressed text--Huffman coding • Taking words as symbols • Use an alphabet of bytes instead of bits • Compressed indices • Inverted Files • Suffix Trees and Suffix Arrays • Signature Files