1 / 41

Text Indexing

Text Indexing. S. Srinivasa Rao April 19, 2007 [based on slides by Paolo Ferragina]. Why string data are interesting ?. They are ubiquitous: Digital libraries and product catalogues Electronic white and yellow pages Specialized information sources (e.g. Genomic or Patent dbs)

dacey
Download Presentation

Text Indexing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Text Indexing S. Srinivasa Rao April 19, 2007 [based on slides by Paolo Ferragina]

  2. Why string data are interesting ? They are ubiquitous: • Digital libraries and product catalogues • Electronic white and yellow pages • Specialized information sources (e.g. Genomic or Patent dbs) • Web pages repositories • Private information dbs • ... String collections are growing at a staggering rate: • ...more than 10Tb of textual data in the web • ...more than 15Gb of base pairs in the genomic dbs

  3. The need for an “index” Brute-force scanning is not a viable approach: • Fast single searches • Multiple simple searches for complex queries The index is a basic block of any IR system. An IR system also encompasses: • IR models • Ranking algorithms • Query languages and operations • ... We will concentrate only on “index design” !!

  4. Types of data DNA sequences Audio-video files Executables Linguistic or tokenizable text Raw sequence of characters or bytes Exact word Word prefix or suffix Phrase Arbitrary substring Complex matches Word-based query Character-based query Types of query Two families of indexes Two indexing approaches : • Word-based indexes, here a concept of “word” must be devised ! • Inverted files, Signature files or Bitmaps. • Full-text indexes, no constraint on text and queries ! • Suffix Array, Suffix tree, Hybrid indexes, or String B-tree.

  5. Vocabulary Postings Doc #1 Now is the time for all good men to come to the aid of their country Doc #2 It was a dark and stormy night in the country manor. The time was past midnight 2 Word-based: Inverted files (or lists) Query answering is a two-phase process: midnight AND time

  6. Full-text indexes Their need is pervasive: • Raw data: DNA sequences, Audio-Video files, ... • Linguistic texts: data mining, statistics, ... • Vocabulary for Inverted Lists • Xpathqueries on XML documents • Intrusion detection, Anti-viruses, ... Classes of indexes: • Suffix array, Suffix tree (variants) • Multi-level indexes: Short Pat array • B-tree based data structures: Prefix B-tree, String B-tree

  7. Terminology • An alphabet, denoted ∑ is a set of (ordered) characters. • A string S is an array of characters, S[1,n] = S[1] S[2] … S[n]. • S[i,j] = S[i] … S[j] is a substring of S. • S[1,j] is a prefix of S; S[i,n] is a suffix of S. • ∑* denotes all strings over alphabet ∑. • Lexicographic order: • Example: For ∑ = { a, b, c, …, z }, where a < b < c < … < z, the lexicographic order is the same as in a dictionary. SUF(T) = Sorted set of suffixes of T

  8. Indexed string matching problem • Let T be a set of K strings in ∑*, where N is the total length of all strings in T. • String matching query on T: Given a pattern P find all occurrences of P in the strings in T. • Static problem: Store T in a data structure that supports string matching queries. Such a data structure is called a full-text index. • Dynamic version: Supports also insertions and deletions of strings in the full-text index.

  9. P i T T[i,n] • T =This is a visual example • This is a visual example • This is a visual example 3,6,12 A simple but crucial observation Pattern P[1,p] occurs at position i of T[1,n] iff P[1,p] is a prefix of the suffix T[i,n] Occurrences of P in T = All suffixes of T having P as a prefix Can transform the string matching problem to a “prefix matching problem” over all the suffixes.

  10. 5 Q(N2) space T = mississippi# SA SUF(T) 12 11 8 5 2 1 10 9 7 4 6 3 # i# ippi# issippi# ississippi# mississippi# pi# ppi# sippi# sissippi# ssippi# ssissippi# suffix pointer • Suffix Array • SA: array of ints, 4N bytes • Text T: N bytes •  5N bytes of space occupancy Suffix Array [Manber-Myers, 90] Suffix array: an array of pointers to all the suffixes in the text in their lexicographic order. T = mississippi#

  11. T = mississippi# SA SUF(T) 12 11 8 5 2 1 10 9 7 4 6 3 # i# ippi# issippi# ississippi# mississippi# pi# ppi# sippi# sissippi# ssippi# ssissippi# Two key properties [Manber-Myers, 90] Prop 1. All suffixes in SUF(T) having prefix P are contiguous. Prop 2. Starting position is the lexicographic one of P. T = mississippi# P=si

  12. SA 12 11 8 5 2 1 10 9 7 4 6 3 P is larger 2 accesses for binary step si Searching in Suffix Array [Manber-Myers, 90] Indirected binary search on SA: O(p log2 N) time T = mississippi#

  13. SA 12 11 8 5 2 1 10 9 7 4 6 3 P is smaller si Searching in Suffix Array [Manber-Myers, 90] Indirected binary search on SA: O(p log2 N) time T = mississippi#

  14. SA 12 11 8 5 2 1 10 9 7 4 6 3 si • Suffix Array search • O (p (log2 N + occ)) time • O (log2 N + occ) in practice P is a prefix sippi occ=2 sissippi P is a prefix • External memory • Simple disk paging for SA • O ((p/B) (log2 N + occ)) I/Os issippi P is not a prefix + occ/B logB N Listing the occurrences [Manber-Myers, 90] Brute-force comparison: O(p x occ) time T = mississippi# 4 6 7 12 11 8 5 2 1 10 9 7 4 6 3 12 11 8 5 2 1 10 9 7 4 6 3

  15. Lcp[1,n-1] stores the longest-common-prefix between suffixes adjacent in SA SA SUF(T) Lcp 12 11 8 5 2 1 10 9 7 4 6 3 12 11 8 5 2 1 10 9 7 4 6 3 12 11 8 5 2 1 10 9 7 4 6 3 # i# ippi# issippi# ississippi# mississippi# pi# ppi# sippi# sissippi# ssippi# ssissippi# 0 0 1 4 0 0 1 0 2 1 3 • Suffix Array search • O ((p/B) log2 N + (occ/B)) I/Os • 9 N bytes of space P=si occ=2 Scan Lcp until Lcp[i] < p Output-sensitive retrieval T = mississippi# 4 6 7 base B : tricky !! 0 0 1 4 0 0 1 0 2 1 3 0 0 1 4 0 0 1 0 2 1 3 + : incremental search Compare against P

  16. Min Lcp[i,q-1] < P’s > P’s P q Range Minima > P’s known induct. The cost: O (1) memory accesses Incremental search (case 1) Incremental search using the LCP array: no rescanning of pattern chars SA i j

  17. Min Lcp[i,q-1] < P’s > P’s P q Range Minima known induct. The cost: O (1) memory accesses Incremental search (case 2) Incremental search using the LCP array: no rescanning of pattern chars SA i j

  18. Min Lcp[i,q-1] < P’s > P’s • Suffix Array search • O(log2 N) binary steps • O(p) total char-cmp for routing •  O((p/B) + log2 N + (occ/B)) I/Os L Suffix char>Pattern char P Range Minima Suffix char<Pattern char The cost: O(L) char cmp Incremental search (case 3) Incremental search using the LCP array: no rescanning of pattern chars SA i q j base B : more tricky (we will not consider this)

  19. Summary: suffix array [Manber-Myers, 90] • Space: O(N) • String matching queries: O(p + log N + occ) • Can be constructed O(N log N) time. • Static.

  20. P M SA Copy a prefix of marked suffixes • SA + Supra-index • O((p/B) + log2 (N/s) + (occ/B)) I/Os binary-search inside s Hybrid Index: Short Pat array Exploit internal memory: sample the suffix array and copy something in memory Disk • Parameter s depends on M and influences both performance and space !! (only a huristic)

  21. s b e u u i a n d r l d $ $ l a $ k y $ $ $ Tries • Trie (name from the word “retrieval”): a data structure for storing a set of strings • Let’s assume that all strings end with “$” (not in S) Set of strings: {bear, bid, bulk, bull, sun, sunday}

  22. Tries • Properties of a trie: • A multi-way tree. • Each node has from 1 to S+1 children. • Each edge of the tree is labeled with a character. • Each leaf node corresponds to the stored string, which is a concatenation of characters on a path from the root to this node.

  23. Analysis of the Trie Given k strings of total length N: • Size: • O(N) in the worst-case • Search, insertion, and deletion (string of length m): • O(m) (assuming ∑ is constant) • Observation: • Having chains of one-child nodes is wasteful

  24. s b e u u i a n d r l d $ $ l b a $ sun k y ear$ $ $ day$ ul id$ $ $ l$ k$ Compact Tries • Compact Trie: • Replace a chain of one-child nodes with an edge labeled with a string • Each non-leaf node (except root) has at least two children

  25. 1,1 20,22 2,5 b 27,30 11,12 sun 7,9 5,5 ear$ day$ ul 18,19 id$ $ 13,14 l$ k$ 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 T: bear$bid$bulk$bull$sun$sunday$ Compact Tries • Implementation: • Strings are external to the structure in one array, edges are labeled with indices in the array (from, to) • Improves the space to O(k)

  26. b,1 1,1 s,3 20,22 e,4 2,5 d,4 u,2 27,30 11,12 i,3 7,9 _,1 5,5 l,2 18,19 k,2 13,14 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 T: bear bid bulk bull sun sunday Patricia Tries • Patricia trie: • a compact trie where each edge’s label (from, to) is replaced by (T[from], to – from + 1)

  27. r a (a,1) (r,1) carrara$ (c,8) (r,1) r rara$ (r,5) (a,1) a $ ($,1) a$ 1 1 rara$ (a,2) (r,5) $ (r,3) 3 3 ra$ 7 7 ($,1) 5 5 4 2 2 4 6 6 carrara$ Suffix Trees [McCreight, 76] • Suffix tree – a compact trie (or similar structure) of all suffixes of the text • Patricia trie of suffixes is sometimes called a Pat tree 1 2 3 4 5 6 7 8

  28. 3 1 0 4 2 a b c b b O(p) time a a b b O(N) space a c (5,8) a b b b c c 2 4 1 3 5 7 6 8 4 2 W(p) I/Os W(occ) I/Os?? Packing ?! CPAT tree ~ 5N on average No (p/B), possibly no (occ/B), mainly static and space costly Search in suffix trees P = ba  Search is a path traversal and O(occ) time a b c c b b b c c b What about ST in external memory ? • Unbalanced tree topology • Updates T = abababbc# 1 3 5 79 - Large space ~ 15N

  29. Summary: compact trie K strings of total length N: • Space: O(K) • String matching queries: O(p + occ) time • Can be constructed O(N) time. • Update(S): O(|S|) time

  30. Summary: suffix tree [McCreight, 76] For a string of length n: • Space: O(n) • String matching queries: O(p + occ) • Can be constructed O(n) time. • Static.

  31. The String B-tree(An I/O-efficient full-text index !!)[Ferragina-Grossi, 95]

  32. String B-tree[Ferragina-Grossi, 95] • Index unbounded length keys • Good worst-case I/O-bounds in search and update The prologue We are left with many open issues: • Suffix Array: updates • Hybrid: Heuristic tuning of the performance • Suffix tree: difficult packing and W(p) I/Os B-tree is ubiquitous in large-scale applications: • Atomic keys: integers, reals, ... • Prefix B-tree: bounded length keys ( 255 chars) Suffix trees + B-trees ?

  33. Some considerations Strings have arbitrary length: • Disk page cannot ensure the storage of Q(B) strings • M may be unable to store even one single string String storage: • Pointers allow to fit Q(B) strings per disk page • String comparison needs disk access and may be expensive String pointers organization seen so far: • Suffix array: simple but static and not optimal • Patricia trie: sophisticated and much efficient (optimal ?) Recall the problem: T is a string collection • Search( P[1,p] ): retrieve all occurrences of P in T’s strings • Update( S[1,t] ): insert or delete a text S from T

  34. + B • Search(P) • O ((p/B) log2 N) I/Os • O (occ/B) I/Os It is dynamic !! O(s (s/B) log2 N) I/Os O((p/B) log2 B) I/Os 29 13 20 18 3 23 O(logB N) levels 29 2 26 13 20 25 6 18 3 14 21 23 29 1 9 5 2 26 10 4 7 13 20 16 28 8 25 6 12 15 22 18 3 27 24 11 14 21 17 23 Disk AATCAGCGAATGCTGCTT CTGTTGATGA 1 3 5 7 9 11 13 15 17 19 20 22 24 26 28 30 1º step: B-tree on string pointers P = AT

  35. G A 0 4 6 3 5 6 G A 2 3 1 7 4 5 6 A G A A G A A G A A G G A G A C G C G C A G A G C G C A G G G C G C G G A G C G C G G G A 2º step: The Patricia trie (1; 1,3) (4; 1,4) (2; 1,2) (5; 5,6) (3; 4,4) (6; 5,6) (2; 6,6) (1; 6,6) (5; 7,7) (4; 7,7) (7; 7,8) (6; 7,7) Disk

  36. G • Space PT • O(k) , not O(N) C G 6 4 6 5 3 0 • Search(P): • First phase: no string access 1 5 7 G G G A A max LCP with P 6 2 5 3 1 4 7 A G A A G A A G A A G G A G A C G C G C A G A G C G C A G G G C G C G G A G C G C G G G A mismatch P’s position 2º step: The Patricia trie A Two-phase search: P = GCACGCAC • Second phase: O(p/B) I/Os A C A Just one string is checked !! A G A G G Disk

  37. + • Search(P) • O((p/B) logB N) I/Os • O(occ/B) I/Os Insert(S) O( s (s/B) logB N) I/Os O(p/B) I/Os O(logB N) levels PT PT PT PT PT PT PT PT PT PT 29 1 9 5 2 26 10 4 7 13 20 16 28 8 25 6 12 15 22 18 3 27 24 11 14 29 2 26 13 20 25 6 18 3 14 21 23 21 17 23 Disk AATCAGCGAATGCTGCTT CTGTTGATGA 1 3 5 7 9 11 13 15 17 19 20 22 24 26 28 30 3º step: B-tree + Patricia tree P = AT 29 13 20 18 3 23

  38. Search(P) • O(logB N) I/Os just • to go to the leaf level PT Level i Max_lcp(i) P Level i+1 PT PT Leaf Level adjacent 4º step: Incremental Search First case

  39. Search(P) • O(p/B + logB N) I/Os • O(occ/B) I/Os PT Level i Max_lcp(i) Inductive Step P PT Level i+1 i-th step: O(( lcp i+1 – lcp i)/B + 1) I/Os skip Max_lcp(i) Max_lcp(i+1) 4º step: Incremental Search Second case No rescanning

  40. Summary: String B-tree String B-tree performance: [Ferragina-Grossi, 95] • Search(P) takes O(p/B + logB N + occ/B) I/Os • Update(S) takes O( s logB N ) I/Os • Space is Q(N/B) disk pages Using the String B-tree in internal memory: • Search(P) takes O(p + log2 N + occ) time • Update(S) takes O( s log2 N ) time • Space is Q(N) bytes • It is a sort of dynamic suffix array

  41. Summary • Indexed string matching problem • Word-based • Full-text • Internal memory data structures • Suffix array • Suffix tree • External memory data structures • Patricia trie and Pat tree • Short Pat array • String B-tree

More Related