Download
1 / 19
190 likes | 196 Views
This text discusses the process of tokenization and indexing in information retrieval, including issues such as document parsing, linguistic modules, token streams, and inverted indexing. It also covers topics like stemming, case folding, and the use of thesauri in search engines.
E N D
Information Retrieval Document Parsing
Tokenizer Friends Romans Countrymen Token stream. Linguistic modules friend roman countryman Modified tokens. Indexer Inverted index. Basic indexing pipeline Documents to be indexed. Friends, Romans, countrymen.
Parsing a document • What format is it in? • pdf/word/excel/html? • What language is it in? • What character set is in use? • Plain ASCII, UTF-8, UTF-16,… Each of these is a classification problem, with many complications…
Katakana Hiragana Kanji “Romaji” Tokenization: Issues • Chinese/Japanese no spaces between words: • Not always guaranteed a unique tokenization • Dates/amounts in multiple formats フォーチュン500社は情報不足のため時間あた$500K(約6,000万円) What about DNA sequences ? ACCCGGTACGCAC... Definition of Tokens What you can search !!
Is this the German “mit”? Morgen will ich in MIT … Case folding • Reduce all letters to lower case • exception: upper case (in mid-sentence?) • e.g.,General Motors • USA vs. usa
Stemming • Reduce terms to their “roots” • language dependent • e.g., automate(s), automatic, automationall reduced toautomat. • e.g., casa, casalinga, casata, casamatta, casolare, casamento, casale, rincasare, case reduced tocas
sses ss, ies i, ational ate, tional tion Porter’s algorithm • Commonest algorithm for stemming English • Conventions + 5 phases of reductions • phases applied sequentially • each phase consists of a set of commands • sample convention: Of the rules in a compound command, select the one that applies to the longest suffix. • Full morphologial analysis modest benefit !!
Thesauri • Handle synonyms and homonyms • Hand-constructed equivalence classes • e.g., car = automobile • e.g., macchina = automobile = spider • List of words important for a given domain • For each word it specifies a list of correlated words (usually, synonyms, polysemic or phrases for complex concepts). • Co-occurrence Pattern: BT (broader term), NT (narrower term) • Vehicle (BT) Car Fiat 500 (NT) How to use it in SE ??
Information Retrieval Statistical Properties of Documents
Statistical properties of texts • Token are not distributed uniformly • They follow the so called “Zipf Law” • Few tokens are very frequent • A middle sized set has medium frequency • Many are rare • The first 100 tokens sum up to 50% of the text • Many of these tokens are stopwords
f = c |T| / r a = 1.52.0 a Sum after the k-th element is ≤ fkk/(z-1) The Zipf Law, in detail • K-th most frequent term has frequency approximately 1/k; or the product of the frequency (f) of a token and its rank (r) is almost a constant r * f = c |T| f = c |T| / r General Law For the initial top-elements is a constant
Consequences of Zipf Law • Do exist many not frequent tokens that do not discriminate. These are the so called “stop words” • English: to, from, on, and, the, ... • Italian: a, per, il, in, un,… • Do exist many tokens that occur once in a text and thus are poor to discriminate (error?). • English: Calpurnia • Italian: Precipitevolissimevolmente (o, paklo) Words with medium frequency Words that discriminate
Other statistical properties of texts • The number of distinct tokens grows as • The so called “Heaps Law” (|T|b where b<1) • Hence the token length is (log |T|) • Interesting words are the ones with • Medium frequency (Luhn)
