Download
1 / 17
170 likes | 347 Views
N-grams to Process Hindi Queries. Aparna Kulkarni Nachal Ramasamy Rashmi Havaldar. Topics. Introduction N-gram strategies Transliteration Methodology Document Preparation Hand Transliteration Number of N-grams & Choice of N Ranking & Retrieval. Summary. Abstract.
E N D
N-grams to Process Hindi Queries Aparna Kulkarni Nachal Ramasamy Rashmi Havaldar
Topics Introduction N-gram strategies Transliteration Methodology Document Preparation Hand Transliteration Number of N-grams & Choice of N Ranking & Retrieval. Summary
Abstract • Retrieval systems based on N-grams have been used as alternatives to word-based systems. • A query that contains misspellings or differences in transliteration can defeat word-based systems. N-gram systems are more resistant to these problems. • N-grams offer a language-independent technique. • We present a retrieval system based on N-grams that uses a collection of Hindi songs. Within this retrieval system, we study the effect of varying N on retrievability. • We Rank the N-grams produced by different values of N and select the top 10 song with higher rank.
Introduction N-grams are consecutive overlapping N-character sequences formed from an input stream. We Extracted N-grams by means of an example with “N = 3” and the string: “salt in the coffee”. There are three N-gram Strategies. Only method (a) is truly language-independent because it avoids the concept of “words.” Hence we conducted our experiments using (a).
Transliteration Transliteration is a process where an input string in some alphabet is converted to a string in another alphabet based on the phonemes in the string in contrast to translation. As the Hindi words can be transliterated in many different ways, the query-to-index mapping may not be accomplished readily and automatically. The Devnagari alphabet used in Hindi can be transliterate in multiple ways to Roman script because of the absence of direct correspondence between phonemes in the two alphabets There does not exist one accepted system of transliteration also users may not be consistent in transliteration. Despite the diversity in transliteration schemes, the transliterations of a single word somewhat resemble each other. some transliterations of the Hindi word for “law” are: “kaanoon”, “kanoon”, “kaanun” and “kanun”.
Methodology The techniques we used in our project are “the various N-gram extraction techniques” and “the choice of N for the N-gram”. The vector space model for document retrieval represents documents as vectors of (term,weight) tuples. Here each “term” is N-gram of the text in the collection. Also stopword removal and stemming need not be done to the term since N-grams is language-independent technique. The weight for each term was calculated as (add) where ‘tfij’ is the term frequency of term ‘i’ in document ‘j’ or the number of times term ‘i’ occurs in document ‘j’, ‘n’ is the total number of documents and ‘dfi’ is the number of documents that contain term ‘i’. we chose to use multiple values of N (Say 3,4 & 5)to create the terms which composed a single vector space
Document Preparation • The document collection consist of titles of 152 Hindi film songs. The “titles” of the songs are indexed into our song database. • The vector-space model is used to represent the song title documents in our collection for our experiments. • Each document (song) in our collection had multiple vector representations. • For each song, we generated N-grams for N = 3, 4, 5. Treating these N-grams as vector terms, we built separate vector representations for each song for each value of N. • We also built a retrieval system based on N-grams for our particular collection. • Users may enter a few words of a desired song as a query. The system responds with a number of songs sorted in increasing order of rank; • The N-gram strategy can be varied easily in order to compare the effect of changing N.
Hand-transliteration • Our system performs very well for hand-transliterated queries. • Most queries will have a non-zero similarity with the target song and also for all values of N • Results are very similar for N = 3, 4, 5. • Example for a Hand-transliteration handled in a query. The result is still accurate even with a garbled query
Choice of N • Our experiments suggest that N = 3 is an acceptable values of N due to size of the document vectors and the time taken to process a query. • Other considerations that may influence the choice of N could be the size of the document vectors and the time taken to process a query. • We found that the size in bytes of the document vectors fell for N = 3, 4, 5 • Query processing times is low as N was increased. • As N increases only fewer songs returns non-zero similarity with the query. Therefore, the number of responses to sort and rank was reduced.
Ranking and Retrieval • After retrieving the correct set of songs closing matching the query for different values of N(3,4 &5), We report the song Id and Rank for each retrieval of N. • To calculate Rank, we specially formulated an equation: Rank={[0.5*Rank(3-gram)]+ [0.2*Rank(4-gram)]+ [0.3*Rank(5-gram)] 3 for a particular song Id. • We take a song Id and look for the rank of the Song in 3-gram,4-gram, 5gram respectively for atleast10 songs in each retrieval. • If all the three ranks are same for a particular song we are calculating the rank based on the above equation. • If for a particular song from 3-gram was not returned in the 4 and 5 gram retrieval ,we are giving a rank of 11 for that song –Id in 4 and 5 as the probability of the song occurring in the retrieval to be 11.
Ranking and Retrieval(Cont’d) • our refined evaluation records the rank at which the correct song was found if the correct song has a non-zero similarity with the query. • In the final list will have a list of 30 Song Id’s and their corresponding Rank. • Now we sort the list and merge to take the top 10 Songs with higher rank. • The higher rank is 1.0 and the least is 10.0 in the list of 10 songs. • The trigrams have a higher weight due to more accurate results, specifically for Hindi language.
Conclusion The retrieval system performed over the Hindi database is novel. N-gram searches retrieved documents fairly accurately despite garbled queries compared to word based. N-gram techniques are language-independent. Therefore, they are well-suited for collections having documents in different languages or multi-lingual documents. Based on our completed studies, we recommend N-grams as a strong alternative to word-based search techniques.
Scope of the Project • Here, we used only the first lines of every song to keep the experiment within manageable bounds. • Our system can be extended easily on procurement of a full-text collection of songs. • Also, the canonical responses from our system may be used as indices into databases of complete songs that use the same transliteration for all song titles.
