1 / 21

LATENT SEMANTIC INDEXING

LATENT SEMANTIC INDEXING. Hande Zırtıloğlu Levent Altunyurt. OUTLINE. Problem History What is LSI? What we really search? How LSI Works? Concept Space Application Areas Examples Advantages / Disadvantages. PROBLEM. Conventional IR methods depend on boolean vector space

josh
Download Presentation

LATENT SEMANTIC INDEXING

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. LATENT SEMANTIC INDEXING Hande Zırtıloğlu Levent Altunyurt

  2. OUTLINE • Problem • History • What is LSI? • What we really search? • How LSI Works? • Concept Space • Application Areas • Examples • Advantages / Disadvantages

  3. PROBLEM • Conventional IR methods depend on • boolean • vector space • probabilistic models • Handicap • dependent on term-matching • not efficent for IR • Need to capture the concepts instead of only the words • multiple terms can contribute to similar semantic meanings • synonym, i.e. car and automobile • one term may have various meanings depending on their context • polysemy, ie. apple and computer

  4. IR MODELS

  5. THE IDEAL SEARCH ENGINE • Scope • able to search every document on the Internet • Speed • better hardware and programming • Currency • frequent updates • Recall: • always find every document relevant to our query • Precision: • no irrelevant documents in our result set • Ranking: • most relevant results would come first

  6. What a Simple Search Engine Can Not Differentiate? • Polysemy • monitor workflow, monitor • Synonymy • car, automobile • Singular and plural forms • tree, trees • Word with similar roots • different, differs, differed

  7. HISTORY • Mathematical technique for information filtering • Developed at BellCore Labs, Telcordia • 30 % more effective in filtering relevant documents than the word matching methods • A solutionto “polysemy” and “synonymy” • End of the 1980’s

  8. LOOKING FOR WHAT? • Search • “Paris Hilton” • Really interested in The Hilton Hotel in Paris? • “Tiger Woods” • Searching something about wildlife or the famous golf player? • Simple word matching fails

  9. LSI? • Concepts instead of words • Mathematical model • relates documents and the concepts • Looks for concepts in the documents • Stores them in a concept space • related documents are connected to form a concept space • Do not need an exact match for the query

  10. CONCEPTS

  11. HOW LSI WORKS? • A set of documents • how to determine the similiar ones? • examine the documents • try to find concepts in common • classify the documents • This is how LSI also works. • LSI represents terms and documents in a high-dimensional space allowing relationships between terms and documents to be exploited during searching.

  12. HOW TO OBTAIN A CONCEPT SPACE? • One possible way would be to find canonicalrepresentations of natural language • difficult task to achieve. • Much simpler • use mathematical properties of the termdocumentmatrix, • i.e. determine the concepts by matrix computation.

  13. TERM-DOCUMENT MATRIX • Query:Human-Computer Interaction • Dataset: • c1 Human machine interface for Lab ABC computer application • c2 A survey of user opinion of computersystemresponsetime • c3 The EPS user interface management system • c4 System and humansystem engineering testing of EPS • c5 Relations of user-perceived responsetime to error measurement • m1 The generation of random, binary, unordered trees • m2 The intersection graph of paths in trees • m3 Graphminors IV: Widths of trees and well-quasi-ordering • m4 Graphminors: A survey

  14. CONCEPT SPACE • Makes the semantic comparison possible • Created by using “Matrices” • Try to detect the hidden similarities between documents • Avoid little similarities • Words with similar meanings • occur close to each other • Dimensions: terms • Plots: documents • Each document is a vector • Reduce the dimensions • Singular Value Decomposition

  15. APPLICATION AREAS • Dynamic advertisements put on pages, Google’s AdSense • Improving performance of Search Engines • in ranking pages • Spam filtering for e-mails • Optimizing link profile of your web page • Cross language retrieval • Foreign language translation • Automated essay grading • Modelling of human cognitive function

  16. GOOGLE USES LSI • Increasing its weight in ranking pages • ~ sign before the search term stands for the semantic search • “~phone” • the first link appearing is the page for “Nokia” although page does not contain the word “phone” • “~humor” • retrieved pages contain its synonyms; comedy, jokes, funny • Google AdSense sandbox • check which advertisements google would put on your page

  17. ANOTHER USAGE • Tried on TOEFL exam. • a word is given • the most similar in meaning should be selected from the four words • scored %65 correct

  18. + / - • Improve the efficency of the retrieval process • Decreases the dimensionality of vectors • Good for machine learning algorithms in which high dimensionality is a problem • Dimensions are more semantic • Newly reduced vectors are dense vectors • Saving memory is not guaranteed • Some words have several meanings and it makes the retrieval confusing • Expensive to compute, complexity is high.

More Related