1 / 44

Contextual Image Search

Contextual Image Search. Wenhao Lu , Jingdong Wang , Xian- Sheng Hua , Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,. MM 2011. Traditional image search. MM 2011. Contextual image search. company. iPhone.

ailani
Download Presentation

Contextual Image Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Contextual Image Search Wenhao Lu , Jingdong Wang , Xian-ShengHua, Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China, MM 2011

  2. Traditional image search MM 2011

  3. Contextual image search company iPhone MM 2011

  4. Commercial image search engine query: Funny George Bush MM 2011

  5. System overview Text input MM 2011

  6. System overview Image input 2. Annotating images by mining search result(2008) MM 2011

  7. Image input example Candidate queries: “Blue mosque”, “Istanbul”, “Turkey travel”, “Istanbul turkey” The mosque is one of several mosques known as the Blue Mosque for the blue tiles adorning the walls of its interior Input image Similar image Search Result MM 2011

  8. Contextual Image Search With Text Input 1. Context Capturing • textual contexts: page title / document title • local context • visual contexts: vision-based page segmentation algorithm • (VIPS) MM 2011

  9. Contextual Image Search With Text Input 2. Contextual Query Augmentation • Goal: remove possible ambiguities • Augmented query = query + textual context Candidate augmented query evaluate the relevance between the context and augmented query (Okapi BM25) MM 2011

  10. Contextual Image Search With Text Input 3. Image Search by Text: Microsoft Bing image search 4.Contextual Reranking: Combine textually and visually context MM 2011

  11. Quantitative Evaluation 0.95 0.65 nDCG curves MM 2011

  12. YouPivot: Improving Recall with Contextual Search Joshua Hailpern 1, Nicholas Jitkoff 2, Andrew Warr 2 Karrie Karahalios 1,Robert Sesek 3, Nikita Shkrob 4 1 University of Illinois Urbana, IL USA 61801 2 Google Mountain View, CA USA 94043 3 Boston University Boston, MA USA 02215 4 University of Waterloo Waterloo, Ontario, Canada N2L 3G1 SIGCHI 2011

  13. Why Contextual Search? “what was that website I was looking at when Yesterday by The Beatles was last playing? ” SIGCHI 2011

  14. Improving Recall ? • Contextual cues: temporally related activities • Cognitive science: • leveraging context improves speed and • accuracy in recall tasks • Loses car keys: • “retrace your steps since the last time you know you had them.” SIGCHI 2011

  15. Interface SIGCHI 2011

  16. Features and Functionality Sarah Where is the website? ?title ?domain ?when Mr. Richfield graphic designer SIGCHI 2011

  17. Features and Functionality Websites Layout SIGCHI 2011

  18. Context Sensitive Paraphrasing with a Single Unsupervised Classifier Michael Connor and Dan Roth Department of Computer Science University of Illinois at Urbana-Champaign ECML2007

  19. Context Sensitive Paraphrasing ‘X commanded Y’ ‘X spoke to Y’ • When can ‘speak to’ replace ‘command’ • in the original sentence and not change the • meaning of the sentence? ECML2007

  20. Polysemous Nature of Verbs - command ECML2007

  21. Definition of Context • derived from parsing information • subject and objectof the verb Marshall Formby of Plainview suggested a plan to fill by appointment future vacancies in the Legislature and Congress, eliminating the need for special elections. Local Context: obj:plan , subj:NE:PER ECML2007

  22. Modeling Context Sensitive Paraphrasing obj:plan , subj:NE:PER 1. Context Sensitive Decisions v: original verb u: substitute verb creating, breaking or presenting type c contextual features of v/u c: sub / obj ECML2007

  23. Unsupervised Training: Bootstrapping Local Classifiers ECML2007

  24. Experimental Results • AQUAINT Corpus (News Articles) • test set has 721 S, v, u examples with 57 unique v verbs and 162 unique u. (random selection of polysemous verbs that occur in WordNet 2.1) ECML2007

  25. Experimental Results ECML2007

  26. Hierarchical summarization for delivering information to mobile devices Jahna Otterbacher a,*, Dragomir Radev b, Omer Kareem b a University of Cyprus, Nicosia, Cyprus b University of Michigan, Ann Arbor, MI 48109, United States SIGIR 2007

  27. Recent News • 17歲天才寫APP　幫新聞摘要精華(2012/11/3) • 一名英國的17歲男生，設計出一款可以幫新聞摘要出「精華版」 • 的APP-Summly，這款應用程式在業界廣受好評。 SIGIR 2007

  28. Limitation of Mobile Device • small screens • constrained wireless bandwidth SIGIR 2007

  29. Architecture of summarization method SIGIR 2007

  30. Sentence scoring • Centroid value: the importance of the sentence :TF*IDF values of word w in • Positional value: More weight is given to sentences that appear • earlier in the document than those that appear later.(news articles) :first sentence centroid value SIGIR 2007

  31. Sentence scoring • First sentence overlap value: The first sentence in a text is likely • to convey information about its main theme or topic. SIGIR 2007

  32. Hierarchical nesting SIGIR 2007

  33. Experiment ─ 39 subjects in the experiment(student studying information and computer science) ─10 articles 10 questions SIGIR 2007

  34. A Maximum Entropy Web Recommendation System: Combining Collaborative and Content Features Xin Jin, Yanzan Zhou, Bamshad Mobasher Center for Web Intelligence School of Computer Science, Telecommunication, and Information Systems DePaul University, Chicago, Illinois, USA SIGKDD 2005

  35. Web Recommendation System SIGKDD 2005

  36. About Web Recommendation ─ Goal: help users locate information on the Web ─ Input: • Web users’ navigation or rating data • contentfeatures of the items ─ Approach: Data mining or Machine Learning todiscover usage patternsthat represent aggregate user models. SIGKDD 2005

  37. Maximum Entropy Recommendation Model ─ Maximum Entropy: P(飛行 | fly) + (P(搭機 | fly) + P(蒼蠅 | fly) = 1 s.t. P(飛行 | fly) + (P(搭機 | fly) = 4/5 P(蒼蠅 | fly)=1/5 SIGKDD 2005

  38. Maximum Entropy Recommendation Model ─ Offline: 1.accept constraints to form the model 2.estimate the model parameters ─ Online: 1.reads an active session 2.runs the recommendation algorithm SIGKDD 2005

  39. Maximum Entropy Recommendation Model ─ Distribution form: : a page being visited next : user’s recent navigational history : weight of = SIGKDD 2005

  40. Maximum Entropy Recommendation Model ─ identification of features (navigation data): if ( ) SIGKDD 2005

  41. Maximum Entropy Recommendation Model ─ identification of features (rating data): select highly correlated item pairs SIGKDD 2005

  42. Maximum Entropy Recommendation Model ─ identification of features (content information): In a movie site, high ratings for “Indiana Jones” and “Air Force One” may suggest that a user is a Harrison Ford’s fan and enjoys Action-Adventure movies. Use Latent Dirichlet Allocation(LDA) to find the class of each item. SIGKDD 2005

  43. Maximum Entropy Recommendation Model ─ identification of features (content information): SIGKDD 2005

  44. Experiment ─Realty data,24,000 user sessions from 3,800 unique users SIGKDD 2005

More Related