1 / 78

Content-Based Image Retrieval: Reading One’s Mind and Making People Share

Content-Based Image Retrieval: Reading One’s Mind and Making People Share. Oral defense by Sia Ka Cheung Supervisor: Prof. Irwin King 31 July 2003 . Flow of Presentation. Content-Based Image Retrieval Reading One’s Mind Relevance Feedback Based on Parameter Estimation of Target Distribution

lynelle
Download Presentation

Content-Based Image Retrieval: Reading One’s Mind and Making People Share

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Content-Based Image Retrieval: Reading One’s Mind and Making People Share Oral defense by Sia Ka Cheung Supervisor: Prof. Irwin King 31 July 2003

  2. Flow of Presentation • Content-Based Image Retrieval • Reading One’s Mind • Relevance Feedback Based on Parameter Estimation of Target Distribution • Making People Share • P2P Information Retrieval • DIStributed COntent-based Visual Information Retrieval

  3. Content-Based Image Retrieval • How to represent and retrieve images? • By annotation (manual) • Text retrieval • Semantic level (good for picture with people, architectures) • By the content (automatic) • Color, texture, shape • Vague description of picture (good for pictures of scenery and with pattern and texture)

  4. R G B Feature Extraction

  5. Indexing and Retrieval • Images are represented as high dimensional data points (feature vector) • Similar images are “close” in the feature vector space • Euclidean distance is used

  6. DatabaseIndex and Storage Feature Extraction Lookup Query Image Query Result Typical Flow of CBIR Images

  7. Reading One’s Mind Relevance Feedback

  8. Images Feedback Feedback Query Image Better Result Better Result Why Relevance Feedback? • The gap between semantic meaning and low-level feature  the retrieved results are not good enough DatabaseIndex and Storage Feature Extraction Lookup Result

  9. 1st iteration Display UserFeedback Feedbackto system Estimation & Display selection 2nd iteration Display UserFeedback

  10. Problem Statement • Assumption: images of the same semantic meaning/category form a cluster in feature vector space • Given a set of positive examples, learn user’s preference and find better result in the next iteration

  11. Former Approaches • Multimedia Analysis and Retrieval System (MARS) • IEEE Trans CSVT 1998 • Weight updating, modification of distance function • Pic-Hunter • IEEE Trans IP 2000 • Probability based, updated by Bayes’ rule • Maximum Entropy Display

  12. Comparisons

  13. Data points selected as relevant Estimation of Target Distribution • Assume the user’s target follows a Gaussian distribution • Construct a distribution that best fits the relevant data points into some “specific” region

  14. Data points selected as relevant Estimation of Target Distribution • Assume the user’s target follows a Gaussian distribution • Construct a distribution that best fits the relevant data points into some “specific” region

  15. Data points selected as relevant Estimation of Target Distribution • Assume the user’s target follows a Gaussian distribution • Construct a distribution that best fits the relevant data points into some “specific” region

  16. Expectation Function • Best fit the relevant data points to medium likelihood region • The estimated distribution represents user’s target

  17. Updating Parameters • After each feedback loop, parameters are updated • New estimated mean = mean of relevant data points • New estimated variance  found by differentiation • Iterative approach

  18. Display Selection • Why maximum entropy principle? • K-NN is not a good way to learn user’s preference • The novelty of result set is increased, thus allowing user to browse more from the DB • How to use maximum entropy? • PicHunter – Select a subset of images which entropy is maximized. • Our approach – data points inside boundary region (medium likelihood) are selected

  19. Querytargetclustercenter Selectedby knnsearch Selectedby Max.Entropy Simulating Maximum Entropy Display • Data points around the region of 1.18 δ away from μ are selected • Why 1.18? • 2P(μ+1.18 δ)=P(μ) P(μ) P(μ+1.18 δ)

  20. Experiments • Synthetic data forming mixture of Gaussians are generated • Feedbacks are generated based on ground truth (class membership of synthetic data) • Investigation • Does the estimated parameters converge? • Does it performs better?

  21. Convergence of Estimated Parameters • More feedbacks are given, estimated parameters converge to original parameters used to generate mixtures

  22. Precision-Recall • Red – PE • Blue – MARS • More experiments in later section

  23. Precision-Recall

  24. Problems • What if user’s target distribution forms several cluster? • Indicated in Qcluster (SIGMOD’03) • Parameters estimation failed because single cluster is the assumption • Qcluster solve it by using multi-points query • Merge different clusters into one cluster !!

  25. The Use of Inter-Query Feedback • Relevance feedback information given by users in each query process often infer a similar semantic meaning (images under the same category) • Feature vector space can be re-organized • Relevant images are moved towards to the estimated target • Similar images no longer span on different clusters • Parameters estimation method can be improved

  26. 1st Stage of SOM Training • Large number of data points •  SOM is used to reduce data size •  Each neuron represent a group of similar images •  original feature space is not changed directly

  27. Procedure of Inter-query Feedback Updating • User marked a set of images as relevant or non-relevant in a particular retrieval process • The corresponding relevant neurons are moved towards estimated target • Where • M’R – set of relevant neurons • c – estimated target • αR – learning rate • The corresponding non-relevant neurons are moved away from estimated target

  28. SOM-based Approach Neuron Class 1 Neuron Class 2 Neuron Class 3

  29. SOM-based Approach • After each query process Relevant Neuron Non- Relevant Neuron

  30. SOM-based Approach Estimated Target

  31. SOM-based Approach • Relevant neurons are moved towards estimated target

  32. SOM-based Approach

  33. SOM-based Approach • Feature vector space re-organized

  34. SOM-based Approach • After several iterations (users’ queries)

  35. SOM-based Approach

  36. SOM-based Approach • Similar images cluster together instead of spanning across different clusters in the new, re-organized feature vector space

  37. Experiments • Real data from Corel image collection • 4000 images from 40 different categories • Feature extraction methods • RGB color moment (9-d) • Grey scale cooccurence matrix (20-d) • 80 queries are generated evenly among 40 classes • Evaluations • MARS • PE without SOM-based inter-query feedback training • PE with SOM-based inter-query feedback training

  38. Precision vs Recall

  39. Conclusion • We propose a parameters estimation approach for capturing user’s target as a distribution • A display set selection scheme similar to maximum entropy display is used to capture more user’s feedback information • A SOM-based inter-query feedback is proposed • Overcome the single cluster assumption of most intra-query feedback approach

  40. Making People Share DIStributed COntent-based Visual Information Retrieval

  41. How to locate relevant images In an efficient manner? P2P Information Retrieval Images … Feature Extraction Peer databases Lookup Query Image Query Result

  42. Contributions • Migrate centralized architecture of CBIR to distribution architecture • Improve existing query scheme in P2P applications • A novel algorithm for efficient information retrieval over P2P • Peer Clustering • Firework Query Model (FQM)

  43. Existing P2P Architecture • Centralized • Napster, SETI (Berkeley), ezPeer (Taiwan) • Easy implementation • Bottleneck, single point failure • Legal problems update answer query transfer

  44. Existing P2P Architecture • Decentralized Unstructured • Gnutella (AOL, Nullsoft), Freenet (Europe) • Self-evolving, robust • Query flooding Peer TCP connection

  45. Existing P2P Architecture • Decentralized Structured • Chord (SIGCOMM’01), CAN(SIGCOMM’01), Tapestry (Berkeley) • Efficient retrieval and robust • Penalty in join and leave Files shared by peers Distributed Hash Table (DHT) CAN model TCP connection Peer in the network

  46. DISCOVIR Approach • Decentralized Quasi-structured • DISCOVIR (CUHK) • Self-organized, clustered, efficient retrieval attractive connections random connections

  47. Design Goal and Algorithms used in DISCOVIR • Peers sharing “similar” images are interconnected • Reduce flooding of query message • Construction of self-organizing network • Signatures calculation • Neighborhood discovery • Attractive connections establishment • Content-based query routing • Route selection • Shared file lookup

  48. Construction of Self-Organizing Network • Signatures calculation • Signatures discovery of neighborhoods • Comparison of signatures • Attractive connection establishment

  49. Signatures Calculation Feature vector space

  50. Signatures Calculation Centroid of peer Peer B Peer A

More Related