1 / 19

Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney

Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney. - Pratiksha Shah. What’s the problem. We want to automatically annotate and index images (considering their ever growing number)

wood
Download Presentation

Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Watch, Listen and Learn Sonal Gupta, Joohyun Kim, Kristen Grauman and Raymond Mooney -Pratiksha Shah

  2. What’s the problem • We want to automatically annotate and index images (considering their ever growing number) • Visual cues alone can be ambiguous (depend on lighting, and variety exhibited even by objects of the same kind)

  3. Previous methods • Previous work was focused on learning the association between visual and textual information • Many researchers have worked on activity recognition in videos using only visual cues • Some used co-training but in a different flavour (co-SVM, 2 visual views, 2 text views)

  4. How we propose to solve it • The two major factors are the features and approach. • We want to use : {Visual information + Linguistic information + Unlabeled multi-modal data } for the learning process using a co-training approach. • Visual and linguistic information are used as separate cues and we expect they will complement each other during training

  5. What is co-training • Training using 2 different (conditionally independent and sufficient) views • First learn a separate classifier for each view • Most confident predictions of these classifiers are then used to iteratively construct labeled training data • Change made: an unlabeled example is only labeled if a pre-specified confidence threshold for that view is exceeded • Used for classifying : Webpages (based on content and hyperlink views) Email ( based on header and body) object detection models

  6. Co-training Algorithm: Algorithm: A classifier for each view using the labeled data with just the features for that view. – Loop following steps until there are no more unused unlabeled instances: 1. Compute predictions and confidences of both classifiers for all of the unlabeled instances. 2. For each view, choose the m unlabeled instances for which its classifier has the highest confidence. For each such instance, if the confidence value is less than the threshold for this view, then ignore the instance and stop labeling instances with this view, else label the instance and add it to the supervised training set. 3. Retrain the classifiers for both views using the augmented labeled data.

  7. Text Feature • Pre process text by removing stop-words • Stem the remaining words using Porter stemming • Frequency of the resulting word stems comprises of the final textual features. (“bag of words” representation)

  8. Captioned images Features used : • We want to capture overall texture and color distributions in local regions • Texture – Gabor filter with 3 scales and 4 orientations • Color – Mean, Standard deviation and skewness of per-channel RGB and lab color pixel values

  9. Method (for captioned images) • Divide each image into a 4 by 6 sized cells • Compute texture feature using Gabor filter for each • The resulting feature vector for each region is then clustered using k-means • Each region is then assigned to one of the k-clusters based on its closeness to cluster centroids • Final “bag of visual words” represents every image with a vector of k values, each denoting number of regions of the image close to that value.

  10. Capture texture and color distributions of each cell into 30-dim vector Cluster the vectors using k-Means to quantize the features into a dictionary of visual words N  30 Represent each image as histogram of visual words Image Features Divide images into 46 grid [Fei-Fei et al. ‘05, Bekkerman & Jeon ‘07] The University of Texas at Austin

  11. Example dataset

  12. Results for captioned images • Compare co-training to supervised SVM • Compare co-training to Semi-supervised EM • Compare co-training to Transductive SVM

  13. Commented videos • Features used: • we use features that describe both salient spatial changes and interesting movements. • Maximize a normalized temporal laplacian operation over spatial and temporal scale • HOG – 3x3x2 spatial temporal blocks,4-bin HOG descriptor for every block => 72 element descriptor

  14. Method (for commented videos) • Use spatio-temporal descriptor motion descriptor (Laptev) • To detect events, use significant local changes in image values in both space and time. • Estimate the spatio-temporal extent of the detected events by maximizing a normalized spatiotemporal Laplacian operator over both spatial and temporal scales • A HOG(histogram of oriented gradients) is calculated at each interest point. The patch is partitioned into a grid with 3x3x2 spatio-temporal blocks, and four-bin HOG descriptors are then computed for all blocks and concatenated into a 72-element descriptor • These descriptors are clustered to form a vocabulary

  15. Describe Interest Points Histogram of Oriented Gradients (HoG) Create Spatio-Temporal Vocabulary Quantize interest points to create 200 visual words dictionary N  72 Represent each video as histogram of visual words Video Features Detect Interest Points Harris-Forstener Corner Detector for both spatial and temporal space [Laptev, IJCV ‘05] The University of Texas at Austin

  16. Example dataset

  17. Results for commented video • Compare co-training with supervised SVM for commented video dataset • Compare co-training with supervised SVM when commentary is not available during testing

  18. What does the future look like? • Larger dataset + more categories for testing • Labeled data versus associated text • Already the results show that co-training gives better results than existing semi-supervised and supervised methods.

  19. Questions ?

More Related