1 / 27

HANOLISTIC: a h ierarchical automatic image an notation system using h olistic approach

A hierarchical system that automatically assigns keywords to digital images using a holistic approach. Overcomes labor-intensive and time-consuming annotation process.

ewoodward
Download Presentation

HANOLISTIC: a h ierarchical automatic image an notation system using h olistic approach

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. HANOLISTIC: a hierarchical automatic image annotation system using holistic approach Özge Öztimur Karadağ & Fatoş T. Yarman Vural Department of Computer Engineering Middle East Technical University, Ankara, Turkey

  2. Automatic Image Annotation • Image Annotation : Assigning keywords to digital images. • Labor intensive • Time consuming • Need a system that automatically annotates images.

  3. Image Annotation Literature • Annotation problem has become popular since 1990s. • Related to CBIR. • CBIR processes visual information • Annotation processes visual and semantic information • Relate visual content information to semantic context information.

  4. Problems About AutomaticImage Annotation • Human subjectivity • Semantic Gap • Availability of datasets

  5. Image Annotation Approaches in the Literature… • Segmental Approaches • Segment or partition the image into regions • Extract features from the regions • Quantize features into blobs • Model the relation between the image regions and annotation words • Holistic Approaches • Features are extracted from the whole image.

  6. SegmentalAnnotation: sky, boat, sea, tree

  7. Holisticannotation: Tiger, tree, snow

  8. The Proposed System: HANOLISTIC • Introducing semantic information as supervision. • each word is considered as a class label, • an image belongs to one or more classes • Holistic Approach: multiplevisual features are extracted fromthe several whole image. • Multiple feature spaces

  9. Description of an Image • Content Description by Visual Features of Mpeg-7 • Color Layout • Color Structure • Scalable Color • Homogenous Texture • EdgeHistogram • Context Description by Semantic Words • Annotation words

  10. System Architecture of HANOLISTIC • Level-0 : consists of level-0 annotators, one annotator for each visual description space. • Meta-level : consists of a meta-annotator

  11. Level-0 Annotator • refers to the features of the i th image in the j th description space • refers to the membership value of the l th word for the i th image in the j th description space.

  12. Meta-Level • The results of level-0 annotators are aggregated. • is a vector, referring to the final word membership values for the i th image.

  13. Experimental studies • Realization of HANOLISTIC • Instance based realization of Level-0 • Eager realization of Level-0 • Realization of Meta-level • Performance criteria • Results

  14. Experimental Setup • Data set: A subset of Corel Stock Photo Collection, consisting of 5000 images. • Training set: 4500 images (500 images for validation) • Testing set: 500 images • Each image is annotated with 1-5 many words.

  15. Instance based Realization of Level-0 Annotator by Fuzzy-knn • Level-0 annotators are realized by fuzzy-knn. • For each description space; k nearest neighbors of the image is determined. • Word membership values are estimated considering the neighbors’words and their distance from the image. • High membership values are assigned to words that appear in close neighborhood.

  16. Eager Realization of Level-0 by Ann • For a given image Ii, ANN receives visual description of the image as input and semantic annotation words of the image as target. • Each ANN is trained with backpropagation and a randomly selected set of images is used for validationto determine when to stop training. K-fold cross validation is applied.

  17. Realization of Meta-Levelby Majority Voting • Adds the membership values returned by level-0 annotators using the formula • where, Pi,j is a vector containing the word membership values returned from the jth level-0 annotator. • For each word select the maximum of the five word membership values estimated by the level-0 annotators.

  18. Performance Criteria • Precision • Recall • F-score

  19. Performance of Level-0 Annotators • Performance of Level-0 annotatorswith fuzzy-knn

  20. Performance of HANOLISTIC • Comparison of HANOLISTICwith other systems in the literature:

  21. Annotation Examples

  22. Annotation Examples…

  23. Conclusion • We proposed a hierarchical automatic image annotation system using holistic approach. • We tested the system both with an instance based and an eager method. • We realized that the instance based methods are more promising in the considered problem domain.

  24. Conclusion… • The power of the proposed system comes from thefollowing main principles: • Simplicity • Fuzziness • Simultaneous processing of content and context information • Holistic view of image through different perspectives

  25. Future Work • Conduct experiments on other descriptors • Test other algorithms at level-0 conforming to the principle of least commitment • Apply holistic approach followed by a segmentation process , for annotation or intelligent segmentation.

  26. Thank you • Questions and Comments

  27. References • Duygulu, Barnard, Freitas and Foryth ‘Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary’ in ECCV’02: Proceedings of the 7th European Conference on Computer Vision,2002 • Jeon, Lavrenko, Manmatha ‘Automatic image annotation and retrieval using cross-media relevance models’, in SIGIR’03 • Monay and Perez ‘Plsa-based image auto-annotation: constraining the latent space’ in MULTIMEDIA’04 • Akbaş and Vural ‘Automatic image annotation by ensemble of visual descriptors’ in CVPR’07 • Feng, Manmatha and Lavrenko ‘Multiple bernoulli relevance models for image and video annotation’. CVPR’02. • Tang and Lewis, ‘Image auto-annotation using ‘easy’ and ‘more challenging’ training sets’, 7th International Workshop on Image Analysis for Multimedia Interactive Services, 2006

More Related