Automatic Image Annotation Using GHSOM

1. Automatic Image Annotation Using GHSOM Hsin-Chang Yang Associate Professor Department of Information Management National University of Kaohsiung

2. Outline • Introduction • Preprocessing and Clustering by GHSOM • Automatic Image Annotation • Experimental Results • Conclusions

3. Introduction • Recent development of digital imaging technology has produced enormous amount of images stored in large repositories, especially the World Wide Web. • It is necessary to have a way to retrieve images from such large repositories. • How?

4. Introduction • Three main approaches • Keyword-based approach • easy, precise, but laborious • Content-based approach • imprecise and inconsistent • Semantic-based approach • difficult to derive semantics from images

5. Introduction • Keyword-based image retrieval (KBIR) is easy, intuitive, and precise, provided correct annotations have been added to images. • Only few images have been annotated nowadays. • manual annotation is not possible for online, collective repositories such as the WWW • We need an automatic scheme to annotate images.

6. images incoming images image vectors preprocessing Train by GHSOM annotations annotation vectors image associations Hierarchy of images Hierarchy of annotations Association discovery image/annotation associations annotated images annotation associations association discovery process annotation process System Architecture

7. Preprocessing and Clustering by GHSOM • Image preprocessing • transform an image I into a feature vector FI which is composed by its intensity histogram HI and power spectrum density PI

8. Preprocessing and Clustering by GHSOM • Annotation preprocessing • stemming • stopword elimination • Annotation encoding • The annotation of image I, denoted by AI, is encoded into a vector AI = {aIj}, 1  i  |V|, where V denotes the vocabulary, aIj = 1 if the j-th keyword in V appears in Ai, otherwise aIj = 0.

9. Preprocessing and Clustering by GHSOM • GHSOM was proposed by Rauber et al. to provide the SOM with capabilities of dynamic map expansion and hierarchy construction. • had been applied to expertise management, failure detection, and multilingual information retrieval • We used GHSOM to organize images and their annotations into hierarchies.

10. Layer 0 Layer 1 Layer 2 Layer 3 Preprocessing and Clustering by GHSOM • A typical structure of GHSOM

11. image hierarchy annotation hierarchy Aq Ik Ap I4 A3 I1 I2 A1 A2 A5 A4 I3 labelling I5 Preprocessing and Clustering by GHSOM • Image and annotation clustering • Image/annotation vectors were trained by GHSOM. • Two hierarchies were constructed.

12. Automatic Image Annotation • The constructed hierarchies reveal associations among images and annotations, respectively. • However, associations across images and annotations are much difficult to find because there is no direct mapping between these hierarchies.

13. Automatic Image Annotation • Finding Associations • to associate an image cluster with an annotation cluster • a kind of general problem of ontology alignment • To associate an image cluster Ik with some annotation cluster Al, we use a voting scheme to calculate the likelihood of such association.

14. Automatic Image Annotation • Voting for best-matched cluster • For each pair of image documents Ii and Ij in Ik, we should find the neuron clusters which their corresponding annotations Ai and Aj are labelled to in the annotation hierarchy. Let these clusters be Ap and Aq. • Find the shortest path between Ap and Aq in the annotation hierarchy. • Add score 1 to Ap and Aq. Add 1/(dist(Ap, Aq)-1) to all other clusters in the path. • Repeat 1-3 for all pairs of documents in Ik.

15. annotation hierarchy 0.83 2 1.33 0.83 2 2 0 Automatic Image Annotation • We associate Ik with Al when it has the highest score. • An example

16. Automatic Image Annotation • To annotate an image I: • Label I to some image cluster, say Ik • Find the associated annotation cluster of Ik, say Al • Image I is then annotated with all keywords in Al.

17. Experimental Results • The Groundtruth dataset in Washington University were used • all images were annotated by hands • Our corpus contains 1109 images as well as their annotations, classified into 22 classes. • We have a annotation vocabulary of size 435. • Each image and annotation is transformed into a vector. The lengths of the image and annotation vectors are 512 and 435, respectively.

18. Experimental Results • We used the GHSOM program developed by Rauber’s team to train the image and annotation vectors. • http://www.ifs.tuwien.ac.at/~andi/ghsom/ • We randomly selected 80% of images for training and the other 20% for testing.

19. Experimental Results • Performance Evaluation • Overlap ratio between manual annotations and automatic annotations • A1: set of automatic annotations • A2: set of manual annotations • average overlap ratio is 0.709 in our experiments

20. Experimental Results • Performance evaluation on image retrieval • recall and precision • 21 random query keywords • average recall: 42% • average precision: 53.9%

21. Conclusions • We developed a method to automatically annotate images without human intervention. • GHSOM performs well in clustering and organizing documents. • Our method could be beneficial if manual annotations were not available or possible. • The retrieval result could be improved by developing more sophisticated alignment method and feature representations.

22. Thanks for your attention.