Semantic Image Browser

1. Semantic Image Browser Parker Dunlap 11/15/2013

2. Introduction • Semantic image analysis techniques can automatically detect high level content of images • Lack of intuitive visualization and analysis techniques

3. Goal of the Semantic Image Browser • Allow users to effectively browse/search in large databases • Allow analysts to evaluate their annotation process through interactive visual exploration

4. Common Tasks of Image Exploration • Target search • User knows exactly what they want, a precise image • Search by association • Find interesting things related to certain image • Category search • Retrieve images that are representative of a certain class

5. Problem • Semantic contents of images are more useful for image exploration than low level features but in most large scale image collections (internet) semantics are usually not described • This has given rise to techniques that enable automatic annotation of images according to their semantic concepts

6. Semantic Image Browser (SIB) • Contains semantic image classification process that automatically annotates large image collections • Contains coordinated visualization techniques that allow interactive exploration • Contains visualization techniques that allow analysts to evaluate and monitor annotation process

7. SIB Implementation • Annotation Engine • Image Browsing Interface • Visual Image Analysis

8. Annotation Engine • Abstract Image content by detecting underlying salient objects (distinguishable regions) • Associate salient objects with corresponding semantic objects according to their perceptual properties • Keywords for semantic objects are used to annotate the image

9. Highlighted regions are salient objects detected and associated with semantic object “sand field”

10. Bridge the semantic gap • Goal is to bridge the gap between low-level visual features and high-level semantic concepts • Annotation engine has set of predefined salient objects and functions to detect them from images • Uses techniques like image segmentation and SVM classifiers

11. The Data • Annotation engine assigns a semantic concept to the data based on semantic content • Sand, Field, Water → Seaworld • Flowers, Trees → Garden

12. Image Browsing Interface • Image overview using MDS • Use the annotations to calculate distance matrix and input into MDS algorithm • Distance between each pair of images in the content space • Algorithm outputs a 2D position for each image based on similarity with other images

13. Multi-Dimensional Scaling (MDS) image view • Maps image miniatures onto the screen based on their content similarities • Similar images placed closer to each other • Goal of MDS is to map some high dimensional data into lower dimension (in our case 2D) • To learn more about MDS see MDS Overview

16. Value and Relation (VaR) content view • Visually represents the contents of the entire collection of images • Correlations of different contents and detailed annotations are displayed • Interactively exploring large datasets with real time response (high dimensionality)

17. VaR (Cont.) • Block of pixels to represent images contents • Each image is mapped to a pixel whose color indicates if the image contains/doesn’t contain the content for that block • Pixel representing the same image is the same for all blocks • Allows us to observe content of image collection by scanning labels of the blocks

18. VaR (cont.) • Can see correlations among the contents • Can also select images to see them highlighted in the view • Position of the blocks are determined by similarity with neighboring contents • Pixels are generally created in a spiral arrangement starting from the center and moving out

19. VaR (cont.)

20. VaR (cont.) • Pixel order can greatly effect the looks of VaR view

21. Image Browsing Interface (cont.) • To increase scalability, interface users miniature versions of images • High res original pictures would increase load times • Load image miniatures as textures objects in OpenGL • Allows all interactions to be done in real time

22. Interactions in MDS display • To reduce clutter in the MDS overview, the system provides many interactions • Reordering • Dynamic Scaling • Relocation • Distortion • Showing Original Image • Zoom • Pan

23. Interactions in MDS (cont.) • Reordering • Randomizing order of all images allows each frame to have an equal probability of being visible • User can also explicitly bring certain image to the front by selecting it • Dynamic Scaling • Interactively reduce image miniature size to reduce overlap or increase image size to examine detail

24. Interactions in MDS (cont.) • Relocation • Manually change position of individual image by dragging and dropping • Distortion • Enlarge size of certain image(s) while retaining size of all others

25. Interactions in MDS (cont.)

26. Interactions in MDS (cont.) • Showing Original Image • Actual image (instead of scaled down image used by OpenGL) opens at full resolution in new window • Only loaded when requested to save space/time • Zoom/Pan • Zoom in/out and pan left/right

27. Interactions in MDS (cont.) • Can use multiple techniques at once to achieve some goal • Use Dynamic Scaling with zooming in to examine local details with less clutter

28. Interactions in MDS (cont.) • Selection • Interactively select a sample image to see similar images in display • Can change similarity threshold via a slider to increase/decrease number of results • Sorting • Images can be sorted by concepts or similarity to selected image

29. Rainfall Mode • Inspired by rainfall animation • Correlations between image of interest and other images are modeled through an animation • Focus image is on the bottom (ground) and the other images fall to the ground (rain) at accelerations related to their similarity

30. Interaction in VaR display • Search for images with/without certain content • Reduce a selected subset by requiring images must/not contain certain content • Increase selected subset by adding new images • All these functions done by clicking on images while holding certain function key • Offers many similar interactions as MDS as well

31. Interaction in VaR display (cont.)

32. Interaction in VaR display (cont.)

33. Putting it together • Each image has its visual representations in both MDS and VaR views • Selected images are highlighted in both views • Can use appropriate view as needed • MDS to select image based on relationship to sample image • VaR to select image based on content • Common strategy is to start from VaR and switch to MDS after number of images has been greatly reduced

34. Annotation Analysis • We can use the MDS and VaR views to see how well the annotations of images correspond to their actual content • Select “red-flower” images from VaR view and verify using MDS view to see if the images are actually red flowers • If automatic annotation makes a mistake, user can manually annotate image to fix it

35. Annotation Analysis (cont.) • VaR display also shows the reliability of the annotation by surrounding it with a colored frame • Green is safe to use, Yellow means lower reliability measure • Reliability measure can be determined from annotation process or manually set up by analysts

36. User Study • Comparison of SIB to the sequential thumbnail view from Microsoft Explorer • Modes used in Microsoft Explorer • Random Explorer – images are randomly sorted • Sorted Explorer – images are sorted according to semantic concepts generated by the classification process

37. User Study (cont.) • 10 participants from varying fields • Each subject used both Sorted Explorer and SIB • Random Explorer was only tested on 3 participants since expected results were so low • Participants given 3 tasks to perform on 2 data sets • 180 second timeout window

38. The Tasks • Presented with a particular image and asked to search for it from the 1100 images in the data set • Asked to find images containing particular features (sand, water, sky, etc…) • Asked to approximate what proportion of the images in the dataset contained particular contents (% that contain mountains)

39. Results for Task 1 • Random Explorer • 2/9 trials failed • 81 seconds was average time with 29 seconds standard deviation • Sorted Explorer • 2/30 trials failed • 29 seconds was average time with 20 seconds standard deviation • SIB • 6/30 trials failed • 45 seconds was average time with 26 seconds standard deviation

40. Results for Task 1 • Failure in SIB was due to inaccuracy in the annotation process • SIB tended to be slower than Sorted Explorer because content names could be confusing • This advantage will decrease as the data set grows because Explorer provides no overview model • Task 2 had similar results to Task 1 • Task 3 was where SIB became dominant

41. User Review • Positive feedback for SIB • Enjoyed Search by content feature the most • Enjoyed MDS overview over Windows explorer to see entire collection of images at once • Suggested side-by-side views, example image next to blocks in VaR view

42. Conclusion • Semantic Image Browser was introduced that attempts to bride information visualization with automatic image annotation • MDS image layout that groups images based on semantic similarities • VaR content display to represent large image collections

43. Questions?

44. Resources • Semantic Image Browser: Bridging Information Visualization with Automated Intelligent Image Analysis • Value and Relation Display for Interactive Exploration of High Dimensional Datasets • MDS Overview