Automatic Thumbnail Cropping and Its Effectiveness

1. Automatic Thumbnail Cropping and Its Effectiveness Bongwon Suh, Haibin Ling, Benjamin B. Bederson, and David W. Jacobs Human Computer Interaction Lab. University of Maryland at College Park UIST 2003 November 4th, 2003

2. Problem and Motivation • Thumbnails require significant amount of space on the screen • Small thumbnails are less useful • Make small thumbnails more recognizable and useful PhotoMesa (Zoomable Image Browser) Microsoft Windows Explorer Small screen devices Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

3. Thumbnail Cropping • Image Shrinking (Plain Thumbnail) • We lose detailed information • Image Cropping • We lose a part of information • Select the portion of maximal informativeness • When cropping, keep the more informative part and cut less informative part • Preserve the recognizability of important objects in thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

4. Thumbnail Cropping Example • Crop first, then shrink the cropped images Original Image Shrinking (Subsampling) Cropped Image Generated Thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

5. More Examples • Both sets use the same amount of screen space Plain Thumbnails Cropped Thumbnails Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

6. Automatic Thumbnail Cropping • Automate Thumbnail Cropping Procedure • Which part is more informative? • Need to measure informativeness • Saliency based thumbnail cropping • Improve cropping by using dynamic threshold • Face detection based thumbnails cropping • Applying existing techniques as an example of detecting semantic information in images Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

7. Saliency Based Thumbnail Cropping • Saliency • Visual attention model (color, intensity, etc) • Itti and Koch (1998, 1999) • Does not need prior knowledge on images • Assumption • Saliency  More informativeness Original Image Computed Saliency Map Cropped Image Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

8. Face Detection Based Thumbnail Cropping • When semantic information of images can be detected, more efficient cropping is possible • Face Detection: Schneiderman and Kanade (2000) Thumbnails generated with three techniques Original Image Face Detection Face Detection Based Cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

9. Saliency Based Cropping • Brute force algorithm • Find a minimum size rectangle that contains a certain portion (threshold) of total saliency • Static threshold algorithm • Require exhaustive search Find a minimum rectangle R satisfying: Sum of saliency values inside R Threshold λ≥ Sum of all saliency values in image Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

10. Using a static threshold Increasing cropping bounds until it contains X% of total saliency Keep adding the next most salient point outside the current cropping bounds Greedy Algorithm Points with Weak Saliency Current Bounds New Bounds The most salient point outside the current bounds Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

11. Dynamic Saliency Threshold • Scattered saliency • Need to cut out little • Gathered saliency • Larger cutting is possible • The most effective threshold varies from images to images Gathered saliency Scattered saliency Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

12. Find a point of diminishing returns Adding small amounts of saliency requires a large increase of the cropping bounds Binary search for maximum gradient point Cropping With Dynamic Threshold Maximum gradient point Cropping Area Sum of Saliency Values inside Area Area-Saliency Sum Graph Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

13. 0.8 Static And Dynamic Threshold : Maximum Gradient Point Cutting out too little Cutting out too much 0.8 Result using Dynamic Threshold Area-Saliency Sum Graph Original Image Static threshold 80% Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

14. Works well with Corbis images Professionally prepared and already cropped Immune to already cropped images Not perfect but generates reasonable results Landscape, scenery Artistic images Robust Algorithm Thumbnail Cropping with Dynamic Threshold Original Image Some core parts has been cut out Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

15. User Study Design • Participant • Twenty students recruited on campus • Task • Recognition Task • Visual Search Task • Image Set • Animal Set: Common objects • Corbis Set: Professionally prepared photos • Face Set: Well known figures e.g. Entertainers • Thumbnail Technique • No cropping • Saliency based cropping • Face detection based cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

16. Recognition Task • To measure the effect of thumbnail techniques on object recognition • Target images were shown for two seconds • Participants were asked to click what they saw. • Measured recognition accuracy • # of right answers / # of total tasks Face Set Animal Set Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

17. Recognition Task Hypothesis Big enough to be recognized in both cases Thumbnails are too small anyway Meaningful 100% Recognition Rate Thumbnail Technique A Thumbnail Technique B Difference ? Thumbnail Size Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

18. Recognition Task Result • All curves are different from each others. (p < 0.01) Animal Set Face Set Face Detection Based Cropped Saliency Based Cropped No Cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

19. Visual Search Task • Find an image that matches a given task description • Verbal description (except faces) • PhotoMesa interface • Measured browsing completion time Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

20. No cropping vs. Saliency Based Cropping F(1, 190) = 3.82 p = 0.05 Three Thumbnail Techniques on Face Set F(2, 87) = 4.56 p = 0.013 Visual Search Task Result Browsing Time Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

21. Future Work • Analysis on broad sets of large image corpus • Currently tested on about one thousand images • Performance Issues • Thumbnail cropping is implemented in Matlab Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

22. Conclusion • Saliency Based Thumbnail Cropping • General technique that does not make assumption about images • Does not need humans’ intervention • Dynamic threshold can prevent cutting out too much or too little • Face Detection Based Thumbnail Cropping • Shows how semantic information can be used to enhance the thumbnail cropping Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

23. Thank You

24. Fixed Threshold Brute Force, 90% Greedy, 80% Too big threshold (80% threshold)  Cutting out too less Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

25. Brute Force, 90% Greedy, 80% Too small threshold (80% threshold)  Cutting out too much Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

26. Brute Force, 90% Greedy, 80% Corbis image: Professionally prepared and already cropped. Saliency values are widely scattered. Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003

27. Greedy, 80% Brute Force, 90% The algorithm is not perfect but generates reasonable results. Result Brute Force Binary Dynamic Brute Force Gradient Dynamic Greedy Binary Dynamic Greedy Gradient Dynamic Automatic Thumbnail Cropping and Its Effectiveness UIST 2003