1 / 110

Attention in Computer Vision

Attention in Computer Vision. Mica Arie-Nachimson and Michal Kiwkowitz May 22, 2005 Advanced Topics in Computer Vision Weizmann Institute of Science. Problem definition – Search Order. Vision applications apply “expensive” algorithms (e.g. recognition) to image patches

york
Download Presentation

Attention in Computer Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Attention in Computer Vision Mica Arie-Nachimson and Michal Kiwkowitz May 22, 2005 Advanced Topics in Computer Vision Weizmann Institute of Science

  2. Problem definition – Search Order • Vision applications apply “expensive” algorithms (e.g. recognition) to image patches • Mostly naïve selection of patches • Selection of patches determines number of calls to “expensive” algorithm Object recognition NO

  3. Problem Definition - Search Order • More sophisticated selection of patches would imply less calls to “expensive” algorithm • Attention used to efficiently focus on incoming data (better use for limited processing capacity) Object recognition YES NO

  4. 5 3 2 4 1 6 Problem Definition - Search Order Object recognition

  5. Outline • What is Attention • Attention in Object Recognition • Saliency Model • Feature Integration Theory • Saliency Algorithm • Saliency & Object Recognition • Comparison • Inner Scene Similarity Model • Biological motivation • Difficulty of Search Tasks • Algorithms • FLNN • VSLE

  6. Outline • What is Attention • Attention in Object Recognition • Saliency Model • Feature Integration Theory • Saliency Algorithm • Saliency & Object Recognition • Comparison • Inner Scene Similarity Model • Biological motivation • Difficulty of Search Tasks • Algorithms • FLNN • VSLE

  7. Attention • Attention implies allocating resources, perceptual or cognitive, to some things at the expense of not allocating them to something else.

  8. What is Attention • You are sitting in class listening to a lecture. • Two people behind you are talking. • Can you hear the lecture? • One of them mentions the name of a friend of yours. • How did you know?

  9. Attention in Other Applications • Face Detection (feature selection) • Video Analysis (temporal block selection) • Robot Navigation (select locations) • …

  10. Attention is Directed by: • Bottom-up: • From small to large units of meaning • Rapid • Task-independent

  11. http://www.rybak-et-al.net/nisms.html Attention is Directed by: • Top-down: • Use higher levels (context, expectation) to process incoming information (Guess) • Slower • Task dependent

  12. Outline • What is Attention • Attention in Object Recognition • Saliency Model • Feature Integration Theory • Saliency Algorithm • Saliency & Object Recognition • Comparison • Inner Scene Similarity Model • Biological motivation • Difficulty of Search Tasks • Algorithms • FLNN • VSLE

  13. WHICH? Attention When is information selected (filtered)? • Early selection (Broadbent, 1958) • Cocktail party phenomenon (Moray, 1959) • Late selection (Treisman, 1960) - attenuation • All information is sent to perceptual systems for processing • Some is selected for complete processing • Some is more likely to be selected

  14. Parallel Search Is there a green O ? + A. Treisman, G. Gelade, 1980

  15. Conjunction Search Is there a green N ? + A. Treisman, G. Gelade, 1980

  16. Results A. Treisman, G. Gelade, 1980

  17. Conjunction Search + A. Treisman, G. Gelade, 1980

  18. Color map Orientation map A. Treisman, G. Gelade, 1980

  19. Color map Orientation map A. Treisman, G. Gelade, 1980

  20. Conjunction Search + A. Treisman, G. Gelade, 1980

  21. Intensity Curvature P P P P P P P P P P Line End Orientation P P x x x P P P s P x I x I I P P I I Color x x x x x Primitives Movement x

  22. Feature Integration Theory Attention - two stages: • Attention • Serial Processing • Localized Focus • Slower • Conjunctive search • Pre-attention • Parallel Processing • Low Level Features • Fast • Parallel Search How is the Focus found & shifted? A. Treisman, G. Gelade, 1980

  23. Outline • What is Attention • Attention in Object Recognition • Saliency Model • Feature Integration Theory • Saliency Algorithm • Saliency & Object Recognition • Comparison • Inner Scene Similarity Model • Biological motivation • Difficulty of Search Tasks • Algorithms • FLNN • VSLE

  24. Attention Shifts in Attention “Shifts in selective visual attention: towards the underlying neural circuitry”, Christof Koch, and Shimon Ullman, 1985 • Feature • Maps • Orientation • Color • Curvature • Line end • Movement • Feature • Maps • Orientation • Color • Curvature • Line end • Movement • Feature • Maps • Orientation • Color • Curvature • Line end • Movement • Feature • Maps • Orientation • Color • Curvature • Line end • Movement Saliency • Feature • Maps • Orientation • Color • Curvature • Line end • Movement Central Representation C. Koch, and S. Ullman, 1985

  25. Saliency • Salient - stands out “A model of saliency-based visual attention for rapid scene analysis” Laurent Itti, Christof Koch, and Ernst Niebur, 1998 • Example – telephone & road sign have high saliency L. Itti, C. Koch, and E. Niebur, 1998

  26. from C. Koch L. Itti, C. Koch, and E. Niebur, 1998

  27. Intensity Cells in the retina L. Itti, C. Koch, and E. Niebur, 1998

  28. 0 1 2 8 Intensity Create 8 spatial scale using Gaussian pyramids L. Itti, C. Koch, and E. Niebur, 1998

  29. - + + Fine scale - Coarse scale Intensity Center-Surround difference operator • Sensitive to local spatial discontinuities • Principle computation in the retina & primary visual cortex • Subtract coarse scale from fine scale fine coarse L. Itti, C. Koch, and E. Niebur, 1998

  30. Point-by-point subtraction Gauss Pyramid Interpolation Toy Example Fine level Coarse level Coarse level

  31. Point-by-point subtraction Gauss Pyramid Interpolation Toy Example Fine level Coarse level Coarse level

  32. Intensity Different ratios – multiscale feature extraction Compute:  6 Intensity maps L. Itti, C. Koch, and E. Niebur, 1998

  33. Color Kandel et al. (2000). Principles of Neural Science. McGraw-Hill/Appleton & Lange Same c and s as with intensity 12 Color maps L. Itti, C. Koch, and E. Niebur, 1998 More

  34. Same c and s as with intensity 24 Orientation maps From Visual system presentation by S. Ullman Orientation L. Itti, C. Koch, and E. Niebur, 1998 More

  35. from C. Koch L. Itti, C. Koch, and E. Niebur, 1998

  36. Normalization Operator More L. Itti, C. Koch, and E. Niebur, 1998

  37. Saliency Map L. Itti, C. Koch, and E. Niebur, 1998

  38. Algorithm- up to now 1. Extract Feature Maps 2. Compute Center-Surround (42) • Intensity – I (6) • Color – C (12) • Orientation – O (24) 3. Combine each channel into conspicuity map 4. Compute saliency by summing and normalizing maps

  39. Laurent Itti, Christof Koch, and Ernst Niebur, 1998

  40. Winner Takes All Selection (FOA) Leaky integrate-and-fire neurons “Inhibition of return” FOA – Focus Of Attention L. Itti, C. Koch, and E. Niebur, 1998

  41. Inhibition of return ends Results • FOA shifts: 30-70 ms • Inhibition: 500-900 ms L. Itti, C. Koch, and E. Niebur, 1998

  42. Results Image Saliency SFC Output Spatial Frequency Content, Reinage & Zador, 1997 L. Itti, C. Koch, and E. Niebur, 1998

  43. Results Image (a) (b) Saliency (c) SFC (d) Output Spatial Frequency Content, Reinage & Zador, 1997 L. Itti, C. Koch, and E. Niebur, 1998

  44. Outline • What is Attention • Attention in Object Recognition • Saliency Model • Feature Integration Theory • Saliency Algorithm • Saliency & Object Recognition • Comparison • Inner Scene Similarity Model • Biological motivation • Difficulty of Search Tasks • Algorithms • FLNN • VSLE

  45. Attention & Object Recognition • “Is bottom-up attention useful for object recognition?” • U. Rutishauser, D. Walther, C. Koch and P. Perona, 2004 Attention U. Rutishauser, D. Walther, C. Koch and P. Perona, 2004

  46. saliency model Growing region in strongest map To Object Recognition (Lowe) Object Recognition U. Rutishauser, D. Walther, C. Koch and P. Perona, 2004 More

  47. Attention & Object Recognition Learning inventories – “grocery cart problem” Real world scenes 1 image for training (15 fixations) 2-5 images for testing (20 fixations) U. Rutishauser, D. Walther, C. Koch and P. Perona, 2004

  48. testing training Object recognition Match

  49. “Grocery Cart” Problem training testing1 testing2 U. Rutishauser, D. Walther, C. Koch and P. Perona, 2004

More Related