Using Context to Improve Robotic Vision

1. Using Context to Improve Robotic Vision Catie Meador, Xiang Li, and Dr. Mohan Sridharan Robotics Lab 1

2. Context • “Any information that might be relevant to object detection, categorization, and classification tasks, but not directly due to the physical appearance of the object, as perceived by the image acquisition system” (Marques et al. 2011) • Visual information • Non-visual information • Uses of context in object recognition • Clues for function • Low-res images • Attention Context Context Marques et al. 2011; Hoiem 2004 2

3. Uses of Context Objects at least partially defined by function Some objects tend to occur in pairs Objects tend to occur in specific scenes Marques et al. 2011; Hoiem 2004 3 http://upload.wikimedia.org/wikipedia/commons/5/51/Gold_doorknob.jpg; http://willjlindsey.com/wp-content/uploads/2010/12/gold-fish-bowl.jpg; http://www.theautochannel.com/news/2009/05/21/461948.1-lg.jpg

4. What is This? Marques et al. 2011 4

5. Marques et al. 2011 5

6. Context Used Scene context: office Other object context: computer located above keyboard On a desk Marques et al. 2011 6

7. Related Work • Context-based vision system for place and object recognition (Torralba et al. 2003) • Uses the ‘gist’ of the scene to identify objects and places • Context-based categorization (Galleguillos & Belongie 2010) • Three main types of context: probability (semantic), position (spatial), and size (scale) • Vision-based Autonomous Learning of Object Models on a Mobile Robot (Li & Sridharan) • Learns an object model using local, global and temporal visual cues • Learning is triggered by motion cues 7

8. Why is this important? • Allow robots to identify objects more accurately • Allows for higher functioning capabilities • Faster identification – puzzle • Allows robots to operate more effectively in difficult visual circumstances • Allow robots to learn about interactions between objects • Improves general knowledge • Co-occurrence 8 Li and Sridharan 2012

9. Objectives • Teach a robot to recognize certain objects through Supervised Learning methods • Train a robot to use this information to recognize objects through neighbor-based context • Improve upon the color-segmentation algorithm of Felzenszwalb and Huttenlocher (2004) 9

10. Supervised Learning • Class of machine learning • Program is given labeled data to train on • Generalizes from the training data, uses this to categorize test data • Ex: labeled images used to learn certain objects 10 http://www.python-course.eu/images/supervised_learning.png

11. Segmentation • Used to segment an image into regions which reflect global aspects of the image • Perceptual grouping is crucial in human vision (σ = 0.8, k = 300) 11 Felzenszwalband Huttenlocher 2004

12. Tasks • Recognizing general objects • Supervised learning • Enhance segmentation algorithm to consider color value • Label neighboring regions based on learned objects • Use to identify ROI • Current progress: connecting to robots, taking images 12 Li and Sridharan 2012

13. References • Felzenszwalb, P. & Huttenlocher, D. Efficient Graph-Based Image Segmentation. IJCV 59:2 (2004). • Galleguios C. & Belongie, S. Context based object categorization: A critical survey. CVIU 114 (2010). • Hoiem, D. (2004). Putting Context into Vision [PowerPoint Slides]. Retrieved from: www.cs.uiuc. edu/~dhoiem/presentations/putting_context _into_vision_final.ppt • Li, X. & Sridharan. M. Vision-based Autonomous Learning of Object Models on a Mobile Robot. ARMS 12 (2012). • Marques, O., Barenholtz, E., Charvillat, V. Context modeling in computer vision: techniques, implications, and applications. Multimedia Tools Appl 51 (2011). • Torralba, A., Murphy, K., Freeman, W., Rubin, M. Context-based vision system for place and object recognition. ICCV’03 (2003). 13

14. Learning Layered Object Model • Object model has four components • Gradient features and relative spatial arrangement • Spatial arrangement of gradient features • Connection potentials between neighboring gradientfeatures • Color distribution of pixels between gradient features • Image segments and relative spatial arrangement of segments • Graph-based segmentation • Color distributions • Distributions of pixels 14 Li and Sridharan 2012

15. Learning Layered Object Model 15 Li and Sridharan 2012

16. Context-based vision system for place and object recognition (Torralba et al. 2003) • Goal of identifying familiar locations to categorize new environments • Using this information to provide context for object recognition • Trained to recognize over 60 locations an suggest the presence and location of 20+ different object types • Recognizing object types in a natural, unconstrained setting 16

17. Context-based categorization (Galleguillos & Belongie 2010) • Probability (semantic) – likelihood of an object being found in some scenes but not in others • Statistical methods generalize semantic context for object categorization • Position (spatial) – likelihood of finding an object in some positions and not others with respect to other objects in the scene • Stored in the form of rules and a graph-like structure • Size (scale) – objects have a limited set of size relations to other objects in the scene • Requires processing of spatial and depth relations between the target and other objects 17

18. Context-based categorization (Galleguillos & Belongie 2010) Object categorization system using the three main types of context 18

19. Graph-based Segmentation • Measures evidence of a boundary between two regions through a graph-based representation of the image • Graph-based algorithm: • Nodes: pixels • Edges: some neighboring pixels • Weights measure the dissimilarity between pixels 19 Felzenszwalband Huttenlocher 2004