1 / 49

Object Lesson: Discovering and Learning to Recognize Objects

Object Lesson: Discovering and Learning to Recognize Objects. – Paul Fitzpatrick –. MIT CSAIL USA. robots and learning. Robots have access to physics, and physics is a good teacher Physics won’t let you believe the wrong thing for long

lizbeth
Download Presentation

Object Lesson: Discovering and Learning to Recognize Objects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Object Lesson: Discovering and Learning to Recognize Objects – Paul Fitzpatrick – MIT CSAIL USA

  2. robots and learning • Robots have access to physics, and physics is a good teacher • Physics won’t let you believe the wrong thing for long • Robot perception should ideally integrate experimentation, or at least learn from (non-fatal) mistakes forward seems safe… Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  3. a challenge: object perception • Object perception is a key enabling technology • Many components: • Object detection • Object segmentation • Object recognition • Typical systems require human-prepared training data; can we use autonomous experimentation? Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  4. a challenge: object perception • Object perception is a key enabling technology • Many components: • Object detection • Object segmentation • Object recognition • Typical systems require human-prepared training data; can we use autonomous experimentation? Fruit detection Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  5. a challenge: object perception • Object perception is a key enabling technology • Many components: • Object detection • Object segmentation • Object recognition • Typical systems require human-prepared training data; can we use autonomous experimentation? Fruit segmentation Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  6. a challenge: object perception • Object perception is a key enabling technology • Many components: • Object detection • Object segmentation • Object recognition • Typical systems require human-prepared training data – can’t adapt to new situations autonomously Fruit recognition Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  7. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  8. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  9. active segmentation • Object boundaries are not always easy to detect visually • Solution: Cog sweeps arm through ambiguous area • Any resulting object motion helps segmentation • Robot can learn to recognize and segment object without further contact Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  10. 1 2 3 4 5 6 7 8 9 10 active segmentation • Detect contact between arm and object using fast, coarse processing on optic flow signal • Do detailed comparison of motion immediately before and after collision • Use minimum-cut algorithm to generate best segmentation Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  11. active segmentation Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  12. active segmentation • Not always practical! • No good for objects the robot can view but not touch • No good for very big or very small objects • But fine for objects the robot is expected to manipulate Head segmentation the hard way! Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  13. car table listening to physics • Active segmentation is useful even if robot normally depends on other segmentation cues (color, stereo) • If passive segmentation is incorrect and robot fails to grasp object, active segmentation can use even clumsy collision to get truth • Seems silly not to use this feedback from physics and keep making the same mistake Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  14. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  15. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  16. opportunistic learning • To begin with, Cog has three categories of perceptual abilities: • Judgements it can currently make (e.g. about color, motion, time) • Judgements it can sometimes make (e.g. boundary of object, identity of object) • Judgements it cannot currently make (e.g. counting objects) • With opportunistic learning, the robot takes judgements in the sometimes category and works to promote them to the can category by finding reliable correlated features that are more frequently available • Example: analysis of boundaries detected through motion yields purely visual features that are predictive of edge orientation • Example: assuming a non-hostile environment (some continuity in time and space) segmented views of objects can be grouped and purely visual features inferred that are characteristic of distinct objects Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  17. training a model of edge appearance • Robot initially only perceives oriented edges through active segmentation procedure • Robot collects samples of edge appearance along boundary, and builds a look-up table from appearance to orientation angle • Now can perceive orientation directly • This is often built in, but it doesn’t have to be Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  18. most frequent samples 1st 21st 41st 61st 81st 101st 121st 141st 161st 181st 201st 221st Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  19. some tests Red = horizontalGreen = vertical Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  20. natural images 00 22.50 900 22.50 450 22.50 450 22.50 Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  21. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  22. on to object detection… Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  23. on to object detection… look for this… …in this Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  24. on to object detection… Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  25. on to object detection… Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  26. on to object detection… geometry alone geometry + color Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  27. other examples Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  28. other examples Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  29. other examples Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  30. just for fun look for this… …in this result Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  31. real object in real images Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  32. yellow on yellow Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  33. multiple objects camera image implicated edges found and grouped response for each object Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  34. attention Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  35. first time seeing a ball robot’s current view recognized object (as seen during poking) pokes, segments ball sees ball, “thinks” it is cube correctly differentiates ball and cube Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  36. open object recognition Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  37. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  38. talk overview • Perceiving through experiment • Example: active segmentation • Learning new perceptual abilities opportunistically • Example: detecting edge orientation • Example: object detection, segmentation, recognition • An architecture for opportunistic learning • Example: learning about, and through, search activity Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  39. object segmentation affordance exploitation [with Giorgio Metta] (rolling) manipulator detection (robot, human) edge catalog object detection (recognition, localization, contact-free segmentation) physically-grounded perception active segmentation Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  40. socially-grounded perception Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  41. socially-grounded perception Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  42. use constraint of familiar activity to discover unfamiliar entity used within it reveal the structure of unfamiliar activities by tracking familiar entities into and through them opportunistic architecture: a virtuous circle familiar activities familiar entities (objects, actors, properties, …) Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  43. a virtuous circle poking, chatting discover car, ball, and cube through poking; discover their names through chatting car, ball, cube, and their names Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  44. a virtuous circle poking, chatting, search follow named objects into search activity, and observe the structure of search car, ball, cube, and their names Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  45. learning about search Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  46. a virtuous circle poking, chatting, search follow named objects into search activity, and observe the structure of search car, ball, cube, and their names Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  47. a virtuous circle poking, chatting, searching discover novel object through poking, learn its name (e.g. ‘toma’) indirectly during search car, ball, cube, toma, and their names Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  48. finding the toma Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

  49. conclusion: why do this? • The quest for truly flexible robots • Humanoid form is general-purpose, mechanically flexible • Robots that really live and work amongst us will need to be as general-purpose and adaptive perceptually as they are mechanically Paul Fitzpatrick, MIT CSAIL, Humanoids 2003

More Related