1 / 25

Learning Prepositions for Spatial Relationships in BOLT

Learning Prepositions for Spatial Relationships in BOLT. Soar Workshop 2012 James Kirk, John Laird 6/21/2012. Outline. Prepositions in BOLT Compositional Spatial Relationships Learning Process in Architecture with SVS Projection Conclusions. Prepositions/Spatial Relationships in BOLT.

yitta
Download Presentation

Learning Prepositions for Spatial Relationships in BOLT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning Prepositions for Spatial Relationships in BOLT Soar Workshop 2012 James Kirk, John Laird 6/21/2012

  2. Outline • Prepositions in BOLT • Compositional Spatial Relationships • Learning Process in Architecture with SVS • Projection • Conclusions

  3. Prepositions/Spatial Relationships in BOLT • “What is to the right of the pantry?” • “Put the triangle in the stove.” • “Pick up the red object to the right of the pantry.”

  4. Spatial Relationships as Compositions y-aligned z-aligned x-greater than Z “right of” X Y Other potential compositions: “In front of” y-less than z-aligned x-aligned “Above” z-greater than “Intersecting” y-aligned z-aligned x-aligned

  5. Learn new prepositions that map to spatial relations “The red triangle is right of the blue sphere”

  6. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Phase 1 Perception - Receive object information from input link - Create internal representations Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory Spatial Visual System Action Perception Obj1 - bl3 - sp7 Obj2 -re23 -tr17 Obj3 - bl3 - sq8 Spatial Primitives

  7. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Phase 2 Message Parsing - Parse the given sentence - Create usable message structures Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar LGSoar Working Memory relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere Obj1 - bl3 - sp7 Obj2 -re23 - tr17 Obj3 - bl3 - sq8 Spatial Visual System Action “The red triangle is right-of the blue sphere” Perception Spatial Primitives

  8. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Phase 3 Interaction Stack - Designate message purpose as preposition learning Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Interaction Episodic Memory Interaction Agent’s Experiences LGSoar LGSoar Working Memory relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere Obj1 - bl3 - sp7 Obj2 -re23 tr17 Obj3 - bl3 - sq8 Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  9. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Interaction Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory relation-message ^word right-of ^o1 red, triangle ^o2 blue, sphere red, triangle right-of Obj1 - bl3 - sp7 Obj2 -re23 -tr17 Obj3 - bl3 - sq8 blue, sphere Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  10. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning - Lookup the category of red from smem Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory red color red, triangle right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 blue, sphere Prep. Learning Noun Learning Spatial Visual System Action Perception Spatial Primitives

  11. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Word – Category Mapping Action Knowledge Phase 4 Preposition Learning - Lookup the symbol for red + color from smem Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun/Adjective – Perceptual Symbol Mapping Noun/Adjective – Perceptual Symbol Map Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory color red re23 re23, triangle red, triangle right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 blue, sphere Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  12. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Phase 4 Preposition Learning - Update the descriptions with perceptual symbols Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun/Adjective – Perceptual Symbol Mapping Noun/Adjective – Perceptual Symbol Map Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory re23, triangle re23, tr17 red, triangle right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 bl3, sp7 blue, sphere bl3, sphere Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  13. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Phase 4 Preposition Learning - Look for the objects matching the description Prep Learning Noun/Adjective – Perceptual Symbol Mapping Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory re23, tr17 Obj2 right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 bl3, sp7 Obj1 Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  14. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning - SVS updates model of objects in the world from the perception input. Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory Obj2 right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 Obj1 Prep. Learning Spatial Visual System Action Perception Spatial Primitives Spatial Primitives Obj1 Obj2 Obj3

  15. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning - SVS is queried for primitive spatial relationships between Obj1 and Obj2 Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory Obj2 Obj2 ? right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 Obj1 Obj1 Prep. Learning Spatial Visual System Action Perception Obj1 Spatial Primitives Spatial Primitives Obj2 Obj3

  16. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning • Phase 4 • Preposition Learning • The results of the query are stored in working memory. Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory right-of Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 Prep. Learning Spatial Visual System Action Perception Spatial Primitives y-aligned +x z-aligned

  17. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning - List of spatial primitives are abstracted into general spatial relationship and mapped to “right-of.” Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory SR11 right-of y-aligned z-aligned +x Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  18. Semantic Memory Human: “The red triangle is right-of the blue sphere” Procedural Memory Preposition – Spatial Rel Mapping Preposition – Spatial Rel Mapping Word – Category Mapping Action Knowledge Prep Learning Phase 4 Preposition Learning - Learned spatial relationship mapping to “right-of” is stored in semantic memory. Noun/Adjective – Perceptual Symbol Mapping Noun Learning Primitive Verb – Operator Mapping Verb Learning Verb – Operator Mapping Indexing Episodic Memory Interaction Agent’s Experiences LGSoar Working Memory SR11 right-of y-aligned z-aligned +x Obj2 -re23 -tr17 Obj1 - bl3 - sp7 Obj3 - bl3 - sq8 Prep. Learning Spatial Visual System Action Perception Spatial Primitives

  19. Refining learned knowledge Is this triangle also to the right of the square? Disjunctive primitive relationships in training are removed.

  20. Learning is interactive • Learning can be initiated by instructor or agent “What is behind the blue sphere?” • “I don’t know the preposition behind. Please teach me with examples.” “The green square is behind the red triangle.” “Finished teaching.” - “The blue square is behind the blue sphere.”

  21. Spatial Projection “Put the object to the right of the pantry.” “right of” Z X Y

  22. Current Uses • Describing the scene • Specifying objects in the scene • Tracking objects and relationships • “The goal is that the red triangle is inside the pantry” • Projecting relationships back to a point in 3D space to perform actions

  23. Evaluation Nuggets • Learns very fast • Works well for simple relationships and projection/tracking Coals • Cannot learn complex relationships with many objects • Cannot learn certain types of relationships such as diagonal • Not robust to mistakes in training examples

  24. Future Work • Complex multi-object relationships • For example in a line or a circle • More complex spatial relationships • Using distance/contacting etc. • Pattern matching (diagonal) • Will require more training examples • Projection with multiple relationships • “put this inside the pantry and to the right of the blue object”

  25. Questions?

More Related