Linguistic Spatial Reasoning

1. Linguistic Spatial Reasoning Jim Keller Electrical and Computer Engineering Department University of Missouri-Columbia I get by with a lot of help from my friends…. Big group of faculty and students at the University of Missouri, the University of Florida, Notre Dame University, and the University of Guelph

2. Our Work • Language Generation • Modeling of spatial relationships • Natural scene understanding • Communication: human and machine • Text 2 Sketch • Human Geography • Summarization • Most of the night, Mary had medium restlessness and low motion • Common Theme • Generation of “un-natural” natural language • Modeling and matching based on linguistic statements • Users of natural language understanding systems

3. Outline • Linguistic Scene Description (External) • Linguistic Scene Description (Ego Centered) • Human/Robot Dialog • Sketched Route Understanding • Text-to-Sketch • Inverse of above

4. B A Scene Description • Natural scene understanding is an important aspect of computer vision • Spatial relations among image objects play a vital role in the description of a scene “B is to the right of A”?

5. Scene Description • Human intuition varies considerably • Vague concepts of what spatial relationships should mean • Uncertainty of how they can model differing human perceptions • Using fuzzy set-based definitions supports the “Principle of Least Commitment” of David Marr • Considerable amount of argument about the "best" method • Much of the debate centered on human intuition • Interesting results in Pattern Recognition applications • Need flexible mechanisms that can be tailored • Perhaps to individual human experts

6. angle 180° -180° Relative position between 2-D objects The histogram of angles

7. v A B q angle -180° +180° • Histograms of Forces (Matsakis) • Elegant Theory • Much more flexible than angles • Incorporates Metric information q

8. force j(d) B A Gravitational Forces distance d Choice of the functionj

9. angle -180° +180° Force-Histograms

10. Truth Membership of “A is RIGHT of B” FAB is called the Fhistogramassociated with (A,B) Divided into 4 quadrants

11. A is RIGHTof B • The average direction r(RIGHT) of the effective forces is computed • The degree of truth ar(RIGHT) of the proposition “A is to the right of B” is computed as: • a r (RIGHT) = ( r (RIGHT))  b r (RIGHT)

12. Features for Linguistic Description • Compute • 1, the Primary Direction [d1 = a(1)] • Produces Basic Directional Term (e.g., RIGHT of) • 2, the Secondary Direction [d2 = a(2)] • Supplements the description (e.g., “but a little above”) m1 and m2 specify how good the description is for each direction – combination of F0 and F2

13. 1 constant p p - 2 gravitational 1 1. Each histogram gives its opinion about the relative position between the objects that are considered. p p - 3 2. The two opinions are combined. Four numeric and two symbolic features result from this combination. 3. A system of 27 fuzzy rules and meta-rules allows meaningful linguistic descriptions to be produced. Linguistic Scene Description

14. Fuzzy Rule-based System Each linguistic output uses hedges from a dictionary of about thirty adverbs and other terms that can be tailored to individual users m 1 high medium-high medium-low low high perfectly -- nearly 1 1 no medium-high primary -- nearly loosely 2 2 1 d direction 1 medium-low mostly loosely loosely 2 3 no primary direction low Examples of Fuzzy Rules and Meta-Rules

15. Fuzzy rules for secondary direction m 2 high medium low no high somewhat strongly secondary d medium a little slightly 2 direction no secondary direction low Secondary directions

16. Simple Examples Green = satisfactory; Orange = Rather satisfactory; Red = Unsatisfactory

17. Linguistic Scene Description Example We used LADAR range images of the power-plant at China Lake, CA. They were processed by applying a median filter and a pseudo-intensity filter. The filtered images were segmented and labeled manually.

18. To the LEFT, but a little ABOVE. Linguistic Scene Description Example The system handles a rich language to describe the spatial organization of scene regions. It produces good intuitive results. The system here describes the relative position of the red object with respect to the group of buildings (in blue).

19. Linguistic Spatial Scene Description Pseudo-intensity image of surface-to-air missile (SAM) site with convoy Objects detected and automatically labeled by extended ATR system Output of Scene Description Fuzzy Rule Base (from initial system) There are 5 missile launchers (1, 2, 3, 6, 8) They surround a center vehicle (4) The image includes a SAM site A convoy of vehicles (5, 7, 9, 10) is BelowRight of the SAM site

20. An extension to the third Dimension Horizontal descriptions “The fish can is to the left of the truck box” “The truck box is to the right of the fish can but extends to the front” “The candy box is to the left of the truck box” “The truck box is to the right of the candy box but extends to the front” “The candy box is surrounded” and “The fish can is surrounded.” Vertical descriptions “The fish can is to the left of the truck box” “The truck box is to the right of the fish can” “The truck box is to the right of the candy box” “The candy box is on top of the fish can” “The fish can is below the candy box.” Near descriptions “The fish can and the candy box are near” “The fish can and the truck box are somewhat near” “The candy box and the truck box are somewhat near.”

21. Outline • Linguistic Scene Description (External) • Linguistic Scene Description (Ego Centered) • Human/Robot Dialog • Sketched Route Understanding • Text-to-Sketch • Inverse of above

22. Human/Robot Dialog • Spatial Reasoning incorporated into NRL’s Natural Language Understanding System for mobile robots • Sensed data results in a “grid map” that displays occupancy of cells (doesn’t need to be binary) Grid map after component labeling – robot heading towards Object 5

23. Generating Spatial Descriptions from Robot Sensors (here, NOMAD range sensors) features Fuzzy Rules

24. DETAILED SPATIAL DESCRIPTIONS for 6 OBJECTS: • Object 1 is mostly behind me but somewhat to the right (the description is satisfactory). The object is very close. • Object 2 is behind me (the description is satisfactory) The object is very close. • Object 3 is to the left of me but extends to the rear relative to me (the description is satisfactory). The object is very close. • Object 4 is mostly to the right of me but somewhat forward (the description is satisfactory). The object is very close. • Object 5 is in front of me (the description is satisfactory). The object is very close. • Object 6 is to the left-front of me (the description is satisfactory). The object is close. Scene 1

25. Human-Driven Spatial Language for Human-Robot Interaction • Investigating spatial language for eldercare scenario in the home • An elderly resident has lost an object; the robot will help the resident to find it. • EXAMPLE: The eyeglasses are behind the lamp on the table to the left of the bed in the bedroom • Start with human-subject experiments; Develop spatial language algorithms to match the results • NSF project # IIS-1017097 • U of Missouri (Marge Skubic) • U of Notre Dame (Laura Carlson)

26. The virtual scene used for the human subject experiments Livingroom Bedroom • Robot addressee vs. Human addressee • Vary the speaker-addressee alignment • Where to find it vs. how to find it • Vary candidate reference objects Hallway

27. Example “where” Statements • the key is on the white table in the room to his left • the book is on the wooden table in the back of the room to his right • the wallet's in the right room behind the bed on the table next to the lamp and the plant • the glasses case is down the hall in the right room -on the right side of the room as you enter, it's between the two chairs on a table next to a statue • the mug's down the hall in the left room on a table on the light brown table in front of the couch next to a purse and a hat

28. Example “how” Statements • move forward into the intersection, look left, move to the table at the far end of the room directly across from where he is and then look down and he'll find the car keys • if you turn around and then walk left or walk forward about two steps then walk left into the bedroom, walk forward until you have passed the bed on your left, turn left around the foot of the bed, proceed forward, turn left again, walk forward towards the wall near the head of the bed, look down on the bedside table and there’s the object.

29. Outline • Linguistic Scene Description (External) • Results in Scene Matching • Linguistic Scene Description (Ego Centered) • Human/Robot Dialog • Sketched Route Understanding • Text-to-Sketch • Inverse of above

30. Robot path Extracting a Navigation Path from a Sketch Start: Move forward When Object #3 is loosely to the left-front Then Turn right When Object #3 is to the left Then Move forward When Object #4 is mostly in front Then Turn left When Object #4 is to the right Then Move forward When Object #4 is to the right and Object #5 is in front Then Stop

31. Outline • Linguistic Scene Description (External) • Results in Scene Matching • Linguistic Scene Description (Ego Centered) • Human/Robot Dialog • Sketched Route Understanding • Text-to-Sketch • Inverse of above

32. Consider a frantic (and fictitious) call like this: • I saw terrorist X slip into a building up ahead. I don’t know where I am in the city and I can’t read the signs. I’m walking towards the location. • There is a somewhat long, thin rectangular shaped parking lot that extends forward. • To the immediate right of that parking lot is a parking garage. • I see a moderately small rectangular building close to me that is mostly to my left but partially forward. • Across a 4-way intersection, there is a small rectangular building close to me on my left that extends behind me. • There is another small rectangular building across the street that is mostly to the front of me, but somewhat to the left. • A short distance to the right of that building is a small L-shaped office. • I’ve reached another 4-way intersection, there is a large L-shaped building that extends to the rear. That’s where he entered. • Can we automatically pinpoint the location? Motivation: Funded by the National Geospatial Intelligence Agency

33. MU Text-to-Sketch System Natural Language Text descriptions Histograms of forces + Fuzzy rules Build graphics sketch to match IHMC Parser MATCHING Convert to graph Best approach to date: Evolutionary Computation Algorithm New approach: Hybrid Algorithm EC basic structure Subgraph Isomorphism locally in database in addition to mutation Histograms of forces for red structure (upper left) Structures into database Convert to graph

34. Some Details: Parser Syntactic parse tree for: “There is a large rectangular building is to my left.” Corresponding logical form graph of the deep semantic parse Automatically extract objects and relationships from IHMC’s deep semantic parse Right now, language variation is restricted J. Allen, Natural Language Understanding, 2nd Ed. Redwood City, CA, USA: Benjamin-Cummings, 1995. J. Allen, M. Swift and W. de Beaumont, “Deep Semantic Analysis of Text”, Proc., Symp. On Semantics in Systems for Text Processing, 2008.

35. Building blocks: Intelligent object placement scheme Too expensive to use guided random search only – Directional Fuzzy Templates An illustration of “somewhat to the right” which produces a rather large search region Example of “directly to the front,” which produces a constrained search region, combined with the distance descriptor “somewhat close.” A compound linguistic description: “perfectly to the left of the blue building” and “mostly to the right, but somewhat above the red building” I. Sledge and J. Keller, "Mapping natural language to imagery: Placing objects intelligently", Proc. IEEE Int. Conf. Fuzzy Syst., Jeju Island, Korea, August, 2009, pp 518-524 (Best student paper award).

36. Example of Sketch Creation After Parsing “To the immediate right of that parking lot is a large parking garage that is the same length as the parking lot.” Initial Placement Optimized Placement Fuzzy Initial Placement template Linguistic Descriptions From fuzzy rule base To the right but a little bit forward Perfectly to the right

37. Sketch Building Example • “There is a somewhat long, thin rectangular shaped parking lot that extends forward relative to me on my right.” (c) “I see a moderately small rectangular building close to me that is mostly to my left but partially forward.” (e) “There is another small rectangular building across the street that is mostly to the front of me, but somewhat to the left.” (b) “To the immediate right of that parking lot is a large parking garage that is the same length as the parking lot.“ (d) “Travelling to a 4-way intersection, there is a small rectangular building close to me on my left that extends behind me.” (f) Left image: Final sketch (after a few more descriptions). Right image: Ground truth recovered from matching algorithm

38. Evolutionary Computation Approach Sketch converted to chromosome structure: building = gene Each gene attributed with its Histograms of Forces

39. Initial Generation & fitness evaluation of chromosome Add n random chromosomes to population over the Geospatial database Blue are query HoFs Red are those of current Chromosome Each gene attributed with its Histograms of Forces

40. A walk by Shakespeare’s Pizza in Columbia

41. A walk by Shakespeare’s Pizza in Columbia: Up close and personal

42. Human Geography ProjectUniversity of Missouri and University of Florida HuGeo based knowledge Pattern Recognition Techniques Uncertainty Modeling (Probabilistic, fuzzy, belief ) Classifiers Clustering Ontologies Multi and Hyper Spectral analysis Fusion GIS Geospatial representation and reasoning Conflation Confidence assessment Text2Sketch Simulation and Modeling Math models Repast Sampling methods Swarm Intelligence

43. HuGeo Project Overview • What’s needed/missing in HuGeo-based Knowledge Discovery component? • Extraction of concepts and events from blogs, twitter, news sources, etc • Interpretation of sentences • Placing them geographically • i.e., we need NLP

44. Conclusions • We model and utilize spatial language • We’re great at modeling, pattern recognition, fusion • Our languages are not very flexible • To take the next steps, we need better and deeper language models • Help!