1 / 22

SAMI: Situational Awareness from Multi-modal Input

SAMI: Situational Awareness from Multi-modal Input. Naveen Ashish. Talk Organization. Why are we at RESCUE interested ? Situational Awareness (SA) Introduction System architecture Research challenges Expected outcomes and artifacts Extraction system demonstration. Team. Bhaskar Rao

sherlock_clovis
Download Presentation

SAMI: Situational Awareness from Multi-modal Input

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SAMI: Situational Awareness from Multi-modal Input Naveen Ashish

  2. Talk Organization • Why are we at RESCUE interested ? • Situational Awareness (SA) • Introduction • System architecture • Research challenges • Expected outcomes and artifacts • Extraction system demonstration

  3. Team Bhaskar Rao Mohan Trivedi Rajesh Hegde Sangho Park Shankar Shivappa Naveen Ashish Sharad Mehrotra Nalini Venkatasubramanian Utz Westermann Dmitry Kalashnikov Stella Chen Vibhav Gogate Priya Govindarajan Ram Hariharan John Hutchinson Yiming Ma Dawit Seid Jay Lickfett Chris Davision Quent Cassen Ron Eguchi Mike Mio Jacob Green

  4. Information from Various Sources News, video, audio footage Pushing “Human-as-sensor” Emergency responders People/Victims at disaster GIS, satellite imagery, maps

  5. More Data ≠ More Information Where are the fire personnel ? Have all medical supplies reached ? What areas should we start evacuating first ? SA

  6. Situational Awareness • Wide variety of fields • Beginning in mid-80s, accelerating thru 90s • Fighter aircraft, ATM, Power plants, Manufacturing • Definitions • "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future" • "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take" • Situational awareness and decision making • Areas • Cognitive science • Information processing • Human factors Knowing what is going on

  7. Abstraction of Information Events Multimodal Input: Text, Audio, Video Awareness

  8. First-cut Architecture EVENT EXTRACTION VISUALIZATION and USER INTERFACES KNOWLEDGE: ONTOLOGIES Spatial Indexing PDF Histogram Graph View REFINEMENT Disambiguation Location Text Audio Video Internet Querying and Analysis EVENT BASE RAW DATA Centered around EVENTS as fundamental abstractions

  9. Research Areas Event Modeling Event Extraction Disambiguation GIS Querying Location Uncertainty Graph Analysis

  10. Event Modeling LOCATION TIME NAME LOCATION FROM TO NUMBER TYPE PEOPLE EVACUATION RELIABILITY REPORT AGENCY OPERATION • What is an event ? • Event Representation

  11. Domain Knowledge ROAD EVACUATION THAILAND SOUTHERN REGION ……. PHUKET PHUKET, CHANGWAT • Captured as Ontologies EVACUATION IS-A IS-A AIR EVACUATION

  12. Event Extraction • Long history of information extraction • IR (MUC efforts) • Web data extraction • DARPA ACE • Entities, Relations, Events • Events in 2004 • Event extraction accuracy is still low • SA Domain • Stream of information • Duplicated, ambiguous • Reliability • Conversations • Modalities • Text

  13. Semantics Driven Approach • Semantics Driven • Challenges • Framework • Ontologies • What semantics required for event extraction ? • Application • With NLP, ML techniques • Performance • SA specific • Duplicates, reconciliation, temporal, conversations …..

  14. Disambiguation

  15. Disambiguation

  16. Uncertainty is a Challenge Report 1: “... a massive accident involving a hazmat truck on I5-N between Sand Canyon and Alton Pkwy ...” Report 2: “... a strange chemical smell on Rt133 between I405 and Irvine Blvd ...” • point-location • in terms of landmarks • uncertain, not (x,y) • reasoning on such data • support all types of queries Report 2 Report 1

  17. Implications of Uncertainty in Text How to model uncertainty? • probabilistic model • P(location | report) • e.g. report says “near building A” Queries • cannot be answered exactly... • use probabilistic queries • all events: P(location  R | report) > 0 • SA requirements • triaging capabilities • fast response • top-k • threshold: P(location  R | report) > • -RQ, k-RQ, k -RQ How to map text to probabilities? • use spatial ontologies A B R

  18. Graph Analysis • GAAL • Inherent spatio-temporal properties • Graphs are powerful for querying and analysis

  19. GIS Search Current FGDC Search

  20. GIS Search Progressive Refinement of Data

  21. Deliverables, Outcomes, Artifacts • “Vertical” thrusts • Event extraction system (TEXT) • Disambiguation system • GIS search system • Overall system demonstration ? • “By-products” • Ontologies • Computer science research areas Databases Semantic-Web Information Retrieval Intelligent Agents (AI)

  22. Thank you ! http://sami.ics.uci.edu

More Related