Sami situational awareness from multi modal input
Download
1 / 22

SAMI: Situational Awareness from Multi-modal Input - PowerPoint PPT Presentation


  • 352 Views
  • Updated On :

SAMI: Situational Awareness from Multi-modal Input. Naveen Ashish. Talk Organization. Why are we at RESCUE interested ? Situational Awareness (SA) Introduction System architecture Research challenges Expected outcomes and artifacts Extraction system demonstration. Team. Bhaskar Rao

Related searches for SAMI: Situational Awareness from Multi-modal Input

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'SAMI: Situational Awareness from Multi-modal Input' - sherlock_clovis


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Talk organization l.jpg
Talk Organization

  • Why are we at RESCUE interested ?

  • Situational Awareness (SA)

    • Introduction

  • System architecture

  • Research challenges

  • Expected outcomes and artifacts

  • Extraction system demonstration


Slide3 l.jpg
Team

Bhaskar Rao

Mohan Trivedi

Rajesh Hegde

Sangho Park

Shankar Shivappa

Naveen Ashish

Sharad Mehrotra

Nalini Venkatasubramanian

Utz Westermann

Dmitry Kalashnikov

Stella Chen

Vibhav Gogate

Priya Govindarajan

Ram Hariharan

John Hutchinson

Yiming Ma

Dawit Seid

Jay Lickfett

Chris Davision

Quent Cassen

Ron Eguchi

Mike Mio

Jacob Green


Information from various sources l.jpg
Information from Various Sources

News, video, audio footage

Pushing “Human-as-sensor”

Emergency responders

People/Victims at disaster

GIS, satellite imagery, maps


More data more information l.jpg
More Data ≠ More Information

Where are the fire personnel ?

Have all medical supplies reached ?

What areas should we start evacuating first ?

SA


Situational awareness l.jpg
Situational Awareness

  • Wide variety of fields

    • Beginning in mid-80s, accelerating thru 90s

    • Fighter aircraft, ATM, Power plants, Manufacturing

  • Definitions

    • "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future"

    • "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take"

  • Situational awareness and decision making

  • Areas

    • Cognitive science

    • Information processing

    • Human factors

Knowing what is going on


Abstraction of information l.jpg
Abstraction of Information

Events

Multimodal Input:

Text, Audio, Video

Awareness


First cut architecture l.jpg
First-cut Architecture

EVENT

EXTRACTION

VISUALIZATION and USER INTERFACES

KNOWLEDGE: ONTOLOGIES

Spatial

Indexing

PDF Histogram

Graph View

REFINEMENT

Disambiguation

Location

Text

Audio

Video

Internet

Querying and Analysis

EVENT BASE

RAW DATA

Centered around EVENTS as fundamental abstractions


Research areas l.jpg
Research Areas

Event Modeling

Event Extraction

Disambiguation

GIS Querying

Location Uncertainty

Graph Analysis


Event modeling l.jpg
Event Modeling

LOCATION

TIME

NAME

LOCATION

FROM

TO

NUMBER

TYPE

PEOPLE EVACUATION

RELIABILITY

REPORT

AGENCY

OPERATION

  • What is an event ?

  • Event Representation


Domain knowledge l.jpg
Domain Knowledge

ROAD

EVACUATION

THAILAND

SOUTHERN

REGION

…….

PHUKET

PHUKET,

CHANGWAT

  • Captured as Ontologies

EVACUATION

IS-A

IS-A

AIR

EVACUATION


Event extraction l.jpg
Event Extraction

  • Long history of information extraction

    • IR (MUC efforts)

    • Web data extraction

  • DARPA ACE

    • Entities, Relations, Events

    • Events in 2004

  • Event extraction accuracy is still low

  • SA Domain

    • Stream of information

    • Duplicated, ambiguous

    • Reliability

    • Conversations

  • Modalities

    • Text


Semantics driven approach l.jpg
Semantics Driven Approach

  • Semantics Driven

  • Challenges

    • Framework

    • Ontologies

      • What semantics required for event extraction ?

      • Application

        • With NLP, ML techniques

      • Performance

    • SA specific

      • Duplicates, reconciliation, temporal, conversations …..




Uncertainty is a challenge l.jpg
Uncertainty is a Challenge

Report 1: “... a massive accident involving a hazmat truck on

I5-N between Sand Canyon and Alton Pkwy ...”

Report 2: “... a strange chemical smell on Rt133 between I405

and Irvine Blvd ...”

  • point-location

    • in terms of landmarks

    • uncertain, not (x,y)

  • reasoning on such data

    • support all types of queries

Report 2

Report 1


Implications of uncertainty in text l.jpg
Implications of Uncertainty in Text

How to model uncertainty?

  • probabilistic model

  • P(location | report)

    • e.g. report says “near building A”

      Queries

  • cannot be answered exactly...

    • use probabilistic queries

    • all events: P(location  R | report) > 0

  • SA requirements

    • triaging capabilities

    • fast response

      • top-k

      • threshold: P(location  R | report) >

      • -RQ, k-RQ, k -RQ

        How to map text to probabilities?

  • use spatial ontologies

A

B

R


Graph analysis l.jpg
Graph Analysis

  • GAAL

  • Inherent spatio-temporal properties

  • Graphs are powerful for querying and analysis


Slide19 l.jpg

GIS Search

Current FGDC Search


Slide20 l.jpg

GIS Search

Progressive Refinement of Data


Deliverables outcomes artifacts l.jpg
Deliverables, Outcomes, Artifacts

  • “Vertical” thrusts

    • Event extraction system (TEXT)

    • Disambiguation system

    • GIS search system

  • Overall system demonstration ?

  • “By-products”

    • Ontologies

  • Computer science research areas

Databases

Semantic-Web

Information Retrieval

Intelligent Agents (AI)


Thank you l.jpg
Thank you !

http://sami.ics.uci.edu


ad