Visualization level 2 fusion and homeland defense
1 / 39

Visualization, Level 2 Fusion, and Homeland Defense - PowerPoint PPT Presentation

  • Uploaded on

Visualization, Level 2 Fusion, and Homeland Defense. Dr. James Llinas Research Professor, Director Center for Multisource Information Fusion University at Buffalo [email protected] Outline. Overview of a DARPA-sponsored Workshop on :

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Visualization, Level 2 Fusion, and Homeland Defense' - ivy-richard

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Visualization level 2 fusion and homeland defense

Visualization, Level 2 Fusion, and Homeland Defense

Dr. James Llinas

Research Professor, Director

Center for Multisource Information Fusion

University at Buffalo

[email protected]


  • Overview of a DARPA-sponsored Workshop on :

    • “Ontology Definition and Development, and the Perceptual/Comprehension Interface for Military Concepts”

  • Remarks on Visualization Challenges of Homeland Defense

The workshop ontology action plan perspectives on visualization kesavadas
The Workshop--Ontology Action Plan--Perspectives on Visualization (Kesavadas)

Workshop assertion
Workshop Assertion

  • The Data Fusion community is progressing toward meaningful achievements in Level 2 and 3 fusion processing capability—but there is no community ontology for the L2/L3 products*--a process must be started to assess the need for, nature of, and means to achieve a supporting, consensus L2/L3 Ontology (or Ontologies) that yields the important benefits associated with ontologically-grounded systems, such as Interoperability, Semantic Consistency, Completeness, Correctness, Adaptability, etc

* To include “Threat States”, “Intent”, etc.

Data Fusion Functional Model

(Jt. Directors of Laboratories (JDL), 1993)

Operational Benefits of Multiple SensorData Fusion










  • Multiple

  • Sensors

  • Reliability

  • Improved Detection

  • Extended Coverage

    (spatial and temporal)

  • Improved Spatial


  • Robustness (Weather/visibility, Countermeasures)

  • Improved Detection

  • Improved State Estimation (Type, Location, Activity)

  • CBRN Point and Standoff Sensors

  • CBRN Data Sources

  • Intel Sources

  • Air Surveillance

  • Surface Sensors

  • Standoff Sensors

  • Space Surveillance

  • Multiple



Sensor Mgmt

Process Mgmt

Level 0 — Sub-Object Data Association & Estimation: pixel/signal level data association and characterization

  • Diverse


Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction

State Estimates of Reduced Uncertainty

And Improved Accuracy

Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.

Level 3 — Impact Assessment: [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment

Level 4: Process Refinement: adaptive search and processing (an element of resource management)






Level 0Processing

Sub-object DataAssociation & Estimation

Level 1Processing


Level 2Processing


Level 3Processing



--Combinatorial Optimization

--Linear/NL Estimation



--Control Theoretic


Level 4Processing

Adaptive ProcessRefinement

Data BaseManagement System




Ontology based fusion visualization
Ontology-Based Fusion & Visualization*

Visualization Challenges:

--the Ontology itself (presuming it is large and complex)

--the L2 fusion results (complex, high-dimensional, abstract concepts,

not spatially referenced)

“Raw Data”

(Truly raw and also L1 estimates)

The Results of Which

Provide the Raw Material

For Visualization

Associated to Ontologically-

Based L2 Fusion Process

(Which we don’t have)

* Ontology-based Information Visualization, F. vonHarmelen, et al, Proceedings of the workshop on Visualization of the Semantic Web (VSW'01)", 2001

An Ontology Action Plan for the Information Fusion Community:Results of a DARPA/CMIF Workshop, Nov. 2002

Dr. James Llinas

Dr. Eric Little

Center for Multisource Information Fusion

University at Buffalo

Background Community:

  • Analysis and Decision-Support Needs for New and Diverse defense and national-security problems are demanding major improvements in Level 2 and 3 Information Fusion (IF) capabilities.

  • U.S. and International efforts are underway to address many of the foundational issues associated with achieving such IF capability, especially system architecture and algorithmic processing.

  • However, the topic of Ontological Requirements as a foundation for these L2, L3 initiatives has not been explicitly addressed, although it is agreed that many Ontologically-related activities are underway to include Ontological prototyping but largely addressed from a Computational Ontology point of view.

  • In addition, the abstract nature of many L2, L3 information products also places a demand on the approach to and means for Visualization of such fusion products.

  • In November 2002, a Workshop sponsored by DARPA and the CMIF was held to address these latter two issues.

  • This briefing summarizes thoughts from the Workshop regarding the Ontology topic only.

Ontology track
Ontology Track Community:

A tentative conclusion
A Tentative Conclusion Community:

  • The Data Fusion community is progressing toward meaningful achievements in Level 2 and 3 fusion processing capability—a process must be started to assess the need for, nature of, and means to achieve a supporting, consensus Ontology (or Ontologies) that yields the important benefits associated with ontologically-grounded systems, such as Interoperability, Semantic Consistency, Completeness, Correctness, Adaptability, etc

  • This Workshop opened with the following assertion:

  • This assertion, and the higher-level, implied assertion that “Good Ontologies Yield Good Fusion Systems”, was conditionally accepted by the Workshop attendees.

  • The conditional aspects revolved about the need for some type of experimental proof—there was a consensus on the need for:

    • A Proof-of-Concept Demonstration / Experiment

    • Definition and Employment of Appropriate Metrics and Evaluation Procedures that Quantify:

      • Ontology Quality Per Se

      • “Good” Ontology’s Contribution to Superior Fusion System Performance

  • These activities would comprise just a part of a larger Action Plan.

Ontology related track key issues for an action plan
Ontology-Related Track: Community:Key Issues for an Action Plan

  • An Action Plan for Ontology—What have we learned?

    • Do we agree there is a need for a consensus ontology?

    • Gauging the nature and size of the underlying Taxonomy:

      • The issue of “Admission” to the Taxonomy

      • The issue of the Extent of the Taxonomy

    • Formal Ontological Methods:

      • Degree of formalism required

        • Accommodating a Hybrid approach

        • Research issues

    • Consensus-forming

      • Approach

      • “Configuration Control”, once a baseline is established

    • Construction Methods

      • General approach

      • Automated Tools

Nature and size of the l2 l3 taxonomy
Nature and Size of the L2, L3 Taxonomy Community:

  • Nature: “Admission” to the Taxonomy

    • Coarse Filter: In the main, L2 is about Situational Assessment, and L3 is about Threat and Impact Assessment, and we can easily populate that portion of the taxonomy

    • Fine Filter: To be determined

      • Candidate Approach: Build on the OSD/Decision Support Center’s study of Essential Elements of Information (EEI’s)

        • Cost-Efficient

        • EEI’s well received by operational community

        • Conduct initial analysis before next workshop

      • Incorporate pre-workshop taxonomy

  • Size: estimated as a subset of 3700-long EEI list, TBD

Formality in ontology development
Formality in Ontology-Development Community:

  • Methods for formal ontology development exist—but--

  • Degree of formality fundamentally depends on Ontology Requirements

    • Develop from a Systems-Approach

    • Need to build both application-requirements and technical requirements

      • Application: Requires defining Role for Ontology in IF applications

        • Human understanding

        • Computational benefits

        • Performance/Effectiveness benefits

      • Technical: Requires quantifying technical criteria of goodness:

        • Consistency

        • Completeness

        • Accuracy

        • etc

Selecting the level of formality
Selecting the Level of Formality Community:

Integrated Data Fusion

Dictionary for the designers, users

Computational Ontology suitable for automated reasoning

Ontology suitable for structured data management

from: Deborah McGuiness, “Ontologies Come of Age”

Consensus forming
Consensus-Forming Community:

  • Approach Options Nominated :

    • NATO STANAG-development process

    • Via Int’l Society for Information Fusion (ISIF)

    • U.S. DoD lead but International in scope

  • Link to Computer Science community via:

    • Open Source Consortium

    • IEEE, ACM

  • Link to Int’l Community Required: eg, Canadian and Australian IF communities are addressing Ontological matters; TTCP and NATO both active

  • Broad communication, coordination required:

    • Website(s)

    • VTC’s

    • Use of CSCW technology

    • Specialized Conference sessions

Ontology construction
Ontology Construction Community:

  • Once Requirements have been specified, those reqmts either directly or indirectly influence the overall approach to Ontology construction, eg:

    • Formalism

    • Language

    • Automated Tools

    • Tools for Visualizing the Ontology

    • Strategies for Ontology evaluation

  • In the following we borrow directly from the paper by Anne-Clair Boury-Bisset and M. Gauvin: OntoCINC Server: A Web-based Environment for Collaborative Construction of Ontologies, 19 Sept 2002*

  • Anne-Claire was a workshop attendee and briefed the attendees on the cited topic

Ontology construction approach

1. Community:Identification of the task for which the ontology is being developed;

2. Definition of the requirements for the ontology: purpose and scope;

3. Informal specification: Build informal specification of concepts;

3. Encoding: Formally represent the concepts and axioms in a language;

5. Evaluation of the ontology.

Ontology Construction Approach

1.ID Data Fusion Ontology Task – ID Military Utility

2.Data Fusion Ontology Purpose, Scope, Formality

3.Build Taxonomy; then specify concepts







5. Evaluate DF Ontology




Ontology construction1
Ontology Construction Community:

  • From Boury-Bisset, Gauvin:

Ontology construction2
Ontology Construction Community:

  • From Boury-Bisset, Gauvin:

Ontology construction3

Requirements for an Community:

Ontology-Development Tool

  • Web-based collaborative environment

  • Flexible Meta-model

  • Dynamic configuration of the environment

  • Knowledge-level modelling

  • Ontology editing and discussing

Ontology Construction

  • From Boury-Bisset, Gauvin:

Viewing ontologies
Viewing Ontologies* Community:


Ontology visualization
Ontology Visualization* Community:

* Ontology-based Information Visualization, F. vonHarmelen, et al, Proceedings of the workshop on Visualization of the Semantic Web (VSW'01)", 2001

Visualization of the ontology a consensus development tool need

Ontologies Inherently Community:

Reflect Complex


Visualization of the Ontology Structure is Needed as a Construction Aid

Visualization Tools are

Needed That can Show

Many, Complex


Visualization of the Ontology*:A Consensus Development-Tool Need

* J. Risch of Pacific-NW Battelle was also a workshop attendee and discussed Starlight’s capabilities;

it is a capability reflective of the state-of-the-art in advanced visualization tools

Concurrent information analysis in starlight
Concurrent Information Analysis Community:in Starlight

Summary action plan tasks
Summary Community:Action Plan Tasks

  • Define Participants

  • Begin the Systems Engineering process for Ontology Development

    • Task(s) within an Future Combat System scenario

      • Coordination with CECOM, DARPA

    • Role

      • Coordination with CECOM, DARPA

    • Ontology Requirements to include Formality requirements

      • Define also Visualization-Support Requirements and Visualization Interface

    • Encoding

    • Test and Evaluation

  • Reviewing “master” EEI-set as a foundation for an initial Taxonomy for L2, L3

    • Determine “coarse” and “fine” filters for EEI selection

  • Defining and executing the proof of concept demo

    • Scenario: One of the approved FCS scenarios

    • Metrics and evaluation approach: TBD

    • Scope: TBD

  • Develop an approach to Consensus-forming

    • Coordination with US, NATO, TTCP, ISIF

Visualization challenges of homeland defense
Visualization Challenges of Homeland Defense Community:

  • Homeland defense is protecting a nation-state’s territory, population and critical infrastructure at home by:

  • Deterring and defending against foreign and domestic threats.

  • Supporting civil authorities for crisis and consequence management.

    • Intelligent Response and Recovery

  • Helping to ensure the availability, integrity, survivability, and adequacy of critical national assets.

    • Planning and Mitigation

  • US Army TRADOC White Paper:

Homeland defense and wmd cbrn
Homeland Defense and WMD (CBRN) Community:

  • What’s different about WMD*?

    • Situations not easily recognizable

    • Situations may comprise multiple, phased events

    • Most likely a complex (3D) urban landscape environment

    • Broad repertoire of input sources

      • Typical: Multi-sensor/multi-source

      • Atypical: eg Epi-Intel (human, epizootic, food surety)

    • Responders at high risk; that risk must be factored into response plan

    • Location of incident is a crime scene requiring evidence preservation

    • Subtle contamination-propagation must be accounted for

    • Incident scope may grow exponentially, stressing multi-jurisdictional resources

    • Strong public reaction; fear, panic, chaos, anger

    • Time critical

    • Responder facilities may in fact be targets; eg PSAP’s

* United States Government Interagency Domestic Terrorism Concept of Operations Plan

Homeland defense applications visualization examples wmd and infowar
Homeland Defense Applications Community:Visualization Examples: WMD and InfoWar

Urban landscapes
Urban Landscapes Community:

Urban landscapes1
Urban Landscapes Community:

Urban landscapes2
Urban Landscapes Community:

3d cfd chemical plume dispersion ct analyst @ nrl
3D CFD Chemical Plume Dispersion Community:CT-Analyst @ NRL

Chemical agent dispersion software solutions and environmental services company
Chemical Agent Dispersion Community:Software Solutions and Environmental Services Company

Building internal structures army corps of engrs
Building Internal Structures Community:Army Corps of Engrs

Subway applications argonne natl lab sandia
Subway Applications Community:Argonne Natl Lab + Sandia

Next cmif workshop army sponsored
Next CMIF Workshop: Community:Army-Sponsored

  • “Ontology and Visualization of Data Fusion Concepts: Support to Command and Control in a Network-Centric Warfare Environment “

  • Four Tracks:

    • Evaluation

    • Impacts of the Distributed Environment

    • Notion of Contextual Understanding

    • Homeland Defense Applications

  • Dates: TBD, Summer or early Fall 2003

  • Location: Beaver Hollow Conference Center, Java, NY

Ordnance explosive power from remote sensing oak ridge natl lab
Ordnance Explosive Power from Remote Sensing Community:Oak Ridge Natl Lab