Information fusion technical area overview applications
This presentation is the property of its rightful owner.
Sponsored Links
1 / 38

Information Fusion Technical Area Overview & Applications PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on
  • Presentation posted in: General

Information Fusion Technical Area Overview & Applications. Joseph A Karakowski [email protected] (732)460-7752 November 16, 2011. What we will cover. the “important” parts…..about fusion. Agenda. Background Technology Fusion Definition Fusion Models

Download Presentation

Information Fusion Technical Area Overview & Applications

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Information fusion technical area overview applications

Information Fusion Technical Area Overview & Applications

Joseph A Karakowski

[email protected]

(732)460-7752

November 16, 2011


What we will cover

What we will cover..

the “important” parts…..about fusion


Agenda

Agenda

  • Background Technology

    • Fusion Definition

    • Fusion Models

  • Fusion Technology Sector Applications

    • Military

    • Medical & Non-Military

  • Personal Fusion Areas (Optional)

Questions to be Answered:

What is Fusion Technology and it’s basis?

What are some example fusion applications in specific markets?


Some fusion definitions

Some Fusion Definitions…

…the process of combining data/information to estimate or predict the state of some aspect of the world (Bowman)

…the process of utilising one or more data sources over time to assemble a representation of aspects of interest in an environment (Lambert)

…series of processes performed to transform observational data into more detailed and refined information, knowledge, and understanding (USArmy)

…everything is Connected… a “Global Graph” portrays the connected world; graph nodes are the entities; graph links are the actions or relationships (Walsh)


What is fusion

What is Fusion?

Technical methods/processes which supports, through cognitive/perceptual modeling, the solution of a class “Difficult Problems”

These are a first scientific step to solve these classes of problems, which have not been solvable, up to this time.

Implementation of these processes using information technology, has been moving forward for the last 25+ years, and will probably continue for many more years…

  • Some Typical characteristic of Difficult Problems:

    • Multiple Goals

    • Complexity, with large numbers of items, interrelations and decisions

    • Dynamic, time considerations

    • Cognitive/perceptual problem solving


The blind man the elephant

“The Blind Man & the Elephant”

Question: What is an Elephant?


The fusion elephant

The “Fusion Elephant”

It’s a Cognitive/Perceptual Process!

Question: What is Fusion?

Its Biometric Apps!

It’s Intelligence Apps !

It’s a Global Graph !

It’s the JDL Model !

A State Prediction Problem!


A more realistic fusion elephant

A More Realistic “Fusion Elephant”

Nuclear

This is a new technology, and much RD&E remains


Major fusion process models

Major Fusion Process Models

Many Definitions…and many more models have been proposed and built!

  • Joint Directors of Laboratories Model (JDL)* [1986-Pres]

  • Transformation of Requirements for Information Process (TRIP) Model [2000-?]

  • Visual Situation Assessment Model () [1997]

  • Salerno SA Model [2001-Pres]

  • “Graph” Fusion Model [2005-Pres]

  • Contextual Fusion Model * [2009-Pres]

  • There are many others….


Dfg functional model jdl model

Location;

Attributes

Behavior;

Class; ID

Aggregate

object

refinement

Situation

interpretation

Intent;

Vulnerability

Courses of

Action

DFG Functional Model (JDL Model)

Process

Refinement

Level 4

Source

Input

Human/

Computer

Interaction

Preprocessing/

Predetection

Fusion

Level 0

Single Object

Refinement

Level 1

Situation

Refinement

Level 2

Implications/

Threat

Refinement

Level 3

Database Services

Richard Antony, in DFG Meeting Minutes, W. Doig, 14 March 1997

Relatively static

a priori

Knowledge

Dynamic

Situation

Database


Information fusion technical area overview applications

Antony & Karakowski

Contextual Fusion Model (CFM) 2009

Conceptually organized along three(related) dimensions:

(entity, context1, context2)

AKA “Triple”

“Fusion” ..an “assessment” operation between pairs of Triples:

Lead to 8 fundamental classes of fusion operations

“Fusion as a Process” exhausts all possible “assessment” combinations or fusion in a Triple; the result is a set of discovered concepts & relations from the fusion of the pairs of Triples. This provides a rich discovery space within an existing knowledge source

CFM explicitly fuses diverse context with specific basic entities, all within a computational JDL model framework, resulting in a testable, expandable, and general fusion model


Information fusion technical area overview applications

Information for fusion requires both context and entity

“Entity” is the specific unit of information, or node in a graphical representation

Context allows perception of an Entity with respect to the information of interest

Context gives meaning to an Entity’s “information”

Context is required before an information entity has any meaning

Context must be an integral part of the fusion process & process model, its computation paradigm

Context is knowledge that enhances the more complete understanding of a specific entity of interest and the desired resultant objective information product (s)

Contextual Fusion Model

Context & Content

12


Information fusion technical area overview applications

Two Entities within a graph, with Two sets of Two Entities, as their Contexts

E5/C5

E6/C6

E1

Entities as Context

E4/C4

E3/C3

Graph Entity-Entiy Relations

Entity Graph Nodes

E2

Green = Entity


Information fusion technical area overview applications

Eight Canonical Fusion Forms

Intelligence Traditional Tracking/Correlation Application

Fusion Operation = fusion of two entities with associated context

Fusion Operation = (entity A, location A , time A) x (entity B, location B, time B)

Similarity/Dissim Assessment Op

Contextual Dimensions

Level 1 Fusion

Level 2 Fusion

14


Simple physical context example voltage

Simple Physical Context Example - Voltage

If a voltage(entity) is viewed by itself without any context, we just “see” a value, either static, “semantic” or apparently varying

If voltage has added the context of time, “signals” are created, with the field of electrical electronic engineering and associated signal analysis….

Note the huge information content difference between the entity of “voltage” and the addition of the context “time” and how context gives much more “meaning” to the entity (voltage)

Voltage

.

time

Human Entity, for example, a much more complex entity…. This is like a generalization from “humans” to “human signals”


Information fusion technical area overview applications

Eight Canonical Fusion Forms

Source “Voltage” (just for fun)

Fusion Operation = ( Source V1, location V1 , time V1) x (Source V2, location V2, time V2)

Contextual Dimensions

Level 1 Fusion

Level 2 Fusion

16


Information fusion technical area overview applications

Prior Art: Military Target Entity Model

DRs of Organization to Organization

DRs of Organizations to Events

Level 2

DRs of Events to Events

Organizations

Events

DR: Discovered Relations thru contextual FFs

All Relations based on Location & Time Context Only

DRs of Individuals to Events

Level 2

DRs of Individuals to Individuals

DRs of Individuals to Organizations Level 2

Individuals

Other forms of discovery are possible;

(I O, OE )  (EI ) eg

17


Fusion applications

Fusion Applications


Some fusion military application areas

Some Fusion Military Application Areas

  • Intelligence

  • Bio sensing/biometrics

  • Situation Awareness

  • Imagery

  • SIGINT(COMINT/ELINT)

  • Tracking

Can support at all levels: Hardware, Software, Level 0 – Level 5


Non military fusion application areas

Non-Military Fusion Application Areas

  • Networking/Cellular

  • Homeland Security

  • Medicine

  • Chemistry

  • Cognitive sciences

  • …many others…


Fusion topics from a recent conference

Fusion “Topics” from a recent conference…


Conference shortened c index of fusion topics i

Conference [Shortened-”C”] Index of Fusion Topics I

  • Camera

  • Capability Acquisition Graph

  • CBRN data fusion

  • Cellular automata

  • Centralized processing systems

  • Challenge Problem Set

  • Change detection

  • Chemical plume

  • Classification fusion

  • Classification System

  • Closest point approach

  • Clustering algorithm

  • Clutter

  • Co-ranking

  • Coalition formation

  • Coalition operations

  • Coarsening

  • Coastal radar

  • Cognitive Radio Networks

  • Collaborative systems

  • Collision mitigation

  • Color Clustering

  • ……


Conference shortened c index of fusion topics ii

Conditional independence

Confidence management

Configuration

Conflict analysis

Confusion

Conjunctive operator

Connection Model

Context

Contradiction

Convex optimization

Convoy tracking

Cooperative systems

Coordinate registration

Conference [Shortened-”C”] Index of Fusion Topics II

  • Combination of belief functions

  • Combinatory categorial grammar

  • Communication Decision

  • Communication failures

  • Complex object recognition

  • Compression

  • Computer security

  • Conceptual graphs


Conference shortened c index of fusion topics iii

Coordination

Correlation

Course Of Action

Covariance control

Credal networks

Credibility

Crop modeling

Cross correlation

Cross-cueing

Cubic Spline Curve

Cued Sensors

Cyber fusion

Cyber-security

Conference [Shortened-”C”] Index of Fusion Topics III

From these three slides one can see both very specialized areas and much broader areas which currently utilize information Fusion technology


Overview of specific if apps from selected market areas

Overview of Specific IF Apps from Selected market areas

Military

1…Biometrics

2…Target Detection & Tracking

3…Chemical & Explosives

4…Image Fusion

Medical

5…Breast Cancer

6…Radiology

Non-Military

7…Dept of Homeland Security

8…Cyber Security

Note: All these apps will fall somewhere in the fusion models and fusion definitions which I previously described.

Summaries of specific fusion papers follows…


1 biometrics

1. Biometrics

A Multibiometric Face Recognition Fusion Framework with Template Protection[1]

  • “A fusion framework.. which demonstrates how …algorithms that produce hard decisions can be combined with unprotected algorithms that produce scores or soft decisions”

  • Improving the recognition of fingerprint biometric system using enhanced image fusion[2]

  • “approach to increase the verification and identification of fingerprint recognition. This was achieved by using … linear fusion techniques”

  • Multimodal Eye Recognition[3]

  • “results show that the proposed eye recognition method can achieve better performance…, and the accuracy of…kernel-based matching score fusion methods is higher than PCA and LDA”

Military & Commercial


2 target detection tracking

2. Target Detection &Tracking

Level 0-2 fusion model for ATR using fuzzy logic[4]

  • “use of fusion at the lowest levels has been demonstrated. …provides a structure for fusion of multispectral data at all levels”

  • Long-duration Fused Feature Learning Aided Tracking[5]

  • “Our experiments indicate that the Long-term Hypothesis Tree algorithm, which solves the tracklet-to-tracklet association problem, can be used to strongly disambiguate a multitude of situations and is a more computationally efficient algorithm than previously proposed joint solutions”

Military


3 chemical explosives

3. Chemical & Explosives

Fusing chlorophyll fluorescence and plant canopy reflectance to detect TNT contamination in soils[6]

  • “physiological response of plants grown in TNT contaminated soils and … to detect uptake in plantleaves…use remote sensing of plant canopies to detect TNT soil contamination prior to visible signs”

  • Sensor data fusion for spectroscopy-based detection of explosives[7]

  • “Multi-spot fusion is performed on a set of independent samples from the same region…. Furthermore, the results … are fused using linear discriminators. Improved detection performance with significantly reduced false alarm rates is reported using fusion techniques”

Military Market


4 image fusion

4. Image Fusion

Towards Visual-Data Fusion[8]

  • “Fusion for both data and visual processes are derived as specific transforms from human linguistic requests. Visual “understanding” occurs by human-directed perception of summarized pattern representations within a familiar frame of reference”

  • An orientation-based fusion algorithm for multisensor image fusion[9]

  • “Gabor wavelet transform … to fuse visible images and thermalimages; orientation-based fusion is superior to the results of multiscale fusion algorithms…and can be applied to multiple (more than two) image fusion”

Military & Commercial


5 breast cancer

5. Breast Cancer

Investigation of PET/MRI Image Fusion Schemes for Enhanced Breast Cancer Diagnosis[10]

  • “results indicate that the radiologists were better able to perform a series of tasks when reading the fused PET/MRI data sets using color tables generated by our new genetic algorithm, as compared to commonly used …schemes”

  • Time of Arrival Data Fusion Method for Two- Dimensional Ultrawideband Breast Cancer Detection[11]

  • “A new microwave imaging method is given for breast tumor detection using an ultrawideband (UWB) imaging system. By combining the time of arrival (TOA) measurements from different sensors, the presence and location of small malignant lesions can be identified”

Medical


6 radiology

6. Radiology

KNOWLEDGE BASED FUZZY INFORMATION FUSION APPLIED TO CLASSIFICATION OF ABNORMAL BRAIN TISSUES FROM MRI[12]

  • “automatically classify abnormal tissues in human brain in a three dimension space from multispectral magnetic resonance images such as TI-weighted. T2- weighted and proton density feature images. It consists of four steps: data matching. information modeling, information fusion and fuzzy classification”

  • New Applications of Planar Image Fusion in Clinical Nuclear Medicine and Radiology[13]

  • Fusion of multiple modalities has become an integral part of modern imaging methodology, especially in nuclear medicine where PET and SPECT scanning are frequently paired with computed tomography(CT). Additional fusing of orthopedic radiographs with photographic images of the extremities..

Medical – Add


7 dhs

Military & Non-Military

7. DHS

Information Fusion for CB Defense Applications[14]

  • “With appropriate algorithmic approaches and appropriately resolved tradeoffs, information fusion can offer… the potential of reaching performance that would be difficult, if not impossible, to attain otherwise. Thus, information fusion represents a significant opportunity for the CB defense and homeland security realm”

  • Decision-level Information Fusion to Assess Threat Likelihood in Shipped Containers[15]

  • “details an approach to the decision-level fusion of disparate information to produce an assessment of the presence of a threat in a shipping container”

  • Homeland Security Fusion Application of STEF[16]

  • “fusion system provided sufficient actionable intelligence that could have stopped a .. realistically staged terrorist attack on a US civilian target. …provided sufficient information to allow .. arresting the mastermind of the plot, as well as other key individuals and detaining the lower level individuals in his network, including the suicide bomber”


8 cyber security

8. Cyber Security

Application of the JDL Data Fusion Process Model for Cyber Security[17]

  • “explores the underlying processes identified in the Joint Directors of Laboratories (JDL) data fusion process model and further describes them in a cyber security context”

Military & Non-Military


We have covered the more important parts a warning

We have covered “the more important parts”…a warning


Summary closing comments

Summary & Closing Comments

  • Short background of Fusion Technology & Models/Contextual Fusion Model

  • Few examples of Fusion R&D / Apps

  • A lot was left out ….


Backups

Backups


Some of my personal fusion rde areas

Some of My Personal Fusion RDE Areas

  • UGS tracking/ID L0/L1

  • RADAR ID

    • Signal processing L0/L1 / Fuzzy Expert

    • Confirmation/Disconfirmation

  • Voice Fingerprint ID biometrics L0/L1

  • Visual fusion L0-L2[*]

  • Semantic/contextual unstructured information -- understanding & discovery L1-L3[*]

  • Contextual Fusion System[2006-2010]

  • General Context Fusion Model [2011-?]


Some fusion publications

Some Fusion Publications

  • Karakowski, J.A., “An Application of Text-Independent Speaker Recognition to High Speed Voice Surveillance”, Wide Area Surveillance Symposium, Office of Nat’l Drug Control Policy/Counter Drug Technology Assessment Center(1993)

  • Karakowski, J.A., “Text Independent Speaker Recognition using A Fuzzy Hypercube Classifier”, ICASSP97(1997)

  • Karakowski, J.A., “Towards Visual Fusion”, Invited Paper, Georgia Tech(1998).

  • Antony, R. T. and Karakowski, J. A., “Service-Based Extensions to the JDL Fusion Model,” SPIE Defense Security and Sensing Conference (March 2008).

  • Antony, R. T. and Karakowski, J. A., “Fusion of HUMINT & Conventional Multi-Source Data,” National Symposium on Sensor and Data Fusion, Session SC04 pp. 1-16 (07).

  • Antony, R. T. & Karakowski, J. A., (2007) “Towards Greater Consciousness in Data Fusion Systems,” MSS National Symposium on Sensor and Data Fusion, (June 07).

  • Antony, R. T. and Karakowski, J. A., “First-Principle Approach to Functionally Decomposing the JDL Fusion Model: Emphasis on Soft Target Data,” Fusion (July 08).

  • Antony, R. T. & Karakowski, J. A., “Discovery Tools for Soft Target Applications,” National Symposium on Sensor and Data Fusion(2009)

  • Antony, R. T. and Karakowski, J. A., “First-Principles Mapping of Fusion Applications into the JDL Model,” SPIE Defense Security and Sensing Conference (April 2009)

  • Antony, R.T, & Karakowski, J.A., “Multiple Level-of-Abstraction Tracking and Alias Resolution”, National Symposium on Sensor and Data Fusion(2010)

  • Antony, R.T., & Karakowski, J.A., “Toward more Robust Exploitation of the Asymmetric Threat: Binary Fusion Class Extensions”, (April 2011) SPIE.


  • Login