Speaker: Prof. Sten F. Andler Director, Infofusion Research Program

1. Challenges in Information Fusion Technology Capabilities for Modern Intelligence and Security Problems Speaker: Prof. Sten F. Andler Director, Infofusion Research Program University of Skövde, Skövde, Sweden (*) Author: Dr. James Llinas Center for Multisource Information Fusion University at Buffalo, Buffalo, New York, USA llinas@buffalo.edu (*)

2. Key Information Fusion Challenges Driven by Operational Problems and Modern IT • Heterogeneity of Data, Information • Common Referencing and Data Association Impacts • Dealing with Semantics • The Entry of Graphical Methods • Architecting Systems and Analytic Frameworks

3. Heterogeneity of Data/Information Heterogeneity from modern IT capabilities/problems and networked systems Lack of reliable a priori knowledge to support dynamic deductively-based reasoning “Weak Knowledge” problems • Observational • “Hard” Sensor Data and “Soft” linguistic/reported/unstructured Data • Open-source & Social Media • Issues: Mostly in linguistic form; Trust, Volume, Formats, Modalities • Contextual differences • Issues: Format, Middleware reqmt, dynamics, relevance • Ontological differences • Issues: Multiple-ontology cases, semantics, dynamics, relevance • Learned knowledge • Issues: integrating inductive and other inferencing procedures

4. Some Impacts due to Data Heterogeneity • Soft (linguistic) data -- New preprocessing Front Ends: requirement for semantically robust Text Extraction/NLP processes • Marginally available today • If not extracted, properly labeled entities never enter the Fusion process • If not tagged with some level of (reliable) uncertainty/confidence, entity uncertainty not considered • Confounds both Common Referencing and Data Association • Exploiting Contextual Data requires Middleware to condition data in a form useable by Fusion process (native form-to-useable form) • Can also require hybrid algorithms, eg context-aided Kalman Filter designs • In networked systems, there can be multiple Ontological versions being used • Creates a need for ontological normalization (Common Referencing function) • Also impacts Data Association; inconsistent nomenclature will prevent feasible associations • Information learned in real-time creates a Level 4 Knowledge Management functional requirement, and real-time adaptation that can include dealing with out-of-sequence evidence (retrospective adaptation)

5. Some further Impacts regarding Common Referencing and Data Association • Common Referencing • Temporal alignment within streaming Soft data feeds is challenging • Dealing with linguistic tense: past/present/future • Impacts correct Temporal Reasoning • Creates a need for agile Temporal Reasoning • Networked environments open the possibility for inconsistent forms of uncertainty representation • Creates a need for uncertainty transforms, normalization methods • Data Association • Major impact due to Soft (linguistic) data and availability of Relational links • Association now of higher dimension: Entities/attributes and inter-entity Relations — becomes a Graph Association problem • New scoring functions required; egRelational similarity

6. Representative Impacts regarding Common Referencing and Data Association, cont. G. Tauer, R. Nagi, M. Sudit, The graph association problem: Mathematical models and a lagrangian heuristic, Naval Research Logistics (NRL) Volume 60, Issue 3, pages 251–268, April 2013

7. Representative Impacts regarding Graphical Forms and Operations • Graphs as a Representational Form • The standard for language representation • Deals with Entities and Relations • Quantitatively-based; visually manageable • Graph-based Analytics • Framework for Data Association as shown • Evidential searching/matching (supports query-based, discovery-based analysis) • Variety of Graph-Matching paradigms, issues • Stochastic due to tagged uncertainties in graph elements • Incremental to handle streaming real-time data • Large scale to handle “Big Data”; eg Cloud-based

8. Some further Impacts regarding Semantics • Optimal strategies for semantic “control” – control of semantic complexities • Rigorous control of Ontologies • Controlled vs Uncontrolled Languages • Eg Battle Management Language • Robust Text Extraction, NLP • Role of Human Mediators in system architecture • Speed (automation) vs semantic accuracy • Semantic Uncertainty • Vague predicates; issue of Truth—leads to 3-valued forms of Uncertainty Representation

9. Some Impacts regarding System Architectures and Analytical Frameworks • Many problems are “Weak Knowledge” problems wherein the extent of reliable a priori dynamic knowledge about the domain is limited • This motivates an approach that must combine deductive and inductive (or abductive) methods in an effective way • These tend to require technologies that support discovery and learning-based hypothesis-formulation strategies • Methods such as Complex Event Processing, Probabilistic Argumentation, Graph-based Relational Learning are some of the new inferencing methods being studied.

10. Representative Architectures: Inductive + Deductive Earliest Thoughts on Combining Inductive and Deductive Inferencing for Fusion* * Integrating the Data Fusion and Data Mining Processes Ed Waltz, NatlSymp on Sensor and Data Fusion, 2004

11. Representative Architectures: Hard and Soft Fusion Processes; Disparate Analytic Tools Intel Cell – or – Company Opns Intell Support Team • Evidence and Entity -estimate Foraging Services Analytic Support Sensemaking Services Enterprise Service Bus Services • Information (Evidence) Services on n on on Core Enterprise Servces - g g g encin encin (Sensor) Data and Computational Services Hard (sensor) fusion Soft (intel) fusion

12. Summary • Requirements for Data and Information Fusion Processes and Systems have gone far beyond the goal of estimating properties and geometries of entities • Dealing with complex Semantics, inter-entity Relations, Social Media and other Contextual effects, complex Temporal dynamics, and Heterogeneous Data have made the design of IF Systems a markedly new challenge. • Incremental advances and accomplishments are being realized but there is much to be done • Major advances are needed in dealing with more complex inferencing challenges to support efficient learning and discovery processes. • New partnerships are needed across various multidisciplinary areas in order to address these new complexities