Introduction and Overview of Data and Information Fusion

1. Introduction and Overview of Data and Information Fusion James Llinas Research Professor, Director, Emeritus Center for Multisource Information Fusion University at Buffalo llinas@buffalo.edu

2. CMIF's Approach: A Total Fusion Systems Perspective • Basic through Advanced R&D In: • IF Based Situation and Threat Assessment • Multiple-Sensor System Data Integration/Analysis • Multi-Intelligence Information Environments • COMINT, ELINT, OSINT, HUMINT, IMINT….. • Graph Analysis Methodology and Systems • Applications: • Defense: C4ISR Supportand Tactical Applications • Non-Defense:Disaster Consequence Management, Critical Infrastructure Protection, Medical Informatics Intelligence Analysis Maritime Domain Awareness Situation and Threat Assessment for Multiple Application Domains Focus on High Level Information Fusion and Graph Analytics Contents are CMIF Proprietary

3. CMIF Foundational Sciences and Methods Sampling of Methods Complex Mathematical Optimizations Advanced Techniques In Linguistic Processing Modern Graphical Methods Contents are CMIF Proprietary

4. Collaborative Decision Support:CMIF Command and Control Lab

5. Center for Multisource Information Fusion (CMIF)Flexibility in Research and Development • Core Technology Research & Development • State University of New York at Buffalo, NY, USA • Top level research university and scientific staff— focus is basic research and proof of concept experiments • In existence 20+ years (Average grant revenue ~US$8M) • Collaborative Partner for Technology Transition: • CUBRC, a not-for-profit defense R&D organization in Buffalo, New York, USA (~135 people, ~US$35M) • Cleared, experienced staff, facilities— focus is development and transition Core Technology R&D Broad Base Of Defense R&D UNIVERSITY at BUFFALO Transition & “Hardening” UB & CUBRC

6. History of Information Fusion • Dates to early 80’s—fairly young in the sense of technological history—a maturing technology/field of study • Driven by defense and intelligence needs • Originally as a “data compression” device to digest huge amounts of sensed data as sensors advanced in capability (a “push” requirement) • Later as an important element for decision support (a “pull” requirement) • Matures to very broad range of application • Robotics, medicine, imagery/remote sensing, intelligent transportation, conditioned-based maintenance, biometrics, medicine, etc

7. What is (Automated) Information Fusion? • Information fusion is an Information Process(Software) comprising: • FUNCTIONS: • Alignment • Association, correlation • Combination of data and information from • INPUTS: • Single and multiple sensors or sources to achieve • OUTPUTS: • Refined Estimates of : • parameters, characteristics, events, behaviorsand relations for/among observed entities in an observed field of view • It is sometimes implemented as a Fully Automatic process or as a Human-Aiding process for Analysis and/or Decision Support

8. Data Fusion: Definition " Data Fusion is the process of combining data (or information) for the purpose of estimating or predicting some aspect of the world" Steinberg, Bowman, and White, “Revisions to the JDL Data Fusion Model”, NSSDF 1998

9. Multiple types of data --various types of information --redundant --and complementary) “Associated” or “Correlated” to : --the same object or event or behavior Most Simply-- Observation System So that estimation algorithms (mathematical techniques)—or—automated reasoning methods (artificial intelligence techniques) can produce better estimates (than based on any single type of data) Real World Multiple types of data Related to things of interest To improve estimates about those things Observations (Multiple) Association of Observations Estimation These Basic Ideas are Transferable to Many Types of Problems

10. Basic Role of Fusion Dec-Mkg Analysis etc Estimates Of World States Observational Means (Streaming) Data Association (Dynamic) Real States in the World Common Referencing Alignment Process Refinement Evaluation Actions Requirements driven from here • One means to satisfy user information needs for decision/analysis support, • i.e., most frequently inserted to support human user

11. Everyday Data Fusion Multinodal Fusion Sound Augmented Sensing Smell Taste Images Touch Sensing Association Pain Balance Temperature Body Awareness (Proprioception Robotic Multisensor Fusion

12. DETECTION KINEMATICS CLASSIFICATION CONFIDENCE COVERT COVERAGE RANGE ANGLE CLASS TYPE GOOD GOOD GOOD GOOD GOOD GOOD Sensor Fusion Exploits Sensor Commonalities and Differences, Knowledge of Errors Unknown Moving Object RADAR FAIR POOR GOOD FAIR FAIR POOR EO/IR FAIR FAIR POOR GOOD FAIR FAIR C3I FAIR GOOD FAIR FAIR FAIR FAIR SENSOR FUSION Data Alignment Data Association State Estimation Data Association Uses Overlapping Sensor Capabilities so that State Estimation Can Exploit their Synergies

13. A Persistent Focus: Reduced Uncertainty

14. Ti Tj Tk Tl Need some type of Mapping that determines a good way to allocate Obsvns To Tracks O mi mj mm STATE ESTIMATION & PREDICTION STATE ESTIMATION & PREDICTION STATE ESTIMATION & PREDICTION STATE ESTIMATION & PREDICTION Multisource Data (Evidence) Association M Observations From N Sensors Tracks “T” DATA ASSOCIATION Multiple Observations & Multiple Entities “Assigned” Observations Resulting from some “Best” way to decide which Observations should be “given” to each State Estimator

15. What a “Message” looks like— a Graphical StructureAn Observational Evidence “Atom“--Not a Point Measurement-- An Observation (description—Representation of an Observation) Synonyms Includes judgments as well as observations Multiple Relationships Disconnected semantic fragments Generally all elements have some type of imperfection or error Some errors Quantified

16. Design of the Association Process for Linguistic (Msg) Inputs Good Assignment Solution& Graph Merging Effective Semantic scoring Human Observer 1 Linguistic, Textual (Semantic) Inputs Hypotheses Scored via Semantic Similarity Scores that accnt For uncertainty Hypotheses Evaluation By high-dimensional Assignment problem solution Hypotheses Self-generated by node/arc content Human Observer 2 Pick a node/arc, Search other graph For associable elements (eg exploit ontology) Apply JVC or other Modern assignment Problem solution Smart Graph Search Interdependency with Text, Semantic Operations

17. Situation Weather • Intel Sources • Air Surveillance • Surface Surveillance • Space Surveillance Intelligence Coalition Forces Logistics Imagery Overlays Terrain/Cultural Features Fusion Technology Fusion of Realtime Data and A Priori Data Bases Today—Includes --Sociocultural Info --Social Media “Context” Realtime • Decision-SpecificInformation • Timely • Accurate • Consistent • Structured • Integrated Decision Maker A Priori and Realtime DATA INFORMATION KNOWLEDGE UNDERSTANDING

18. Data Fusion Functional Model (Jt. Directors of Laboratories (JDL), 1993) Operational Benefits of Multiple SensorData Fusion Detection Tracking ID Aggregation Behavior Events Lethality Intent Opportunity • Multiple • Sensors • Reliability • Improved Detection • Extended Coverage • (spatial and temporal) • Improved Spatial • Resolution • Robustness (Weather/visibility, Countermeasures) • Improved Detection • Improved State Estimation (Type, Location, Activity) • Point and Standoff Sensors • Data Sources • Intel Sources • Air Surveillance • Surface Sensors • Standoff Sensors • Space Surveillance Sensor Mgmt Process Mgmt Level 0 — Sub-Object Data Association & Estimation: pixel/signal level data association and characterization Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc. Level 3 — Impact Assessment: [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment Level 4: Process Refinement: adaptive search and processing (an element of resource management) INFORMATION FUSION PROTOTYPE JEM JWARN3 GCCS Level 0Processing Sub-object DataAssociation & Estimation Level 1Processing Single-ObjectEstimation Level 2Processing SituationAssessment Level 3Processing Threat/ImpactAssessment Methods: --Combinatorial Optimization --Linear/NL Estimation --Statistical --Knowledge-based --Control Theoretic • Distributed • Sensors Level 4Processing Adaptive ProcessRefinement Data BaseManagement System SupportDatabase FusionDatabase • Diverse • Sensors State Estimates of Reduced Uncertainty And Improved Accuracy

19. Information Fusion: The Defense Context Multiple types of sensor data Related to things of interest in the Real World To improve estimates about those things Real World • In the defense problem: • Non-cooperative, • Unfriendly • Deceptive “Associated” or “Correlated” to the same object or event or behavior Fusion (Estimation) Techniques !

20. Today’s IF Process Design Environment:Information-space Motivation: Exploitation of all Information Weather Dynamic Real World Financial Sensor Observations Cultural Numbers Human Observations Chat Twitter Web Semantic Label Language Political “HARD” “SOFT” “HARD & SOFT” Contextual Information Observational Data Modern Fusion Process A Priori Dynamic World Model L4 Knowledge Mgmt Declarative Knowledge: Ontologies Learning Processes World State Estimates

21. The Soft Front-end Input Unconstrained Vocabulary (Possibly different languages) (Digitized) Trained Observer Semantics Computational Linguistics, NLP Language Processing Untrained Observer Automated Text Extraction Interview Typical Atomic, Raw Data Input ~ RDF Triples (++) Bystander

22. Soft Data Real World Truth Perceptual and Cognitive Errors in observation Error in oral expression Error in audio capture Error in audio - to - text Conversion conversion Error in text extraction To Common Ref, Data Association Source Characterization Hard Data Calibration (Truth) Target є1 є2 є3 є4 Pd (Obs Params) є5 To Common Ref, Data Association

23. Some Distinctions in Hard and Soft Observational Data Totally distinct from Hard Sensors Philosophy: Relations not directly observable—require reasoning over properties of entities Humans can also judge intangibles --emotional state Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv für Geschichte der Philosophie 83: 225–57.

24. Counterinsurgency Problem Environment MURI Information Fusion Technology Hard Sensing Soft Sensing COIN Decision Support Counter-Insurgency Context * Kinetic Operations Soft Operations * Connable, B, Culture and COIN, www.citadel.edu/.../Connable,%20Culture%20and%20Counterinsurgency%20Brief.ppt

25. Some Remarks on Ontology and Information Fusion Dr. James Llinas Research Professor, Director (Emeritus) Center for Multisource Information Fusion University at Buffalo and CUBRC llinas@buffalo.edu

26. Roles for Ontologies in IF Processes/Systems • Reasonably reliablea priori Declarative Knowledge about some domain • In the face of domains for which reliable a priori Procedural (dynamic) Knowledge is hard to specify • “Weak Knowledge” problems • As such, they provide a framework that connects Entities and Relationships • Of fundamental concern for COIN, Ctr-Terrorism, Irregular Warfare re social structures and militarily-significant entity relations • The basic construct of a “Situation” or a “Threat” and thus Level 2, 3 Fusion estimation

27. Complexities in Distributed and Networked Systems • In modern Distributed/Networked Systems there are No single points of authority: These systems are collages of Legacy systems—Joint/Multiservice systems—Coalition systems • Nodal Ontologies for Fusion/Situational Estimation, and Communication-support Ontologies for Inter-Nodal Communications/Data-sharing (eg JC3IEDM) • Harmonizing NLP Operations and Ontologieswithin and across such systems • The issue of Uncertainty in Ontological specification: • Probabilistic and Non-Probabilistic Ontologies • Is there an Inescapable need for Semantic Mediation? • Mediator systems well-studied and developed* • EgGioWiederhold (June 1, 1993). "Intelligent integration of information". ACM SIGMOD Record22 (2) • (This was a major DARPA program)

28. Semantic Complexity • Controlling Semantic Proliferation/Complexity: • Ontologies • Controlled Languages • Eg Battle Management Language • Eg Shade, U., et al, From Battle Management Language (BML) to Automatic Information Fusion, Chapter in Information Fusion and Geographic Information Systems, Lecture Notes in Geoinformation and Cartography,Popovich, V.V.; Claramunt, C.; Devogele, Th.; Schrenk, M.; Korolenko, K. (Eds.), 2011, Springer • Understanding complexity drivers in text • Eg McDonald, D.D., Partially Saturated Referents as a Source of Complexity in Semantic Interpretation, Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000 • Measuring Semantic Complexity • Eg, Pollard, S and Biermann, A.W., A Measure of Semantic Complexity for Natural Language Systems (2000) Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000

29. The Association Problem • The Ontologically-specified World is controllable—the Real Data World is not • While Ontologies can help in Fusion-based estimation and inferencing problems, the mechanics of exploitation will involve the associability of Real (uncontrolled) data to (controlled) Entities and Relations in the Ontologies • Semantic similarity, metrics, degree (“hops”), etc • Efficient algorithms—eg Cloud implementations • PhD-level research • There is also the issue of “Coverage”—in poorly-understood/known problems, how does one specify an Ontology that has “adequate” coverage? • Issue of negative information

30. Summary • Ontologieshave a useful role in the design and development of Information Fusion systems • Questions regarding issues of: • Authoritative control of semantics in distributed systems • Acceptable, optimal methods for mediation • Complexity of semantics • Understanding, measuring, controlling • Association of semantic terms and complex, high-dimensional semantic structures • Seem to require further, continuing study to better define best ways to employ ontological information in complex, distributed, large-scale Information Fusion systems and applications

31. Unified Research on Network-based Hard and Soft Information Fusion • A CMIF 5-year “Multidisciplinary University Research Initiative (MURI)” Program • Funded by the Army Research Office; ~ $7M • UB/CMIF lead + Penn State + Tenn State • Soft/NLP/Ontology Lead: Prof Stu Shapiro, CSE • Building “TRACTOR” Soft front-end