Ali Karimoddini, PhD

1. Developing a data-driven Perception Inference Engine (PIE) for Test & Evaluation of autonomous systems DoD 2015 “Taking the Pentagon to the People” HBCU/MI Technical Assistance Training Greensboro, NC 8 June 2015 Ali Karimoddini, PhD Autonomous Cooperative Control of Emergent Systems of Systems (ACCESS) Lab, Director TECHLAV Center, Deputy Director and leader of Research Thrust 2 Department of Electrical and Computer Engineering North Carolina A&T State University 1601 E. Market Street/524 McNair Hall Greensboro, NC 27411 Email: akarimod@ncat.edu Website: http://eceserver.ncat.edu/akarimod/ Office: 336-285-3847 Fax: 336-334-7716

2. Focused on the mission of the Test Resource Management Center (TRMC) to address T&E needs of Department of Defense (DoD), ACIT Institute has developed a novel data-driven technique for test and evaluation of autonomous systems using an advance fuzzy expert system. Remark: The views and conclusions being discussed here are those of the panelist and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DoD, TRMC, or the U.S. Government. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

3. A sad moment … On October 28, 2014, the first stage of an Antares rocket on an unmanned resupply mission carrying Cygnus CRS Orb-3 failed catastrophically six seconds after liftoff from Mid-Atlantic Regional Spaceport at Wallops Flight Facility, Virginia. The flight termination system was activated just before the rocket hit the ground, but an explosion and fire destroyed the vehicle and cargo. There were no casualties, and Launch Pad 0A escaped significant damage. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

4. Motivations • Testing of Unmanned Systems is required for the Military Departments to be able to certify compliance with regulations and demonstrate safe operations. • Unmanned Systems must meet the same requirements of a manned systems that is intended to be put into service. Challenge: Testing unmanned systems in general is a significant challenge and can be very costly. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

5. What is a right key? Problem Testing Software Model based algorithmic testing Testing complex systems Testing simple systems Solution Set of experiments Software model checking Formal verifcation Data driven techniques Pros Simplicity Powerful for testing software Guarantee the performance Capture complexity and unmodelled behaviors Cons Not scalable and not expandable for complex systems Specific to software and difficult to be used for hardware testing Not applicable to complex systems with unmodelled behaviors Only valid for the trained range A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

6. Sources of complexity: Cyber-Physical nature Cyber-physical systems (CPS) are engineered systems with tight combination of (large number of) interacting computational systems and physical processes. Control Computation Communication A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

7. Project goals: Project goals: Developing a Data-driven Perception Inference Engine (PIE) tool to 1- Infer the internal states of the system from external observations only 2- Evaluateintelligent systems from a cognitive perspective • 3- Predict behavior and evaluate the performance of increasingly intelligent systems • 4- Capture the dynamic, non-deterministic, uncertain behavior of intelligent, autonomous systems A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

8. Assumptions / Challenges Assumptions: Testers may have only limited knowledge of the internal states of the system under test, but can externally observe the behavior of the system Challenges: How to infer the internal states and dynamics of the system from only external observations and how to use this information to evaluate the performance of the system. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

9. Our approach Approach: Creating a tool to enable users to predict the system’s perception, decision-making, and behaviors by integrating • Type 2 Fuzzy Logic System (FLS) We use Type 2 FLS due to its unique capabilities in handling uncertainty and capturing unmodelled emerging behaviors of the system and environment. • Learning Classifier Systems (LCS) We use LCS as a capable machine learning technique to synthesize the data base to form the knwoledge base. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

10. General Structure of the Fuzzy Type-2 Based PIE PIE Decision making and planning Computer LCS T2FLS Actuators Fuzzifier Fuzzifier Inference System Sensing unit (Knowledge base) Rule Base Rule Generator Adjustment Output Process Teleop Unit Defuzzifier Matrix Translation of Fuzzy Rules Command Center Type Reducer A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

11. Our T&E Team at ACIT Institute Principle Investigators: Dr. A. Karimoddini Dr. A. Homaifar Research Associates: Daniel Opoku Graduate Students Muhammad Sohail Alejandro White Nnamdi J. Enyinna Undergraduate Students Evan Olney Vin K Michael Lowe Nicholas Donald Billy Whitehead Emmanuel Arzate A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

12. Acknowledgment Thanks to • Test Resources Management Center (TRMC) • Scientific Research Corporation (SRC) for supporting the NC A&T project on developing a T&E tool for testing and evaluation of unmanned systems. A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems

13. Q & A A. Karimoddini, Developing a data-driven Perception Inference Engine (PIE) for T&E of autonomous systems