Meeting the challenges of unmanned and autonomous system test and evaluation
Download
1 / 34

Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation - PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on

Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation. Thomas Tenorio, Subject Matter Expert for UAST Executing Agent 10 March 2010, USC. Activities. Working Group Roadmapping , Surveys, Networking, Tech Eval BAA Supply Space Surveys UAST Roadmap

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation' - abel-duran


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Meeting the challenges of unmanned and autonomous system test and evaluation

Meeting the Challenges of Unmanned and Autonomous System Test and Evaluation

Thomas Tenorio, Subject Matter Expert for UAST Executing Agent

10 March 2010, USC


Activities
Activities Test and Evaluation

  • Working Group

    • Roadmapping, Surveys, Networking, Tech Eval

  • BAA Supply Space Surveys

  • UAST Roadmap

    • FY2009 Unmanned Integrated Roadmap Findings

    • FY2010 UAST Roadmap

  • Community / Professional Organization Engagement

    • Test Communities

    • Standards Communities

    • ITEA: tutorials and workshops

    • AUVSI

    • INCOSE (International Council of System Engineers)


The test resource management center
The Test Resource Management Center Test and Evaluation

  • TRMC under AT&L

  • UAST part of T&E S&T

JMETC

6.5

CTEIP

6.4

T&E S&T

6.3

UAST

DET

MST

NII

SET

NST


Trmc mission and vision
TRMC Mission and Vision Test and Evaluation

  • Mission

    • "Plan for and assess the adequacy of the…MRTFB…[and] to provide adequate testing in support of development, acquisition, fielding, and sustainment of defense systems; and, maintain awareness of other T&E facilities and resources, within and outside the Department, and their impacts on DOD requirements."

  • Vision

    • The DoD T&E ranges and facilities will be fully capable of supporting the Department with quality products and services in a responsive and affordable manner.

22 ranges

Tagged as National Asset

Major Range and Test Facility Base


Uas test bed and environment
UAS Test Bed and Environment Test and Evaluation

S&T for physical test capabilities associated with Test Bed and Environment

  • Stimulus selection or generation

  • Sensors

  • Data Acquisition and Management

  • Data compression and characterization (signatures)

  • Data reduction/analysis/interpretation

  • Power Technologies: providing increased mission time and capability without increasing the logistics footprint.

  • Test conducting including situation awareness

  • Test operations safety

  • FAA and other civil coordination

  • C4ISR interoperability


Trmc research and interaction
TRMC Research and Interaction Test and Evaluation

  • Test and Evaluation, Science and Technology

  • Three groups organized by TRL level

    • T&E S&T

      • TRL 3-6 or 6.3 funding

      • $95M / year

    • CTEIP

      • TRL 6-9 or 6.4 funding

      • $140M / year

    • JMETC

      • 6.5 funding

      • $10M / year


Trl technology readiness levels
TRL Technology Readiness Levels Test and Evaluation

  • Criteria used to assess project eligibility and status

    • Measures system maturity

    • Non-linear in time, money and effort

    • Not well understood by community


T e s t
T&E S&T Test and Evaluation

  • Test and Evaluation, Science and Technology

  • 7 Focus Areas

    • UAST

    • SET

    • NST

    • NII

    • MST

    • HSHT

    • DET


Unmanned systems integrated roadmap
Unmanned Systems Integrated Roadmap Test and Evaluation

  • The UAS Challenge

    • Capabilities (311 JCA named targets)

    • Systems (138 systems)

    • Performance Envelope Aspects (41)

    • Technologies (17)


Uast augmented study methodology
UAST Augmented Study Methodology Test and Evaluation

Documented (PoR) UAS Missions and Technologies per USIR

Draft Drivers, Use Cases,

Test Concept ,Test Plans

Draft Test Requirements

Refined Drivers, Use Cases,

Test Concept ,Test Plans

Refined and Extended (beyond PoR) UAS Missions and Technologies with Working Group

Validation

T&E Needed to Test Specific UAS Technologies

Facility Specific Descriptions &

Requirements (Hard Numbers)

Test Resource Requirements

Baseline T&E Capabilities (we do not want to conduct an exhaustive survey of all the T&E capabilities that exist and could support UAS T&E)

Tri-Service Baseline Capability

Test Resource Survey

Baseline

Gaps sensitive to New Approaches and New Paradigms

Draft Test Resource

Need Statements

Needs Analysis (Gap)

Refined Test Resource

Need Statements

Refined Needs Statements

Validation

BAA, RFI, White Papers

& Proposals

Develop Solutions

BAAs, etc

Roadmap

Prioritize Solutions

Road-Map


Relevant commentary on test
Relevant Commentary on Test Test and Evaluation


Urgent needs for uas
Urgent Needs for UAS Test and Evaluation


Outcome timelines
Outcome Timelines Test and Evaluation


Means ways and ends to uast as a value proposition
Means, Ways, and Ends to UAST as a Value Proposition Test and Evaluation

UAS Safety, Suitability, Survivability, Effectiveness

Test and Evaluation of UAS as Highly Complex Systems

V&V

What if --?

Integrity

Assessment

Surrogate &

Simulate

Live

Contrived

Live

in situ

UAST Ends

Emulate

Cross-Domain Commonality ……......... Specificity

Space

Air

Ground

Maritime (surface/under)

Predicting Behavior

UAST Ways

Emulating Mission & Environmental Complexity with Assured Safety

Assessing Effects and Capabilities

UAST Tools & Techniques

Reference Data Sets Ground Truth Decision & Behavior

Protocols & Design

Test Bed and Environment

UAST Means

UAST Technologies

Risks

Constraints

Near

Mid

Far


Use case driven investment
Use Case Driven Investment Test and Evaluation


The target for uast
The Target for UAST Test and Evaluation




Meeting the challenges of unmanned and autonomous system test and evaluation

UAS for Operational Necessity (rapid tempo) Roadmap

Evolutionary acquisition with JUTLS & JUONS

Booming capability development

Capability challenge of 311 named systems

Majority system non-PORs

Fielding tech in months: 4-6 months for joint operational necessity

Ensure test capabilities support the fielding of unmanned systems that are effective, suitable, and survivable

  • Test of UAS (increasing tempo for Integrated T&E)

  • Accelerate incrementally improving T&E

  • Augment legacy and improvised capability

  • Emerging arguments for UAST value proposition

  • AS-IS: OT&E emphasis with emergent Joint T&E

  • Fielding UAST in years: 3 year in S&T and 4 years in T&E

UAST: Knowledge

Risk Reduction

UAS: Warfighter

Capability


The interacting communities of uas advantage
The Interacting Communities of UAS Advantage Roadmap

UAS: Warfighter

Capability

UAST: Knowledge

Risk Reduction

UAST S&T:

Next Gen Tech

For UAST


Value proposition of uast
Value Proposition of UAST Roadmap

  • UAST to further ensure Safe, Suitable, Effective, and Survivable UAS

  • Pace and tempo to secure advantage for UAS

  • Advancing Knowledge Generation capabilities for risk reduction in the production of UAS


Uast driver overview
UAST Driver Overview Roadmap

  • UAST has primary drivers

    • Autonomous Behavior of unmanned and autonomous systems that sense, understand and act upon the environment in which they operate

    • Safety of Autonomous systems in Mission and Environment

  • Secondary drivers

    • Sensory Capacity & Perception Loops (Observe)

    • Knowledge Models of Ground Truth & Behavior (Orientation)

    • Decision Making (Decide)

    • Supervised Autonomy Behavior (Action)

  • Context

    • Systems Testing of Human Independent Behavior

    • Emulating UAS in Mission and Environmental Complexity with Assured Safety

    • Assessing UAS Effects and Capabilities in Joint Capability Areas



Leveraging s t for t e capability development
Leveraging S&T for T&E Capability Development Roadmap

Standard Systems T&E

  • System Level T&E

  • Mission & Environment T&E

  • Joint Capability Areas T&E

Autonomous System T&E

  • Predicting Autonomous Behavior

  • Emulating Mission and Environmental Complexity with Assured Safety

  • Assessing UAS Effects and Capabilities

Unmanned & AutonomousSystems Test

UAS Test & Evaluation Focus Areas

Non-IntrusiveInstrumentation

Spectrum Efficiencies

Multi-Spectral Sensors

Netcentric Systems


Uast systems engineering capabilities reference framework
UAST Systems Engineering RoadmapCapabilities Reference Framework

  • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A)

  • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A)

  • Assessing UAS Effects and Capabilities (E&A)

  • Autonomous Test Protocols and Design (T&E)

  • Test Bed and Environments for UAST (T&E)

  • UAST World Models (Ground Truth, Decision, & Behavior) (T&E)

  • Tools and Techniques for Systemic UAST (T&E/E&A)

1

7

3


T e capabilities reference framework
T&E Capabilities Reference Framework Roadmap

1

7

3

1

2

7

4

5

6

*

* Safety Considerations in all 7 Areas

Predicting Unmanned and Autonomous System Behaviors

Emulating Mission and Environmental Complexity for Assured Safety

Assessing UAS Effects and Capabilities

Autonomous Test Protocols and Design

Test Bed and Environments for UAST

UAST Reference Data Sets (Ground Truth, Decision, & Behavior)

UAST Tools and Techniques


Essence of uast challenge
Essence of UAST Challenge Roadmap

  • At >10-fold reduction in

    • Cycle time

    • Cost

  • And exemplary ROI of S&T

  • Determine

    • False Positives and False Negatives

    • Dynamic Limits of behavior

    • Integrity Limits of intended functions

      Across UAS(s) OODA

  • Regarding Mission

    • Effectiveness

    • Suitability

    • Survivability

    • Safety


S t opportunities for t e of autonomy with assured safety
S&T Opportunities for T&E Roadmapof Autonomy with Assured Safety

Tools for Design of Experiments in system, multisystem, and system of systems scenarios considering also implications of mission scenarios, opposition capability, and physical context.

Ability to incorporate UAS design models into warfighter-scope models/simulations in order to anticipate mission suitability, safety, effectiveness and survivability (including countermeasures). 

Determining how to manipulate live physical scenarios including Red Forces. Acquiring ground truth data during actual test operations.

Bayesian Belief Networks and similar tools for conflating test data to mission-level expectations.

Ensuring that Net Centric Systems and relevant test assets are sufficiently agile to enable span and dynamics of UAS test scenarios.

Systems architecting and engineering of a family of composable UAST’s.


Uast systems engineering capabilities reference framework1
UAST Systems Engineering RoadmapCapabilities Reference Framework

  • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A)

  • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A)

  • Assessing UAS Effects and Capabilities (E&A)

  • Autonomous Test Protocols and Design (T&E)

  • Test Bed and Environments for UAST (T&E)

  • UAST World Models (Ground Truth, Decision, & Behavior) (T&E)

  • Tools and Techniques for Systemic UAST (T&E/E&A)

1

7

3

Readiness

Assessment


Readiness assessment capability
Readiness Assessment Capability Roadmap

What: Discover internal bugs and vulnerabilities and external incompatibilities in UAS’s, across UAS’s and in T&E systems.

Where: In executable code, source code, data bases, system models, mission simulations, and SoS configurations.

At development, integration, warfighter and depot locations.

Why: Generate warfighter-confident knowledge. Cut test cycle time and cost in half. User controllable degree of False Positives and False Negatives.

How: Code inspection. Test beds not required.

Mathematically rigorous assessment method and tools.

Enabled by next generation pattern recognition semiconductor chips with throughput ≈ 1 Gb/sec

When: TRL3@2010, TRL5 @2011, TRL6@2012, TRL9@2014


Uast systems engineering capabilities reference framework2
UAST Systems Engineering RoadmapCapabilities Reference Framework

  • Predicting Unmanned and Autonomous System Behaviors (T&E/E&A)

  • Emulating Mission and Environmental Complexity with Assured Safety (T&E/E&A)

  • Assessing UAS Effects and Capabilities (E&A)

  • Autonomous Test Protocols and Design (T&E)

  • Test Bed and Environments for UAST (T&E)

  • UAST World Models (Ground Truth, Decision, & Behavior) (T&E)

  • Tools and Techniques for Systemic UAST (T&E/E&A)

1

7

3

Safety Assurance *


Safety assurance
Safety Assurance Roadmap

What: Discern and Referee the contest between autonomy and safety, both a) test safety, including, e.g., FAA, and b) Operational safety, e.g., fratricide and innocent civilians.

Enable both static and evolving limits.

Assess efficacy of UAS Self-test capability, resilience to cyber threats, probable error in M&S evaluations of UAS(s).

Where: Throughout 5000.02 phases and Warfighter stages. Across UAS, UAS’s, SoS. Spans both on-board and administrator functions.

Why: Avoid unintended consequences of UAS operations. Generate warfighter-confident knowledge. Cut test cycle time and cost in half. User controllable degree of False Positives and False Negatives.

How: A ‘Do No Harm’ OODA loop inside the autonomy loop of both the UAS and the UAST. Method for Preempting UAS behaviors, separate from Planner capability, preferably non-destructive.

When: for autonomy Level 1@2010, 2@2011, 3@2012, 4@2013, 5@2014


Contact information
Contact Information Roadmap

Thomas.tenorio@us.army.mil

(575) 678-4671