DoD Automatic Test Systems Past, Present, Future

1. DoD Automatic Test Systems Past, Present, Future Bill Ross Assistant Director, DoD ATS Executive Agent Office

2. Past • 5+ years ago, the Automatic Test Systems commodity was flagged for DoD-wide management • A DoD Executive Agent for Automatic Test Systems was established (April ‘94) • Why?? • Many weapon system programs were independently developing and fielding ATS • Redundant ATE development efforts • Many different ATE requiring In-Service Engineering • High cost driver- $51B spent on ATS during the ‘80s

3. Two Primary DoD ATS Goals (April ‘94) 1. Reduce total cost of ownership of DoD ATS 2. Provide greater flexibility to the warfighter through Joint Services interoperable ATS

4. DoD ATS Policy(March ‘96) • DoD Acquisition Regulation 5000.2, ATS-specific acquisition policy • Minimize unique types of DoD ATS • Use DoD ATS Families or COTS with defined critical elements • ATS selections shall be the most cost-beneficial to the DoD over the system life cycle • DoD Joint Technical Architecture (JTA), ATS-specific technical architecture design policy • ATS Technical Architecture Subdomain Annex • “Mandates” critical architecture elements

5. DoD ATS Organization DoD Executive Agent for Automatic Test Systems Assistant Secretary of the Navy (RDA) Dr. Buchanan DoD Automatic Test Systems Executive Agent Office NAVAIRSYSCOM PMA260 Capt Fletcher Bill Ross DoD Automatic Test Systems Management Board Joint Service ATS Leaders Capt Fletcher, USN Col Nodine - USAF Col Hamilton - Army Col Gebhard - SOCOM LCol Juliano - USMC

6. ATS Management Board Organization SOCOM Coord LCOL Ragland USMC Coord Mike Heilman USAF Coord Alton Jenkins Army Coord Dawn Gratz EA Office Asst Dir (PMA-260ATS) Bill Ross IPTs/Working Groups ATS EA Office Navy (PMA-260) Capt Fletcher USMC Rep LCol Juliano SOCOM Rep Col Gebhard Army Rep Col Hamilton USAF Rep Col Nodine

7. Active IPTs and Working Groups Leader • Investment Planning IPT Will Broadus • Program Analysis IPT Jim Deffler • TPS Standardization IPT Ed Holland • ATS Research and Development IPT (ARI) Mike Malesich • ATS Modernization IPT Dawn Gratz John Rosenwald • NxTest Working Group Bill Birurakis

8. Productshttp://dodats.osd.mil • DoD ATS Master Plan • Policies, procedures and DOD/Service ATS plans • DoD ATS Handbook • ATS acquisition in “just plain English” • DoD ATS Selection Process Guide • CBA models • Technical, cost data • Procedures • DoD TPS Performance Specification • Replaces Mil-Std-2077 • DoD ATS Architecture Guide • How to implement the DoD ATS architecture

9. Steering a Course Past • Each program did their own thing Present • Selecting standard DoD ATS Families • Defining a commercial-based standard • open system architecture Near Future • Evolve the open system architecture • Services cooperatively define and implement • the common next generation DoD ATS

10. Steering a Course Past • Each program did their own thing Present • Selecting standard DoD ATS Families • Defining a commercial-based standard • open system architecture Near Future • Evolve the open system architecture • Services cooperatively define and implement • the common next generation DoD ATS DoD NxTest

11. NxTest Vision • Services cooperatively Define a single DoD test environment aimed at: • reducing total ownership cost of DoD ATS, and • effecting interoperability of the Services’ ATS functions • And, Services cooperatively Implement this single test environment in a new class of commercial-based ATS characterized by three main critical factors: • Fully embraces the “mandated” DoD JTA requirements and other ARI “emerging” requirements • Gracefully applies legacy test programs • Inserts technology that significantly reduces the amount of hardware requiring support

12. DoD NxTest • What DoD Expects • Open system based on the DoD JTA “mandated” and ARI “emerging” requirements • Interoperable ATS • More easily share TPSs and ATE • Share diagnostics infrastructure • Multiple run-time systems/languages • Embrace existing ATS environments - graceful TPS rehosting • 2/3 reduction in the amount of ATE hardware to acquire and support • Ease future ATE modernization • Parallel test processing and multi-threading to allow true “functional” testing • Better integrated with diagnostics systems and information (network centric) • Design data • Platform diagnostics data • Historical maintenance data • Improved TPS development environments

13. DoD NxTest Initiated • NxTest approach briefed to the ATS Management Board - Oct ‘98 • Highlighted that each of the Services has aging ATS • Recommended cooperatively define the future DoD ATS solution • AMB principals concurred • Joint Service DoD NxTest Working Group established • Industry briefing held March ‘99 • NxTest Working Group: • Defining future test & operational requirements • Assessing available technology

14. DoD NxTest Plan • Technology Demonstrations 1996 - 2003 • Prototype 2001 - 2004 • TPS Regression Testing 2002 - 2006 • Pre-Production 2004 - 2006 • Production 2006 -

15. DoD NxTest Benefits 1. “Reduce total cost of ownership of DoD ATS” • Technology that significantly reduces hardware, thereby reducing operating and support costs • Open system architecture that allows economical technology insertion for meeting future requirements and resolving obsolescence issues 2. “Provide greater flexibility to the warfighter through Joint Services interoperable ATS” • Services single test environment means inherent ATS interoperable hardware and software elements . . . . while improving the quality of test and diagnostics

16. DoD Joint Technical Architecturehttp://www-jta.itsi.disa.mil/ Modeling & Simulation Domain JTA v 3.0 Information Processing Standards Information Transfer Standards Information Modeling Standards Human-Computer Interface Standards Information Systems Security Standards C4ISR Domain • Airborne Reconnaissance • Command & Control • Communication • Intelligence • Information Warfare • Surveillance/Recon Combat Support Domain Weapon Systems Domain • Automatic Test Systems • Acquisition • Finance/Accounting • HR Management • Legal • Logistics/Material • Medical • Aviation • Ground Vehicle • Missile Defense • Munition Systems • Soldier Systems • Missile • Ship Systems • Space Vehicles

17. Joint Technical Architecture • Objective • Provide DoD systems with the basis for seamless interoperability • Define the service areas, interfaces and standards applicable to all DoD systems • Policy • JTA is mandated for the acquisition of new or improved systems throughout DoD

18. JTA ATS Subdomain Annex • ATS Architecture - 22 critical elements • Current JTA - “mandated” requirements: • Interfaces • Instrument Driver API Standards VPP-3.2 • Digital Test Data Formats IEEE-1445 • Data Networking Standards TCP/IP • Instrument Communication Manager Standards VPP-4.3 • Computer to External Environments TCP/IP • System Framework Standards VPP-2 • Reference Models • Hardware Interface Model • Software Models (Runtime View and TPS Development View) • Rules • Test Program to Operating System Calls

19. ARI, JTA, NxTest Architecture System Acquisition DoD ATS Generic Technical Architecture demonstrated success Commercial Standard ATS Open System Architecture NxTest • Commercial standard • ARI (Ind/Gov’t) • Continuously evolving “Mandated” & “Emerging” Critical ATS Elements • Joint Services NxTest Working Group • Implementation may take many forms • DoD JTA • Updated annually  Based on: - Test requirements - Technical innovation - Open system architecture requirements  Based on demonstrated commercial standards

20. Test & Diagnostics Consortiumwww.test-diagnostics.org • AUTOTESTCON ‘98 Plenary topic - positive response • Drafted essential documents • Business Plan, By-Laws, Operating Procedures • Identified potential initiatives • Established access points • Web site, e-mail, telephone line, mailing address • Major benefits • Partnering/leveraging investments • Rapid development and implementations • Membership Call issued recently

21. Integrated Diagnostics • ATS EAO involved in several OSD Integrated Diagnostics studies and demonstrations • Factory to Field Study - Defined an ATS open architecture that facilitates leveraging investments in test software across multiple test platforms • ID Open System Approach Study - Recommended an open system architecture for Integrated Diagnostics • Diagnostics for Acquisition Study - Can existing support infrastructure be used for performance testing commercial electronics replacements?

22. Integrated Diagnostics (con’t) • NDIA Systems Engineering Committee has established an Integrated Diagnostics subcommittee • Howard Savage, IDA, & Steve Hutti, Boeing, Co-chair • Provides a communications path between the ID community and weapon systems designers • Provides an opportunity to interact with the Weapons Systems engineers from DoD and industry at a high level

23. Agenda DoD ATS Past, Present and Future • Bill Ross, Assistant Director, ATS EAO DoD ATS Acquisition Processes and Tools • Jim Deffler, Chair, Program Analysis IPT DoD ATS TPS Standardization • Ed Holland, Chair, TPS Standardization IPT DoD ATS R&D (Open System Architecture) • Mike Malesich, Chair, ATS R&D IPT DoD Next Generation ATS • Bill Birurakis -- Navy, DoD NxTest WG • Ron Weinland -- Army, DoD NxTest WG • John Rosenwald -- Air Force, DoD NxTest WG • Tom Newton -- Marine Corps, DoD NxTest WG

25. DoD Automatic Test Systems Acquisition Processes and Tools Jim Deffler Chair, Program Analysis IPT

26. Past High Cost of Ownership with no Interoperability! • ATS selection typically driven by support requirements of a single weapons platform or system type: • Higher acquisition costs due to limited production runs and redundant system development. • Little vertical and horizontal ATE commonality at the service level. • Little or no thought given to long term life cycle costs: • Each ATE drove its own logistics tail (spares, technical manuals, training, manpower). • Full burden of ATE in-service engineering (parts obsolescence ECPs, performance improvements) absorbed by individual platforms.

27. Present • DoD policy is to minimize unique types of ATS with emphasis on reduced Total Ownership Costs and joint service Interoperability. • Joint Service Program Analysis IPT established to facilitate implementation of DoD ATS Policy across services. • ATS Selection Process established to provide a structured process to make ATS support decisions. • Several tools available through DoD ATS EAO to assist in the DoD ATS Selection Process.

28. Program Analysis IPT Charter • Develop and update the DoD ATS Selection Process Guide. • Review Policy Deviation Requests and Commercial Tester Acquisition Validation Requests. • Provide Weapons System IPTs with assistance and advice in implementing DoD ATS policy. • Preparation of PDRs and CTAVRs • Use of DoD ATS Selection Process Tools

29. DoD ATS Selection • The choice of an effective ATS solution must be made using a rational ATS selection process whenever the following occur: • A weapon system is being developed • The modernization of an ATS is required

30. DoD ATS Selection Process Define Weapon System Support / Test Requirements Identify ATS Alternatives Analyze Selected Alternatives • Test Requirements • Parametric Analysis • (ATE Capabilities vs. • UUT Test Requirements) • DoD ATS Family • Maintenance Requirements • Commercial Tester • Operational Requirements • Existing Service ATS • Cost and Benefit • Analysis • Other DoD Inventory ATS • Combination of above • Operational • Assessment • New Development ATS Select Alternative

31. DoD ATS Selection Process Guide • Provides the DoD Program Manager with the procedures and tools required to implement the requirements of DoD 5000.2R • Process for preparing requests for deviation when the selection process yields a non-family ATS solution • Validation process for selection of a commercial tester. • Criteria for introducing a new ATS family member into the DoD inventory, • Updated annually by the joint service Program Analysis IPT (last update was 26 October 1998). • Available over the WWW at http://dodats.osd.mil.

32. DoD ATS Selection Process Tools • Several tools have been made available through the ATS EAO to facilitate the DoD ATS Selection Process. • System Synthesis Models (SSM+) • NADEP Jacksonville ROM TPS Cost Model • Standard TPS Cost Management System (STCM) • Cost & Benefit Analysis (CBA) Tool

33. System Synthesis Models • To assist in the DoD ATS Selection Process, SSM+ provides the following: • An automated tool for mapping a weapon system’s Unit-Under-Test (UUT) test requirements to ATS within the DoD Family of Testers or any other target ATS Platform. • SSM+ exception reports which provide an assessment of the limitations of the target ATS to fully support the UUT without Interface Device / Interface Test Adapter (ID / ITA) intervention.

34. System Synthesis Models TEST CATEGORY (28 Total) UUT TEST REQUIREMENTS DC Power Supply Pulse Generation Digital Stimulus RF Measurement …... Electro-Optics ATS TEST CAPABILITIES UUT Pin Type (code) Voltage (volts) Voltage Tolerance (volts) Current (amps) Ripple (volts p-p) > Low Voltage (volts) and < High Voltage (volts) • SSM+ contains 28 distinct test categories. • Each test category defined by a set of parametric fields • UUT parametric test requirements • are mapped to ATS test capabilities.

35. TPS Cost Models • Two (2) TPS cost modeling tools are available to the DoD ATS community through the EAO. • NADEP JAX ROM TPS Cost Model provides quick, Rough-Order-of-Magnitude TPS cost estimates in a matter of minutes with limited input data for initial budget forecasting. • Standard TPS Cost Management System (STCM) provides a more detailed cost estimate based on an in-depth assessment of the planned TPS development effort.

36. NADEP JAX ROM TPS Cost Model • The NADEP JAX ROM TPS Cost Model, Version 6.0 operates in a Windows based environment an provides the following: • ROM TPS cost estimates based on minimum user inputs (# of WRAs/SRAs, # of OTPSs, # of Sites, Labor Rates, & Schedule Dates). • Provides a bottoms-up, WBS tasked-based range-estimate of TPS development & production costs. • Provides a statistical cost forecast based on cost data collected from various CASS TPS contracts. • Provides a “Statistical Confidence” range of minimum and maximum values for the TPS cost estimate.

37. Standard TPS Cost Management System • STCM is a fully integrated suite of models which can be consistently applied across all DoD Services and ATS Platforms to provide the following: • A valid and defensible system to provide improved TPS cost estimating and budget forecasting. • An accurate, repeatable, and traceable system for proposal assessment (cost realism) and change assessment. • A system for tracking TPS development contracts and identifying improvement areas for the TPS development process. • Baseline I is currently available over the World Wide Web with enhancements scheduled for FY-00.

38. Standard TPS Cost Management System UUT ANALYSIS MODELS USER INPUTS USER OUTPUTS JAX AUTO- ID MERGE UUT Test Requirements ID Elec Complexity SCHEDULE & COST MODELS Updated Cost & Schedule UUT Mechanical I/F Rqmnts ID Mech Complexity CASPER COST MODULE OTPS Groupings TPS Development Costs broken out by WBS SSM+ WRA Design Data WRA Complexities CASPER COMPLEXITY MODULE SRA Component and Input/Output Data CASPER SCHEDULE MODULE TPS Development Schedule SRA Complexities ATE Station Loading for TPS Development Efforts Government PM Costs broken out by WBS CDRL Item & Review Requirements JAX SHOULD- COST Project Data Government/Contractor Labor Rates Cost of Production OTPSs UUT& ATE Availability for TPS Development TPS Production Schedule Task Update Editor (TPS Development Tool Impact, Late GFE, etc)

39. Sample STCM Outputs (1) Cost Summary Analysis

40. Sample STCM Outputs (2) Detailed Cost Analysis

41. Sample STCM Outputs (3) OTPS Schedules

42. Sample STCM Outputs (4) ATS Station Loading

43. Planned STCM Enhancements • In FY-00, the STCM team plans to incorporate an Earned Value Module. • The new module will provide a Spend Plan, or Budgeted Cost of Work Scheduled (BCWS), as an optional output of the CASPER Cost Module. • This module will provide full earned value reporting at contract and OTPS level based on milestone completions and inter-milestone interpolations. • The Earned Value module will also provide value and trend analysis for cost and schedule variances at contract at OTPS levels.

44. Cost Benefit Analysis Tool • The CBA is the component of the ATS Selection Process to ensure that the ATS chosen is the most cost beneficial to the DoD over the life cycle. • The CBA Tool, developed by ATS EAO in MS Excel format to assist in this process, has the following two major components: • Quantitative (Cost) Factors • Qualitative Factors, Weights, & Analysis

45. CBA Quantitative Factors Reduced Total Ownership Costs! Investment Costs • ATE Non-recurring • ATE Recurring • TPS Non-recurring • TPS Recurring • Initial Training • Interim Support • ATE Support Initial Acquisition Sustaining Costs • Manpower • Sustaining Training • ATE Support/ Maintenance • ATE/TPS In-Service Engineering

46. CBA Qualitative Factors Ease of Use Operational Suitability TPS Transportability Upgradeability Age of ATS Improved ATS Interoperability! • Vertical Commonality • Horizontal Commonality • Life Cycle Supportability • Ease of TPS Development • Adaptability

47. What We’ve Done • Supported numerous WIPTs in implementing DoD ATS Policy in their programs. • Reviewed 23 individual deviation requests and CTAVRs and made acquisition recommendations to appropriate Milestone Decision Authorities resulting in a total cost avoidance of $212M. • Recommended the addition of TETS, JSECST, and GWTS to the DoD Family of Testers to supplement operational and specific testing limitations of CASS and IFTE.

48. What We’ve Done • Made annual updates to the DoD ATS Selection Process Guide to reflect lessons learned. • Captured operational and parametric test requirements from a variety of WIPTs that will be invaluable in defining NxTest as the DoD’s next generation standard family tester.

49. Future Today weapon system support requirements drive ATS test capabilities. Tomorrow ATS test capabilities will drive how we support weapon systems! • Open Systems approach will provide for a common DoD ATS platform capable of supporting all weapons system support requirements across a wide spectrum of operational scenarios. • Designed and developed with use at multiple maintenance levels in mind, providing an opportunity to re-engineer traditional DoD O, I, and D level maintenance concepts to reduce cost and improve efficiency and mission readiness.

50. Summary For more information contact Jim Deffler at DefflerJP@navair.navy.mil or (732)323-1202. • The DoD ATS Selection Process Guide provides direction to make technically sound, cost beneficial ATS support decisions. • The use of ATS Selection Process Tools facilitates consistent and comprehensive ATS selection analyses. • The joint service Program Analysis IPT is available to facilitate implementation of DoD ATS Policy across the services.