The Future of Ground Testing

1. The Future of Ground Testing Dr. Ed Kraft AEDC/CZ Presented at the AIAA Ground Test Conference June 29, 2010

2. Key Points • The Nation’s test infrastructure is under serious distress - time to change the dialogue from reactive to proactive • Need to quantify, articulate and improve • Effectiveness • Relevance • Value • People and processes may be more important than facilities for increasing effectiveness and value

3. Current State of Affairs Analogous to Early Civil War Medicine • Cause and effect not understood • Minor wounds could lead to serious consequences • Amputation primary form of surgery • Not a very high survival rate We need to better quantify cause and effect of test capabilities on aeronautical system development outcomes

4. Efficiency vs Effectiveness GAO Program Reviews+ Contractor System T&E Cost (Average $795.6M in FY 2001 $)* + GAO-05-301, Assessment of Selected Major Weapon Programs, March 2005 *Fox et al “Test and Evaluation Costs for Aircraft and Guided Weapons” Rand Project Air Force Report., 2004

5. Acquisition Cycle Time Key T&E Effectiveness Parameter Cycle Time ~ Workload • Workload– Process driven, currently ~22,000 of wind tunnel testing and 13,000 of propulsion cell testing • q (inverse of rework) – Process driven, typically have 10 structural failures found in flight; control surface resizing • Capacity – Budget driven, availability x staffing x throughput q · Capacity 50% reduction in wind tunnel costs equates to just a few tenths of a percent reduction in program costs – Reducing acquisition cycle time by a month could save more than the cost of the entire wind tunnel campaign • Kraft, Edward M. “ Integrating Computational Science and Engineering with Testing to Re-engineer the Aeronautical Development Process,” AIAA Paper 2010-0139 , January, 2010. • Kraft, Edward M. and Huber, Arthur F II “A Vision for the Future of Aeronautical Ground Testing,” The ITEA Journal of Test and Evaluation, Vol 30, No 2, June 2009.

6. Identifying ValueMeasures of Effectiveness and Key Leverage Points in the Acquisition Process CycleTime Systems Delivered Technology Maturity @ Milestone B Suitability Average Fleet Age RDT&E$/Total $ Design Closure @ CDR RDT&E Fraction is revealing metric IOT&E Pause Test Rate Late Defects

7. RDT&E Overrun RDT&E Budget 1 - Fraction of Systems Actually Delivered = 1- Top Line Economic ModelQuantifying the Fraction of Systems Delivered Fraction of Systems Actually Delivered • Final number of systems actually delivered driven by: • Overruns in RDT&E • Changes in the average per unit cost (APUC) during production • Congressional or DoD dictates • Final Budget constraints Fraction of Systems Lost to RDT&E Overrun Budget + 1 Budget - APUC Proc Overrun Procurement “Recurring Cost” - Delta Budget Fraction of Budget RDT&E Overrun Proc Overrun + + RDT&E “Non-Recurring Cost” RDT&E Budget RDT&E Budget fraction amplifies the RDT&E and Procurement overruns plus Budget changes! 0 1 Fraction of Systems Delivered

8. Comparing RDT&E and Procurement Costs to RDT&E Fraction DoD Total Outlay for Improvements(FY08 $, Millions) 0.5 RDT&E Fraction Procurement 0.4 0.3 0.2 RDT&E 0.1 Note: Discreet jumps in RDT&E Fraction are triggered by Procurement downturns, but sustained by relative increase in RDT&E for each Period 0

9. Correlating Time History of Decline in Capacity, Capability, and Competence • Industry consolidation • Reduced capability, investments • Reduced IR&D • Loss of intellectual capital • RDT&E infrastructure • Reduced investments • Staff reductions • Loss of intellectual capital • Reduced staffs • Loss of govt expertise through reductions in hands on experience, staffing,TSPR, etc • Loss of capacity • Availability of infrastructure • Staffing to perform RDT&E • Throughput/Quality All correlate with Macro Cycles and have become deeper declines with each cycle

10. Trend Parameter (Peak Output) IR&D % Industry Cash Flow (22%) % Aerospace to Total R&D S&E (25%) US Aircraft Companies (10) Aircraft Production Lines (30) MRTFB Workyears (40K) AEDC Workyears (4K) Reduced Aerospace Capacity Capacity Trend Data from Multiple Elements of the Aerospace Industry Supportive of RDT&E 1.0 0.5 Fraction of Peak Output 0 70 80 90 00 10

11. Exploring Cause and Effect Simple Dynamic Model • Simple sinusoidal Proc $ with 20 yr period , $90B±$30B • Baseline RDT&E $ expended at 0.25 Acq $ • With perfectly balanced, infinitely elastic capacity RDT&E $ would stay at 0.25 Acquisition $ • Reduced capacity consistent with previous chart • -15% in 70’s, • constant in 80’s @ 85%, • further reduced 25%in 90’s • constant in 00’s @ 60% • Replicates major trends, Total RDT&E $ and RDT&E Fraction escalate after each cycle

12. SummaryNow What Do We Need to Do? • Quantify effectiveness and value • Send me any data / case studies supportive of analysis • Build white papers targeted at key decision makers* • Optimize total investments in capabilities • Optimum local elements ≠ optimum system • Use policy for Major Range Test Facility Base* • Build, validate, and apply tools to increase effectiveness • Integrate computational science and engineering*, fly the mission testing*, DOE response surface methodology to reduce cycle time and late defects • Invest in technical excellence • Train in DOE, CSE, integrated T&E* • Fund hands-on approaches to learn craft* • Maintain a community with critical mass • Change the total development process • Target success at leverage points • Reduce cycle time by >50% Challenge Can the TC Lead This ?

13. Additional Information • Contact me Edward.Kraft@arnold.af.mil • Additional Reading • Kraft, Edward M. “ Integrating Computational Science and Engineering with Testing to Re-engineer the Aeronautical Development Process,” AIAA Paper 2010-0139 , January, 2010. • Kraft, Edward M. “After 40 Years Why Hasn’t the Computer Replaced the Wind Tunnel?” The ITEA Journal of Test and Evaluation, September 2010 • Kraft, Edward M. “DOE Application to Ground Testing – Advances and Challenges” Invited Panel Presentation at the AIAA Air Force T&E Days Conference, Nashville, TN, Feb 3, 2010. • Kraft, Edward M. “Macro-Dynamics of Acquisition,” Draft White Paper available on request • Kraft, Edward M. and Huber, Arthur F II “A Vision for the Future of Aeronautical Ground Testing,” The ITEA Journal of Test and Evaluation, Vol 30, No 2, June 2009. • Huber, Arthur F II, Kraft, Edward M,, Best, John T., Stich, Phillip B., Eldridge, David A., Baker, William B., and Montgomery, Peter A. “Growing Technical Excellence in the AEDC Workforce.” AIAA Paper 2009-1762, February 2009. • Best, John, T., Kraft, Edward M., and Huber, Col Arthur F. “Revitalizing the Technical Excellence at the Arnold Engineering Development Center (AEDC) and Beyond,” AIAA 2008-16115-7, Feb 2008.