ilities Tradespace Workshop Summary Barry Boehm, Supannika Koolmanojwong USC-CSSE ARR 20 March 14, 2013

1. ilitiesTradespace Workshop SummaryBarry Boehm, SupannikaKoolmanojwongUSC-CSSE ARR 20 March 14, 2013

2. Importance of ility TradeoffsMajor source of DoD system overruns • System ilities have systemwide impact • System elements generally just have local impact • ilities often exhibit asymptotic behavior • Watch out for the knee of the curve • Best architecture is a discontinuous function of ility level • “Build it quickly, tune or fix it later” highly risky • Large system example below

3. Importance of Cyber-Physical Systems Major gap in tradespace analysis capabilities • Current ERS, DARPA tradespace research focused on physical system tradeoffs • Range, payload, size, weight, lethality, power and fuel consumption, communications bandwidth, etc. • Some focus on physical modularity, composability • Current cyber tradespace research focused on software, computing, human factors tradeoffs • security, safety, interoperability, usability, flexibility, adaptability, dependability, response time, throughput, etc. • Gaps in capabilities for co-design of hardware and software, integration of tradespace analyses

4. iTAP Startup Results • Top-level ility hierarchy view • Relation to JCIDS combat command user view • Draft survey to compare with acquirer, developer, supporter views • Affordability means-ends framework view • Quantification via COCOMO, COSYSMO, CORADMO models • Architecture-strategy synergies and conflicts views • Conflicts with other capabilities • Change-Oriented Views: Incremental Commitment • The Cones of Uncertainty • MIT Epoch-Era Approach • Agile SE Schedule Acceleration Model • Domain-Oriented Views • Ground: Wayne State, Georgia Tech • Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia

5. SERC Value-Based ilities HierarchyBased on ISO/IEC 9126, 25030; JCIDS; previous SERC research • Individual ilities • Quality of Service: Performance, Accuracy, Usability, Scalability, Versatility • Resource Utilization:Cost, Duration, Personnel, Scarce Quantities (size, weight, energy, …) • Protection: Safety, Security, Privacy • Robustness: Reliability, Availablilty, Maintainability • Flexibility: Modifiability, Tailorability/Extendability, Adaptability • Composability: Interoperability/Portability, Openness/Standards Compliance, Service-Orientation • Composite ilities • Comprehensiveness/Suitability: all of the above • Dependability: Quality of Service, Protection, Robustness • Resilience: Protection, Robustness, Flexibility • Affordability: Quality of Service, Resource Utilization

6. Prioritized JCIDS ilitiesUser View by Combatant Commands: Top priority first • Intelligence, Surveillance, and Reconnaissance • Comprehensive Persistent Survivable Integrated Timely Credible Adaptable Innovative • Command and Control (note emphasis on Usability aspects) • Interoperability Understanding Timeliness Accessibility Simplicity Completeness Agility Accuracy Relevance Robustness Operational Trust • Logistics: Supply • Responsiveness Sustainability Flexibility Survivability Attainability Economy Simplicity • Logistics: Maintenance • Sustainability Responsiveness Attainability Flexibility Economy Survivability Simplicity • Net-Centric: Information Transport • Accessible Capacity Accurate Timely Throughput Expeditionary Latency

7. Means-Ends Framework: Affordability Staffing, Incentivizing, Teambuilding Get the Best from People Facilities, Support Services Kaizen (continuous improvement) Tools and Automation Make Tasks More Efficient Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Eliminate Tasks Affordability Improvements and Tradeoffs Task Automation Model-Based Product Generation Early Risk and DefectElimination Eliminate Scrap, Rework Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Simplify Products (KISS) Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Reuse Components Domain Engineering and Architecture ComposableComponents,Services, COTS Legacy System Repurposing Reduce Operations, Support Costs Automate Operations Elements Design for Maintainability, Evolvability Value- and Architecture-Based Tradeoffs and Balancing Streamline Supply Chain Anticipate, Prepare for Change

8. USC: COCOMO II-Based Tradeoff AnalysisBetter, Cheaper, Faster: Pick Any Two?Slider-based equalizer version being developed (RELY, MTBF (hours)) • For 100-KSLOC set of features • Can “pick all three” with 77-KSLOC set of features -- Cost/Schedule/RELY: “pick any two” points

9. iTAP Startup Results • Top-level ility hierarchy view • Relation to JCIDS combat command user view • Draft survey to compare with acquirer, developer, supporter views • Affordability means-ends framework view • Quantification via COCOMO, COSYSMO, CORADMO models • Architecture-strategy synergies and conflicts views • Conflicts with other capabilities • Change-Oriented Views: Incremental Commitment • The Cones of Uncertainty • MIT Epoch-Era Approach • Agile SE Schedule Acceleration Model • Domain-Oriented Views • Ground: Wayne State, Georgia Tech • Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia

10. Architecture-Based Attribute Trades: Flexibility Example (RT-18a)

11. MIT: ilities in Tradespace ExplorationBased on Lean Aerospace, DARPA research Changeability Enabling Construct: Tradespace Networks Survivability Enabling Construct: Epochs and Eras Value Robustness Set of Metrics

12. Sight Weapon drive drive Turret drive Chassis suspension Running Gear WSU: Versatility Factors and Physical OrganizationComponents that Can be in Different Positions or OrientationsIsolated or Separated Compartments • Mass & Structure Properties • Mass • Angular moments • Imbalances* • Load bearing wall strength • Deck surface area • Interior volumes** • Interior surface areas** *Angular moments of the CG about axes of rotation ** By crew station and compartment

13. Workshop objectives and approach • Workshop objectives • Identify interested collaborators and data • Identify user needs for better reasoning about ility tradeoffs and affordability • Identify improved approaches for cyber-physical system co-design • Workshop approach • Ask participants about their current and likely future challenges and research needs • For ilities and their tradeoffs • For cyber-physical co-design • Prioritize research with respect to strength of need, difficulty

14. Workshop approach • Ask participants about their current and likely future challenges and research needs • For -ilities and their tradeoffs • For cyber-physical co-design • Prioritize research with respect to strength of need, difficulty

15. Participants • Qi Li • Lori Vaughan • Qing Wang • Jing Du • Liming Zhu • Da Yang • Lee Osterweil • Barry Boehm • Shawn Rahmani • Gary Hafen • Winsor Brown • J.D. Baker • Ed Colbert • Thammanoon K. • Peter Suk • Rachchabhorn W. • Sue K.

16. Current and likely future challenges and research needs for -ilities and their tradeoffs • Simulation model to address performance, usability model, comparing -ilities • Define hierarchy, top factors • Quantification of value of each –ilities, matrices • Contribution of each –ilities (H/M/L. option1/2) • Prototype for a short life vs requirements in terms of ilitlities • Scale the tradeoff, scale up? • Affordability (prioritizing issues, not only low cost) • Inter-dependency of -ilities • Pair-wise comparison tool (lockheed) • Issue of subjective tradeoff study • Modeling and trade study, matrices of architecture and –ilities • Healthcare group (right distribution of data vs performance, privacy control/security) • Usability vs design • Parametric diagram • Agile architecture development – tradeoff, architecture your asset & product • How to specify requirements in a measurable way • Security, accessibility, QoS • Scalability – esp. diseconomy of scale, e.g. mobile network • Reliability - 5 9s system in banking sector • Macro view & micro view; complimentary process modeling (agent, task, resources) optimize the resources& others to optimize the choices, a tool that guide resource allocation (TWINS) • Statistical mechanic to understand the process & relationship to COCOMO • How to deploy, data /design selection; tradeoff guideline / tool for system implementation • Architectural style tradeoff; compare & normalize & confidence level; validation tool &Sensitivity analysis tool to improve confidence level • Fault tolerance; process view supporting tradeoff analysis • How many (functional ) requirements is enough; size, # per release , customized development vs maintenance/enhancement • unstated non-functional requirements • Estimation – over/under-estimation • Adaptability & flexibility – adapt to emergent behavior/requirements – not enough representative users • How to help decision makers – can we perform as stated in the proposal • Cyber vs physical; dual cone of uncertainty; • Guidance /model ; how to do the trade/ how to weigh, checklist ; human process part • Process for continuous delivery, how to prioritize requirements and assign them into each release based on revenue (considering dependency & process adoption) • Using checklist – definition of Done(value, cost, quality) • Scenario generators • Product vs program characteristics - executability

17. Prioritization result

18. Afternoon Agenda • Exploring high score -ilities factors • Exploring degree of difficulties • Exploring high score tools • Exploring cyber-physical co-design methods

19. Affordability • Definition = Effectiveness & Cost (INCOSE) • Fixed effectiveness & try to reduce cost • Define cost (Total cost of ownership, life cycle cost) • Cost = including operational & support cost? System cost (e.g. cheap to build to maintain but not cheap to operate) • What is the biggest gap that is not covered when doing affordability analysis? • Technical debt • As long as you are aware of your debt, give you leeway • Technical debt identification tool • HW SW integration

20. Timeliness • Define timeliness • Meet deadline subject to definition of effectiveness (timebox – discard low priority items to meet schedule; agile rebaselining) • Version control issues • Conflict in reusing items

21. Reliability / Fault Tolerance /Safety • Standard assurance issue • Define “safe” • Reliability = impact of defect (loss of life/properties/ $) • Dependent on other systems • Cloud – no full control • How can your system tolerate other systems? • Not equal (reliable <> fault tolerance <> safety) • Acceptable levels • A tool to provide analysis of balancing these factors • KPP – key performance parameter – no single number • Estimating certification cost & schedule • Level of testing

22. Scalability • Scalability of Product • Architecture evaluation tool • Something like static source code analysis tool • Modeling tool • Network traffic & overhead • Communication mechanism • Co-dependent systems (with internet connection? ) • Diversity • Versions or platforms • Horizontal vs Vertical (scale out vs scale up) • Consistency among diversity/nodes • Timing • How to upgrade

23. Security / Privacy • Acceptable levels / level of assurance • Security vs complexity tradeoff (also with reliability, availability) • multiple independent level of security • Certification cost & schedule • Scope of certification (privacy) • Lifetime of data (dynamic data) • Anti-tamper • communication security • Scaling • How to show feasibility evidence • How to justify being assured (how the entire thing is developed) • Proof of correctness • Domain dependent

24. Prioritization result

25. Models & Simulations (Multilevel) • Accuracy / fidelity • VV&A (verification, Validation & accreditation) • Scalability • Modeling environment (jungle/desert) • Modeling language • Visualization • Test cases, usage(scope of use), underlying assumption

26. Architecting • Language (dependent) • Visualization (static & dynamic) • Executable architecture • Analyzable • Multiple views (like DODAF), integration of views • Generation of the system from the architecture • Architecture pattern, reuse

27. Prioritization • Multiple stakeholder value proposition, criteria analysis • Tradeoff • Interdependencies • Cost & schedule • Value estimation • Environment (scenario)