COTS Metrics and Modeling Working Group

1. COTS Metrics and Modeling Working Group Chris Abts, Vic Basili, Jim Cannon, Brad Clark, Nancy Eickelmann, Peter Hantos, Keun Lee, Gary Thomas, Lori Vaughn

2. What should I measure? • In what activities did I spend my time, money? • In what phases did I spend my money? • What problems are we having? • What advantages does the vender give us over our internal staff? • What are the vender capabilities? • How do I characterize the environment? • How do I characterize the volatility of the system? • How do I characterize the volatility of each COTS product? • How do I aggregate that volatility?

3. Engineering Activity Area Evaluation Construction Configuration Management Deployment and Sustainment System Context Architecture and Design Marketplace Business Activity Area Vendor Relationship Intergovernmental Supplier Relationships COTS Business Case COTS Cost Estimation SEI Activity Sets Ref: "An Activity Framework for COTS-Based Systems,” CMU/SEI-2000-TR-010, by Oberndorf, Brownsword, Sledge.

4. Possible Effort Model Outputs • Amount of time we should spend on evaluation • Product (effort) • Vendor (rating) • Confidence in our assessment • Risk exposure associated with the decision

5. Product Evaluation Effort Model Inputs(In what activities did I spend my time, money?) How much effort does it take? • Number of potential products available • Technology maturity / volatility • Number of features to be evaluated • Number / complexity of interfaces • Number of other COTS products it has to interface to • Criticality to system • Organization maturity with COTS based-development • Vendor Rating How much confidence do we have in our answers

6. Vendor Relations Effort Model(What is the Vendor’s influence on my evaluation effort?) Inputs • What size of the vendor’s market share does your need represent? • Vendor’s market share • Vendor’s product maturity • Criticality of COTS product to system • Vendor responsiveness • Trade exposure • Vendor level of support (training, help desk, tech. support) How much confidence do we have in our answers

7. Risk Model for Exposure(How much resource should I invest?) Risk Exposure Effort Spent in Evaluation

8. Evaluation Effort Model Discussion Effort to evaluate = F (#potential products to evaluate, # interfaces, #features) But most important factor is organization maturity with COTS Meta Model (what have you got available)

9. Priority Chart for Models H X SIZE X Meta Model X Risk Exposure Importance M X Vendor Evaluation X Product Evaluation L L M H Difficulty

10. Conclusion • There is a lot we don’t know about how to conduct a COTS based project so that it ends successfully • Modeling activity is very important to project success • How much time should I spend doing an activity? • What happens when I spend more or less time doing an activity • Building activity models is very hard • Hard to identify a primary relationship between effort and something else (e.g. size and effort in COCOMO II) • Lots of different things affect effort in different activities • Little information is collected in addition to effort (if effort is collected at all down to the activity level)