Outsourcing Metrics and Benchmarking

1. Outsourcing Metrics and Benchmarking Barbara Beech AT&T Labs Group Manager IT Vendor Management 908-824-9018 bbeech@att.com

2. What Metrics (Service Levels) Are Needed in a Typical IT Contract. What Processes and Procedures are Needed for Review of Metrics. Multi-Supplier impact on Metrics Benchmarking. Improvements Achieved over 5 years. Lessons Learned Along the Way – What Traps to Avoid and What to do Differently. Topics

3. Outsourcing Metrics (or Service Levels)

4. Should cover Operations as well as Development and provide a tie to business process impacts. Typical technical metrics are still needed (e.g. availability, response time). Development metrics are also important (e.g. defects, cost). Even in a BPO that incorporates IT. Incorporate business process metrics if you can. Developing a better link to business metrics Search for existing business metrics that can be incorporated in to a contract Ensure vendors contribution to the metric can be quantified. Consider a “business impact” measure that covers various areas of the business and identifies the system impacts to those areas. Develop criteria for identifying business impacts as high, medium and low. Consider carefully how you will assess the SLA’s Annual, quarterly, monthly. Aggregate or individual. Penalty structure (tiered penalty; 1 miss, 2 miss concept; bonus) What are the Right Service Levels?

5. Metrics Areas

6. Establish Procedures with the Vendor. Changing service levels. Root Cause analysis. Reports delivered and format. Establish a Performance Management Group. This group will review all supplier results and negotiate SLA changes. This group will also validate the detail data and analyze for discrepancies. Review Data Monthly with Suppliers. Set up standard meetings and agendas. Include multiple supplier in these meetings if applicable. Include business partners in review meetings. Ensure You Get Detail Data from the Vendor So You Can Validate Results. This is critical- the vendor needs to know you are looking at the data in detail and analyzing it! Processes & Procedures

7. Factor in Cross Vendor Impacts to your SLA’s existing and new. SLA’s should account for impacts that one vendor can have on another vendor’s application performance. SLA’s should account for impacts that the vendor can have on process that are not considered in-scope to the contract(out-of-scope processes). For Example: Contract covers Customer Care processes and applications but not Billing. Ensure impacts to Billing processes or systems are covered in your SLA’s Consider joint metrics. Joint metrics have each vendor accountable for the same results and eliminate finger pointing. Vendors will work more closely together to resolve problems. You will end up with some similar metrics in contracts Impact of Multiple Vendors on your Service Levels

8. Multiple vendors that all have a role in a system or specific deliverable make obtaining the correct root cause of a problem critical. This ensures the vendor at fault is assessed the missed Service Level. Develop a standard format for documenting Root Causes. Develop a standard procedure for timeframes for receiving root cause analyses from vendors. Develop a standard approval process for approving all Root Cause Analyses. Ensure Root Cause Analysis gets at the true “root” of the problem not just the symptoms(e.g. didn’t find in testing). Ensure preventive action items are included, and follow up on these. Root Cause Analysis- Why this is so important.

9. Benchmarking How this helps

10. Benchmarking Contract Clause • State what will be benchmarked in the contract - Needs to be industry standard so data is available -Be specific about the scope of the benchmarking • Other Items to be considered for the agreement - When and how often benchmarking will be done - Vendor selection process • Who pays for the benchmarking • Most Importantly - Be as specific as possible about the consequences of the results

11. Getting Started with the Benchmarking Process • Determine what is to be benchmarked • -Product • -Physical good • -A service • -Process • -Price • Decide whom or what to compare • -Leadership companies & business functions • -“Apples to Apples” companies & functions • -Insourcing or Outsourcing • -Domestic or Blended Rates (Off Shoring) •  Settle on how the data will be collected • Who builds & maintains the repository • - What needs to be in the repository

12. Things to look for in Selecting a Vendor • Expertise in areas to be benchmarked • Understanding definitions of work, e. g. • - Development and Enhancement • - Maintenance • Scope of Data Base • - Industry & Types of Applications • - Outsource Arrangements • - Domestic versus Off Shore • - Commercial Off the Shelf Comparisons • Benchmarking methodology • Don’t let price be the only determining factor in the selection

13. Data Repository • A robust repository insures an “Apples to Apples” Comparison • Determine the data specifications • -Project details (price, function points, schedule, cycle time) • - List of applications including age • - Platform technology type • - Detailed defect data by Critical & Non-Critical Applications, by release, by month • Needs to match the work in the Benchmarker’s data base • - Activities included in Development • - Requirements, Design, Coding, Testing • - User Acceptance Testing • - Table Updates • - Screen Script Changes • - Activities included in Software Maintenance

14. Benchmark Results • What to do just in case there is a gap between Supplier results and the industry average or performance! • - Understand results and need for change • - Develop and manage an action plan • - Establish metrics or service level agreements with committed • improvements • - Monitor improvements • - Benchmark again to insure continuous improvement

15. System Availability results have improved 58% over 5 years. Software quality has greatly improved. Delivered Defect Density has improved 97% and Residual Defect Density has improved 60% over 5 years. Software development Cost has been reduced 47% over 3 years. Other Service Levels have been maintained while increasing the amount of work. Service Level Improvements Achieved

16. Metrics Lessons Learned • Know what you are outsourcing- have a good application inventory ahead of time. • Establish as many joint metrics as possible if you have multiple suppliers. • Tie metrics to business impacts. • Avoid aggregation of metric results • Avoid averages for metric results – does not get at true results, use “percentage of projects meeting target” instead.

17. Metrics Lessons Learned • Develop metrics that are assessed annually, monthly and quarterly based on the type of metric and impact to the business. • Ensure that the contract gives you enough freedom to add and change metrics annually. You will find you will add and delete various metrics from year to year. • Ensure metric definitions are correct in the contract. Avoid ambiguous language. • Establish cost metrics that incent the supplier to reduce cost both for development and operations.

18. Metrics Lessons Learned • Begin collecting and reporting metrics as soon as possible, even if there is no target. Remember, what gets measured improves. • Do your own validation of vendor results- they will make mistakes. This is very important in the beginning as data is more likely to be incorrect due to inadequate collection procedures. • Benchmark the vendor annually. • Be reasonable. It will help during negotiation.