Software Quality Metrics

1. Software Quality Metrics Analyzing & Measuring Customer Satisfaction (Chapter 14) By Zareen Abbas Reg# 169/MSSE/F07 Usman Thakur Reg# 181/MSSE/F07

2. Overview-Quality Product quality and customer satisfaction together form the total meaning of quality. Customer satisfaction is the ultimate validation of quality. Quality is combination of two things: Product Quality and Customer Satisfaction Small q and Big Q

3. Overview-Quality

4. Overview-Quality • Customer buy not product but assurance • Producer sell their assurance not product • TQM-Total Quality Management sole focus is linking quality with customer satisfaction

5. What is Customer Segment • Every business must decide "what market it is in" - what products and services it offers, to whom they will be offered, in which geographic area and which firms are its competitors • Once these strategic decisions are made, the business can refine "to whom it offers" its products and services into a set of customer or market segments

6. Process Model Approach

7. Why Customer Satisfaction

8. Why Customer Satisfaction • Many studies suggested that enhancing customer satisfaction is the bottom line of business success: • Ever-increasing market competition • Only way to retain the Customers • To expand market share • To gain more profit • To enhance/ improve product satisfaction level • eg. 90% to 95% • Studies show that it is five times more costly to recruit a new customer than it is to keep an old customer: • Why is it costly?

9. Why Customer Satisfaction • It is fact that dissatisfied customers tell • 7 to 20 people about their experiences • While satisfied customers tell • Only 3 to 5 people

11. ISO Quality Policy and Customer Focus

12. C:\Documents and Settings\Administrator\Desktop\Prest Customer satisfaction\Customer Metrics 411 Customer Satisfaction gooood.mht

13. Customer Satisfaction is the crucial requirement for a “good reputation”. Those who have it, will find all doors open • But those who have lost it, frequently perish or have to struggle hard to have their names associated with that magic word “Quality”

14. Customer Satisfaction • Total customer satisfaction is the primary quality issue. • Customers are the only people who can determine total customer satisfaction. • To achieve total customer satisfaction, the organization must know the customer, itself, its product, and its competition.

16. Metrics for Customer Satisfaction Surveys

17. Metrics for Customer Satisfaction Surveys • TQM, more and more companies are conducting surveys to measure their customers' satisfaction. • We will discuss customer satisfaction surveys and the analysis of survey data. • As an example, we describe an analysis of the relationship between overall customer satisfaction and satisfaction with specific attributes for a software product.

18. Why to Surveys for Customer Satisfaction • There are various ways to obtain customer feedback with regard to their satisfaction levels with the products and the company. • Like, telephone follow-up regarding a customer's satisfaction at a regular time after the purchase is a frequent practice by many companies. • Other sources include customer complaint data, direct customer visits, customer advisory councils, user conferences, and the like. • To obtain representative and comprehensive data, representative of the entire customer base.

19. Methods of Survey Data Collection • There are three common methods to gather survey data: • Face-to-face interviews (Personal) • Telephone interviews • Mailed questionnaires • The personal interview method requires the interviewer to ask questions based on a pre-structured questionnaire and to record the answers. The primary advantage of this method is the high degree of validity of the data.

20. Advantages and Disadvantages of Three Survey Methods

21. Sampling Methods • When the customer base is large, it is too costly to survey all customers. Estimating the satisfaction level of the entire customer population through a representative sample is more efficient. • To obtain representative samples, scientific probability sampling methods must be used. There are basic types of probability sampling: • Simple random sampling, • Systematic sampling, • Cluster (group) sampling.

22. How large a sample is sufficient? • The answer to this question depends on the confidence level we want and the margin of error we can tolerate. • The higher the level of confidence we want from the sample estimate, and the smaller the error margin, the larger the sample we need, and vice versa. • For each probability sampling method, specific formulas are available for calculating sample size, some of which (such as that for cluster sampling) are quite complicated. • The following formula is for the sample size required to estimate a population proportion (e.g., percent satisfied) based on simple random sampling:

23. Formula for sampling

24. Formula for sampling • N: Population = 10,000 • Z = confidence level 90 % = 1.65 • p = satisfaction level = 80 % • B: margin of error = 5 % n = 30,000*(1.65)2 x 80(1-80) . 30,000*52 x [1.652 x 80(1-80)] n = ???

25. Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

26. Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

27. Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

28. Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

29. Sample Size (for 10,000 customers) in Relation to Confidence Level and Error Margin

30. Analyzing Satisfaction Data

31. Analyzing Satisfaction Data • Customer satisfaction is often measured by customer survey data via the five-point scale: • Very satisfied • Satisfied • Neutral • Dissatisfied • Very dissatisfied.

32. Analyzing Satisfaction Data • Satisfaction with the overall quality of the product and its specific dimensions is usually obtained through various methods of customer surveys. • For example, the specific parameters of customer satisfaction by IBM include CUPRIMDA: • Capability • Functionality • Usability • Performance • Reliability • Installability • Maintainability • Documentation/information • Availability • HP include FURPS: functionality, usability, reliability, performance, and service

33. Analyzing Satisfaction Data • The data are usually summarized in terms of percent satisfied. In presentation, run charts or bar charts to show the trend of percent satisfied are often used. • We recommend that confidence intervals be formed for the data points so that the margins of error of the sample estimates can be observed immediately

34. Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing.

35. Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing.

36. Quarterly Trend of Percent Satisfied with a Hypothetical Product Traditionally, the 95% confidence level is used for forming confidence intervals and the 5% probability (p value) is used for significance testing.

37. Approaches for Analyzing • Satisfied • Non-satisfied • Usually the 1st metric, percent satisfaction, is used. In practices that focus on reducing the percentage of non-satisfaction, much like reducing product defects, non-satisfied metric is used.

38. Non-satisfied metrics • Although percent satisfied is perhaps the most used metric, some companies, such as IBM, choose to monitor the inverse, the percent non-satisfied. • Non-satisfied includes the neutral • dissatisfied, and very dissatisfied in the five-point scale. The rationale to use percent non-satisfied is to focus on areas that need improvement. This is especially the case when the value of percent satisfied is quite high. • Figure 12.3 in Chapter 12 shows an example of IBM Rochester's percent non-satisfied

39. In addition to forming percentages for various satisfaction or dissatisfaction categories, the weighted index approach can be used. • For instance, some companies use the Net Satisfaction Index (NSI) to facilitate comparisons across product. • The NSI has the following weighting factors: • Completely satisfied = 100% • Satisfied = 75% • Neutral = 50% • Dissatisfied = 25% • Completely dissatisfied = 0%

40. Specific Attributes and Overall Satisfaction Co-relationship between Attributes and overall satisfaction

41. Specific Attributes and Overall Satisfaction • The major advantage of monitoring customer satisfaction with specific attributes of the software, in addition to overall satisfaction, is that such data provide specific information for improvement. • The profile of customer satisfaction with those attributes (e.g., CUPRIMDA) indicates the areas of strength and weakness of the software product. One easy mistake in customer satisfaction analysis, however, is to equate the areas of weakness with the priority of improvement, and to increase investment to improve those areas. • Example Documentation and Reliability

42. On the other hand, customers may not like the product's documentation. To answer the question on priority of improvement, therefore, the subject must be looked at in the broader context of overall customer satisfaction with the product. • Specifically, the correlations of the satisfaction levels of specific attributes with overall satisfaction need to be examined. After all, it is the overall satisfaction level that the software developer aims to maximize

43. Variables • Dependent variable : Overall customer satisfaction • Independent variables: Attribute (Reliability)

44. How to see relationship of attributes with overall satisfaction Two statistical approaches • Least-squares multiple regression (Ordinal) • Logistic regression

45. Logistic Regression • In statistics, logistic regression (sometimes called the logistic model) is used for prediction of the probability of occurrence of an event by fitting data to a logistic curve. • It makes use of several predictor variables that may be either numerical or categorical. For example, the probability that a person has a heart attack within a specified time period might be predicted from knowledge of the person's age, sex and body mass index. • Logistic regression is used extensively in the medical and social sciences as well as marketing applications such as prediction of a customer's propensity to purchase a product or cease a subscription.

46. Logistic Regression • For the logistic regression approach, we classified the five-point scale into a dichotomous variable (categorizes data into two groups) • Very Satisfied • Satisfied 1,2,3 = 0 • Neutral vs • Dissatisfied 4,5 = 1 • Very dissatisfied • Very satisfied and satisfied (4 and 5) versus non-satisfied (1, 2, and 3). Categories 4 and 5 were recoded as 1 and categories 1, 2, and 3 were recoded as 0. The dependent variable, therefore, is the odds (probability) ratio of satisfied and very satisfied versus non-satisfied. • DV: A variable that categorizes data into two groups such as bankrupt versus solvent or energy company versus not energy company. Also called Dummy Variable

47. Logistic Regression • The odds ratio is a measurement of association that has been widely used for categorical data analysis.

48. Ordinary regression • Both approaches overall customer satisfaction is the dependent variable, and satisfaction levels with UPRIMD-A are the independent variables. • The purpose is to assess the correlations between each specific attribute and overall satisfaction simultaneously. • For the ordinary regression approach, we use the original five-point scale. The scale is an ordinal variable. Sensitivity research in the literature, however, indicates that if the sample size is large (such as in our case), violation of the interval scale and the assumption of Gaussian distribution results in very small bias. In other words, the use of ordinary regression is quite robust for the ordinal scale with large samples.

49. Analysis: Ordinary regression

50. Analysis: Ordinary regression