SENG 530: Software Verification and Validation

1. SENG 530: SoftwareVerification and Validation V&V Processes and Techniques Prof. Bojan Cukic Lane Department of Computer Science and Electrical Engineering West Virginia University

2. Overview • Software Inspections. • 02/14/2002 • Software Metrics. • Today • Software Reliability Engineering. • 02/28/2002

3. Agenda • Software Engineering Measurements. • Measurement Theory. • A Goal-Based Framework for Software Measurement. • Verification and Validation Metrics.

4. Measure? Why? • Developers angle. • Completeness of requirements, quality of design, testing readiness. • Managers angle. • Delivery readiness, budget and scheduling issues. • Customers angle. • Compliance with requirements, quality. • Maintainers angle. • Planning for upgrades and improvements.

5. Measurement • A process by which numbers (symbols) are assigned to attributes of entities in the real world in such a way to describe them according to clearly defined rules. • Measurement process is difficult to define. • Measuring colors, intelligence is difficult. • Measurement accuracy, margin of errors. • Measurement units, scales. • Drawing conclusions from measurements is difficult.

6. Measurement (2) • “What is not measurable make measurable” [Galileo, 1564-1642]. • Increased visibility, understanding, control. • Measurement: • Direct quantification of an attribute. • Calculation: • Indirect, a combination of measurements used to understand some attribute. • (Ex. Overall scores in decathlon).

7. Measurement in Software Engineering • Applicable to managing, costing, planning, modeling, analyzing, specifying, designing, implementing, verifying, validating, maintaining. • Engineering implies understanding and control. • Computer science provides theoretical foundations for building software, software engineering focuses on controlled and scientifically sound implementation process.

8. Measurement in Software Engineering • Considered somewhat a luxury?!? • Weakly defined targets: “Product will be user-friendly, reliable, maintainable”. • Gilb’s Principle of Fuzzy Targets: “Projects without clear goals will not achieve their goals clearly.” • Estimation of costs. • Cost of design, cost of testing, cost of coding… • Predicting product quality. • Considering technology impacts.

9. Software Measurement Objectives • “You cannot control what you cannot measure.” [DeMarco, 1982] • Managers angle. • Cost: Measure time and effort of various processes (elicitation, design, coding, test). • Staff productivity: Measure staff time, size of artifacts. Use these for predicting the impact of a change. • Product quality: Record faults, failures, changes as they occur. Cross compare different projects. • User satisfaction: Response time, functionality. • Potential for improvement.

10. Software Measurement Objectives (2) • Engineers angle. • Are requirements testable? • Instead of requiring “reliable operation”, state the expected mean time to failure. • Have all faults been found? • Use models of expected detection rates. • Meeting product or process goals. • <20 failures per beta-test site in a month. • No module contains more than x lines (standards). • What the future holds? • Predict product size from specification size.

11. The scope of software metrics • Cost and effort estimation. • COCOMO 1, Function points model, etc. • Effort is a function of size (LOC, function points), developer’s capability, level of reuse, etc. • Productivity models and measures. • Simplistic approach: Size/Effort. • Can be quite misleading, even dangerous.

12. A productivity model

13. The Scope of Metrics (2) • Data collection • Easier said than done. • Must be planed and executed carefully. • Use simple graphs and charts to present collected data (see next slide). • Good experiments, surveys, case studies are essential.

14. Presenting collected data

15. The Scope of Metrics (3) • Quality models and measurements. • Quality and productivity models are usually combined. • Advanced COCOMO (COCOMO II), McCalls model. • Usually constructed in a tree-like fashion. • At a high level are indirect factors, at a low level are directly measurable factors.

17. The Scope of Metrics (4) • Reliability models. • Performance evaluation and modeling. • Structural and complexity metrics. • Readily available structural properties of code (design) serve as surrogate for quality assessment, control, prediction. • Management by metrics. • Many companies define standard tracking and project monitoring/reporting systems. • Capability maturity assessment.

18. Agenda • Software Engineering Measurements. • Measurement Theory. • A Goal-Based Framework for Software Measurement. • Verification and Validation Metrics.

19. The basics of measurement • Some pervasive measurement techniques are taken for granted • A rising column of mercury for temperature measurement was not so obvious 150 years ago. • But, we developed a measurement framework for temperature. • How well do we understand software attributes we want to measure? • What is program “complexity”, for example?

20. The basics of measurement (2) • Are we really measuring the attribute we want to measure? • Is the number of “bugs” in system testing a measure of quality? • What statements can be made about an attribute” • Can we “double design quality”? • What operations can be applied to measurements? • What is “average productivity” of the group? • What is “average quality” of software modules?

21. Empirical relations

22. Empirical relations (2) • “Taller than” is an empirical relation. • Binary relation (x is taller than y). • Unary relation (x is tall). • Empirical relations need to be mapped from real world into mathematics. • In this mapping, real world is the domain, mathematical world is the range. • Range can be the set of integers, real numbers, or even non-numeric symbols.

23. Representation condition • The mapping should preserve real world relations. • A is taller than B iff M(A) > M(B). • Binary empirical relation “taller than” is replaced by the numerical relation >. • So, “x is much taller than y” may meanM(x)>M(y)+15.

24. Stages of formal measurement

25. Agenda • Software Engineering Measurements. • Measurement Theory. • A Goal-Based Framework for Software Measurement. • Verification and Validation Metrics.

26. The framework • Classifying the entities to be examined • Determining relevant measurement goals • Identifying the level of maturity reached by the organization

27. Classifying software measures • Processes are collections of software related activities. • Associated with time, schedule. • Products are artifacts, deliverables or documents that result from a process activity. • Resources are entities required by a process activity.

28. Classifying software measures (2) • For each entity, we distinguish: • Internal attributes • Those that can be measured purely in terms of the product, process or the resource itself. • Size, complexity measures, dependencies. • External attributes • Those that can be measured in terms of how the product, process or the resource relate to their environment. • Experienced failures, timing and performance.

29. Process • Measures include: • The duration of the process or one of its activities. • The effort associated with the process or activities • The number of incidents of the specific type arising during the process or one of its activities. • Ave. cost of error=cost/#errors_found.

31. Products • External attributes: • Reliability, maintainability, understandability (of documentation), usability, integrity, efficiency, reusability, portability, interoperability… • Internal attributes • Size, effort, cost, functionality, modularity, syntactic correctness.

32. Product measurements • Direct measure example: • Entity: Module design document (D1) • Attribute: Size • Measure: No. of bubbles (in flow diagram). • Indirect measure example: • Entity: Module design document (D1, D2,…) • Attribute: Average module size • Measure: Average no. of bubbles (in flow diagram).

33. Resources • Personnel (individual or team), materials (including office supplies), tools, methods. • Resource measurement may show what resource to blame for poor quality. • Cost measured across all types or resources. • Productivity: • amount_of_output/effort_input. • Combines resource measure (input) with the product measure (output).

34. GQM Paradigm • Goal-Question-Metric • Steps: • List the major goals of the development effort. • Derive from each goal questions that must be answered to determine if goals are being met. • Decide what must be measured to answer the questions adequately.

35. Generic GQM Example

36. AT&T GQM Example

37. Goal definition templates • Purpose: • To (characterize, evaluate, predict…) the (process, model, metric…) in order to (understand, assess, manage, learn, improve…) • Perspective • Examine the (cost, correctness, defects, changes…) from the viewpoint of (developer, manager, customer…) • Environment • The environment consists of process factors, people factors, methods, tools, etc.

38. Process improvement • Measurement is useful for understanding, establishing the baseline, assessing and predicting. • But the larger context is improvement. • SEI proposed five maturity levels, ranging from the least to the most predictable and controllable.

39. Process maturity levels

40. Maturity and measurement overview • Level Characteristic Metrics • Initial Ad hoc Baseline • Repeatable Process depends Project on individuals management • Defined Process defined & Product institutionalized • Managed Measured process Process+ feedback • Repeatable Improvement fed back Process+feedback to the process for changing the process.

41. Repeatable process

42. A defined process

43. A managed process

44. Applying the framework • Cost and effort estimation • E=a*Sb • E is effort (person months), S is size (thousands of delivered source statements) • A, b are environment specific constants. • Data collection • Orthogonal defect classification

45. Applying the framework (2) • Reliability models • JM model: MTTFi=a/(N-I+1) • N: total no. faults, 1/a is the “fault size”. • Capability maturity assessment • The maturity attribute is also viewed as an attribute of contractor’s process.

46. Applying the framework (3) • Evaluation of methods and tools

47. Agenda • Software Engineering Measurements. • Measurement Theory. • A Goal-Based Framework for Software Measurement. • Verification and Validation Metrics.

48. V&V application of metrics • Applicable throughout the lifecycle. • Should be condensed for small projects. • Used to assess product, process, resources. • V&V metric characteristics: • Simplicity • Objectivity • Ease of collection • Robustness (insensitive to changes) • Validity