Effective Verification and Validation in Software Project Management

1. SE 556 Project Management Session VII: Verification and Validation / Testing

2. Deliverables by Phase

3. If 99.9% Were Good Enough • 9,703 checks would be deducted from the wrong bank accounts each hour • 27,800 pieces of mail would be lost per hour • 3,000,000 incorrect drug prescriptions per year • 8,605 commercial aircraft takeoffs would annually result in crashes Futrell, Shafer, Shafer, “Quality Software Project Management”, 2002

4. Development Costs

5. Quality Assurance • QA or SQA (Software Quality Assurance) • Good QA comes from good process • When does SQA begin? • During requirements • A CMM Level 2 function • QA is your best window into the project

6. Test Plans (SQAP) • Software Quality Assurance Plan • Should be complete near end of requirements • Use the IEEE standards

7. SQAP • Standard sections • Purpose • Reference documents • Management • Documentation • Standards, practices, conventions, metrics • Quality measures • Testing practices

8. SQAP • Standard sections (continued) • Reviews and Audits • Process and specific reviews • Requirements Review (SRR) • Test Plan Review • Code reviews • Post-mortem review • Risk Management • Tie-in QA to overall risk mgmt. Plan • Problem Reporting and Corrective Action • Tools, Techniques, Methodologies • Records Collection and Retention

9. Software Quality • Traceability • Ability to track relationship between work products • Ex: how well do requirements/design/test cases match • Formal Reviews • Conducted at the end of each lifecycle phase • SRR, CDR, etc.

10. Verification and Validation Assuring that a software system meets a user's needs

11. Verification vs Validation • Verification: "Are we building the product right" • The software should conform to its specification • Validation: "Are we building the right product" • The software should do what the user really requires

12. The V & V Process • As a whole life-cycle process - V & V must be applied at each stage in the software process. • Has two principal objectives • The discovery of defects in a system • The assessment of whether or not the system is usable in an operational situation.

13. Static and Dynamic Verification • Software inspections Concerned with analysis of the static system representation to discover problems(static verification) • May be supplement by tool-based document and code analysis • Software testing Concerned with exercising and observing product behaviour(dynamic verification) • The system is executed with test data and its operational behaviour is observed

14. Static And Dynamic V&V

15. Program Testing • Can reveal the presence of errors NOT their absence !!! • A successful test is a test which discovers one or more errors • The only validation technique for non-functionalrequirements • Should be used in conjunction with static verification to provide full V&V coverage

16. Types of Testing • Defect testing • Tests designed to discover system defects. • A successful defect test is one which reveals the presence of defects in a system. • Statistical testing • tests designed to reflect the frequency of user inputs. Used for reliability estimation.

17. Testing Strategies • Testing strategies are ways of approaching the testing process. • Different strategies may be applied at different stages of the testing process. • Strategies covered: • Top-down testing • Bottom-up testing • Stress testing • Back-to-back testing

18. V& V Goals Verification and validation should establish confidence that the software is fit for purpose • This does NOT mean completely free of defects • Rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed

19. V & V Confidence Depends on system’s purpose, user expectations and marketing environment • Software function • The level of confidence depends on how critical the software is to an organisation • User expectations • Users may have low expectations of certain kinds of software. • Now it is less acceptable to deliver unreliable systems, so software companies must devote more effort to V&V! • Marketing environment • Getting a product to market early may be more important than finding defects in the program

20. Testing and Debugging Defect testing and debugging are distinct processes • (!) Verification and validation is concerned with establishing the existence of defects in a program Debugging is concerned with - locating and - repairing these errors • (!!) Debugging involves • formulating a hypothesis about program behaviour • then testing these hypotheses to find the system error

21. The Debugging Process

22. V & V Planning • Careful planning is required to get the most out of testing and inspection processes • Planning should start early in the development process • The plan should identify the balance between static verification and testing • Test planning is about defining standards for the testing process rather than describing product tests

23. V Process Model This diagram shows how test plans should be derived from the system specification and design.

24. The Structure Of A Software Test Plan • The testing process(a description of the major phases) • Requirements traceability(a part of the user) • Tested items • Testing schedule • Test recording procedures(it is not enough simply to run tests ) • Hardware and software requirements • Constraints

25. Software Inspections • Involve people examining the source representation with the aim of discovering anomalies and defects • Do not require execution of a system so may be used before implementation • May be applied to any representation of the system (requirements, design, test data, etc.) • Very effective technique for discovering errors

26. Inspection Success • Many different defects may be discovered in a single inspection. In testing, one defect, may mask another so several executions are required • The reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly arise

27. Inspections and Testing • Inspections and testingare complementary and not opposing verification techniques • Both should be used during the V & V process • Inspections can check conformance with a specificationbut not conformance with the customer’s real requirements • Also inspections cannot check non-functional characteristics such as performance, usability, etc.

28. Program inspections – are reviews whose objective is program defect detection. • Formalised approach to document reviews • Intended explicitly for defect DETECTION (not correction) • Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialised variable) or non-compliance with standards

29. Inspection Pre-conditions • A precise specification must be available • Team members must be familiar with the organisation standards • Syntactically correct code must be available • An error checklist should be prepared • Management must accept that inspection will increase costs early in the software process • Management must not use inspections for staff appraisal

30. The Inspection Process

31. Inspection Procedure • System overview presented to inspection team • Code and associated documents are distributed to inspection team in advance • Inspection takes place and discovered errors are noted • Modifications are made to repair discovered errors • Re-inspection may or may not be required

32. Inspection Checklists • Checklist of common errors should be used to drive the inspection • Error checklist is programming language dependent • The 'weaker' the type checking, the larger the checklist • Examples: • Initialisation, • Constant naming, • loop termination, • array bounds, etc.

33. Inspection checks

34. Automated Static Analysis • Static analysers are software tools for source text processing • They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team • Very effective as an aid to inspections. A supplement to but not a replacement for inspections

35. Static Analysis Checks

36. Stages Of Static Analysis • Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc. • Data use analysis. Detects uninitialised variables, variables written twice without an intervening assignment, variables which are declared but never used, etc. • Interface analysis. Checks the consistency of routine and procedure declarations and their use

37. Stages Of Static Analysis • Information flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or review • Path analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review process • Both these stages generate vast amounts of information. Must be used with care.

38. Use Of Static Analysis • Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler • Less cost-effective for languages like Java that have strong type checking and can therefore detect many errors during compilation

39. Testing Stages • Unit testing • testing of individual components • Module testing • testing of collections of dependent components • Sub-system testing • testing collections of modules integrated into sub-systems • System testing • testing the complete system prior to delivery • Acceptance testing • testing by users to check that the system satisfies requirements. Sometimes called alpha testing

40. Black-Box Testing • Functional Testing • Program is a “black-box” • Not concerned with how it works but what it does • Focus on inputs & outputs • Test cases are based on SRS (specs)

41. White-Box Testing • Accounts for the structure of the program • Coverage • Statements executed • Paths followed through the code

42. Unit Testing • a.k.a. Module Testing • Type of white-box testing • Sometimes treated black-box • Who does Unit Testing? • Developers • Unit tests are written in code • Same language as the module • a.k.a. “Test drivers” • When do Unit Testing? • Ongoing during development • As individual modules are completed

43. Unit Testing • Individual tests can be grouped • “Test Suites” • JUnit • Part of the XP methodology • “Test-first programming”

44. Integration Testing • Testing interfaces between components • First step after Unit Testing • Components may work alone but fail when put together • Defect may exist in one module but manifest in another • Black-box tests

45. System Testing • Testing the complete system • A type of black-box testing

46. User Acceptance Testing • Last milestone in testing phase • Ultimate customer test & sign-off • Sometimes synonymous with beta tests • Customer is satisfied software meets their requirements • Based on “Acceptance Criteria” • Conditions the software must meet for customer to accept the system • Ideally defined before contract is signed • Use quantifiable, measurable conditions

47. Regression Testing • Re-running of tests after fixes or changes are made to software or the environment • EX: QA finds defect, developer fixes, QA runs regression test to verify • Automated tools very helpful for this

48. Compatibility Testing • Testing against other “platforms” • Ex: Testing against multiple browsers • Does it work under Netscape/IE, Windows/Mac

49. Cleanroom software development • The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal • Software development process based on: • Incremental development • Formal specification. • Static verification using correctness arguments • Statistical testing to determine program reliability.

50. The Cleanroom Process