CS4723 Software Validation and Quality Assurance

1. CS4723Software Validation and Quality Assurance Lecture 15 Advanced Topics Test Plans and Management

2. Test Plan & Management • Something you are going to think about when you are a project manager instead of a tester / developer • About handle the testing progress of a whole software project • About quality assurance at highest level • Approach: risk based testing

3. Presentation Outline • Introduction - The Challenge Why? • Risk Analysis and Testing • Risk Based Test Approach What? • Test Metrics • Risk Based Test Metrics • Risk Analysis in Risk Based Testing • Risk Analysis in Practice • Plan and Manage - Metrics How? • Quality and Progress Monitoring • Warning! • Organization and Process...

4. The Challenges • Time Constraints • Resource Constraints • Quality Requirements • Risk Factors: • New technology • Lack of knowledge • Lack of experience • Take Control!

5. Presentation Outline • Introduction - The Challenge • Risk Analysis and Testing What? • Risk Based Test Approach • Test Metrics • Risk Based Test Metrics • Risk Analysis in Risk Based Testing • Theory and Practice • Plan and Manage • Quality and Progress Monitoring • Warning! • Organization and Process...

6. Test Plan Test Item Tree Risk Strategy Risk Identfication Testing, Inspection etc. Risk Assessment Risk Mitigation Metrics: Cost and Probability Test Metrics Risk Reporting Risk Prediction Risk Analysis and Testing

7. Risk Based Testing - Approach • Risk Identification / Test Items • Identify the items to be tested • Functions, classes, features, transactions, … • Risk Strategy / Test Plan: • Setup approaches to generating test cases • Automatic / Manual • Original / Regression • Load / Security / Usability / ,,, • Risk Assessment / Metrics • Assess risk of each test items

8. Risk Based Testing - Approach • Risk Mitigation / Testing • Generate and execute the concrete test cases • Risk Reporting • Measure how much have we mitigated the risk • Test coverage • Found and fixed bugs • Risk Prediction • Based on the risk reports, predict risks of test items for new round of testing

9. Risk Based Testing - Metrics • Identify Areas with “High Risk Exposure” • Probability and Cost • All functions/modules should be tested to a “minimum level” • “Extra Testing” in areas with high risk exposure • Establish Test Plan and Schedule • Monitor Quality • Number of Faults per function and time • Monitor Progress • Number of hours in test and fix -> ETC

10. Risk Based Testing - Theory • The Formula • Re(f) - Risk Exposure of function f • P(f) - Probability of a fault in function f • C(f) - Cost related to a fault in function f

11. Simple Test Metrics • Test Planning • Number of test cases per function • Number of hours testing per function • Progress Tracking • Number of tests planned, executed and completed • Number of faults per function • Number of hours used for test and fix • Estimated to Complete

12. Simple Test Metrics • Probability of faults - Indicators • New functionality • Size • Complexity • Quality of previous phases and documents • Cost of Faults

13. Presentation Outline • Introduction - The Challenge • Risk Analysis and Testing • Risk Based Test Approach • Test Metrics • Risk Based Test Metrics • Risk Analysis in Risk Based Testing How? • Theory and Practice • Plan and Manage • Quality and Progress Monitoring • Warning! • Organization and Process...

14. Risk Based Testing - Example • Ranking the functions based on Risk Exposure • The Probability of a Fault • The Cost of a Fault • Example: Other Probability Factors might include: Function Points, Frequency of Use etc.

15. High 1 Low Medium High Probability 2 Probability 3 4 Low Medium High Low Consequence Low High Consequence Risk Based Testing - Reporting

16. 1 2 3 Risk Based Testing - Practice Prior to test execution: identify critical transactions Test Execution identifies “bad” transactions Extra Testing: - Additional testing by product specialist - Automated regression testing

17. Presentation Outline • Introduction - The Challenge • Risk Analysis and Testing • Risk Based Test Approach • Test Metrics • Risk Based Test Metrics • Risk Analysis in Risk Based Testing • Theory and Practice • Plan and Manage How? • Quality and Progress Monitoring • Warning! • Organization and Process...

18. Two Strategies • On-line Risk Analysis • Usually used for first time testing • Cold start with little data • Minimal testing and a small list of critical features provide by customers • Batch Risk Analysis • Assume a large amount of data at the beginning • Different metrics

19. Planned Executed QAed Planning and Progress Tracking

20. “To be fixed” vs. “Actually fixed” To Be Fixed and Actually Fixed To Be Fixed Number of Faults Actually fixed To Be Restested, Actually Retested and Rejected To be retested Act. retested Number of Faults Rejected Progress Indicators - “To be vs. Actual” “To be Retested” vs. “Actually Retested”

21. Hours per Fault for Test and Fix Hours per Fault Fix Test Fix Test Hours per Fault for Test and Fix Hours per Fault Date Progress Indicators - Hours Used • Online • Number of hours for finding one fault and for fixing one • Batch • Number of hours for finding one fault and for fixing one

22. Actual to Complete at Time t “Estimated to Complete” • ETC for system test based on: • Number of hours testing per fault found • Number of hours fixing per fault • Number of faults found per function • Number of fixes being rejected • Number of remaining tests (functions to be tested) Estimated to Complete at Time t

23. Benefits of Risk Based Testing • Improved Quality? • all critical functions tested • Reduced Time and Money in Testing • effort not wasted on non critical or low risk functions • Improved customer confidence • due to customer involvement and good reporting and progress tracking

24. Presentation Outline • Introduction - The Challenge • Risk Analysis and Testing • Risk Based Test Approach • Test Metrics • Risk Based Test Metrics • Risk Analysis in Risk Based Testing How? • Theory and Practice • Plan and Manage • Quality and Progress Monitoring • Warning! • Organization and Process...

25. Objectives, policy and Test plan strategy for the test project Test item spec. Test item tree Test case spec. Application based STATIC groups TEST PLANNING DYNAMIC Function based TEST PLANNING groups 2 Test data Test design specific. TEST EXECUTION Test log Change Test procedure v222 request LOG KEEPING Error / Test Event log summary change log report WARNING! • Require a focused Organization… • Strong Test Management • Testers to “drive” the Programmers • Responsibility • …and a well defined Process • Documentation • Planning and Execution Procedures • Control Procedures • Planning and Progress Tracking

26. Risk Identification PD LD Risk Assessment Case Build Case Quality Test Procedure Standards Test Case QC / QA Completed Basic Test Good Data Test Exec. Good/ Test Exec Procedure Risk Mitigation Bad Bad PTDs Problem Mngmnt. ProAte Regression Risk Reporting Raised Procedure Test Re- Fix Procedure Fix test Risk Prediction Change Mngmnt. CR Procedure Test Process Work Flow

27. Summary • Risk Based Test Approach • Focused Testing • Reduced Resources • Improved Quality • Metrics are fundamental • Process and Organization must support the new strategy • Metrics must support the organization and process