Testing Workflow Purpose

1. Testing Workflow Purpose • Purpose • Verify the result from implementation by testing each build • Plan the tests in each iteration • Integration tests for every build within theiteration • System tests for the end of the iteration • · Design and implement tests by creating • test cases that specify what to test • test procedures that specify how to perform the tests • executable test components to automate the tests

2. Testing Workflow Flow

3. Testing Workflow with Phases

4. Testing Workflow in Phases Generally, when there is an implementation result, there is a test --- so Test at each build Inception phase: initial test planning, prototype testing Elaboration phase: test architectural baseline Construction phase: significant testing at each build Transition phase: re-test fixes and regression tests

5. Testing Workflow Other Tasks Regression tests: in a new build, re-apply tests from old builds to make sure nothing “broke” in new build Evolve test model create new test cases for next build refine prior test cases into regression tests remove obsolete tests and corresponding test procedures and components

6. Testing Workflow Models Test model artifacts Test cases Test procedures Test components Test subsystem packages for complex test Other artifacts Test Plan Defects Test Evaluation

7. Testing Workflow Model TEST MODEL Describes how executable components (e.g., builds) in the implementation model are tested by integration tests and system tests Describes how specific system aspects are to be tested e.g., usability testing of user interface or user manual

8. Testing Workflow – Test Case TEST CASE Specifies one way to test the system What to test (requirements) With which inputs Expected results Test conditions and constraints

9. Testing Workflow Test Cases TEST CASE EXAMPLES Test a use case or a scenario in a use case (black box)§ actor interaction, pre-conditions, action sequences, post-conditions Test a use-case realization (white box) verify interaction between components. Installation tests, configuration tests, negative (abuse) tests, stress tests Organize similar tests in input/result matrix Family of test for different paths through a use case

10. Testing Workflow Special Tests Installation tests verify the system can be installed on the customer platform and that it executes correctly when installed Configuration tests verify the system works correctly in different configurations Negative tests (abuse tests) what causes failure Test outside the intended design, e.g., wrong configuration, wrong input, etc. Reveals weaknesses in design Stress tests show problems with insufficient or shared resources

11. Testing Workflow Procedures • Specify how to perform one or several test cases or • parts of them • Instructions for tester in manual test case • Instructions for interacting with test automation tool to • create executable test components, then instructions to • integrate and execute those test components • One test procedure may cover several test cases • One test case may require several test procedures • Test procedure instructions often reflect flow-of events • descriptions, including input values, how to enter input • values, and how to verify results

12. Testing Workflow Components Automate one or several test procedures or parts Provide test inputs, control and monitor test execution, and possibly analyze and report results Developed with scripting languages, programming languages, or test automation tools test drivers, test harnesses, and test scripts Complex test cases and procedures may require formal analysis, design, and implementation modeling of the test components

13. Testing Workflow Plan Describes testing strategies, resources and schedule · Strategy What kind of tests to perform for each iteration Test objectives Required level of test and code coverage Test acceptance criteria Trade-off cost vs. improved quality and reduced risk Test Plan may be defined in conjunction with the model artifacts from other workflows analysis (black box) design and implementation (white box) “Tests” may include artifact reviews

14. Testing Workflow Results Results are the output data of the test and/or the behavior of the system during test Analysis of results: pass or defect Defect: a system anomaly (software fault or a defect found in artifact review) capture symptom in sufficient detail to allow problem resolution capture defect in problem and resolution tracking system Test evaluation: summary of overall test results test-case coverage, code coverage defects and defect status

15. Testing Workflow Results

16. Testing Workflow Workers Test designer: responsible for integrity of the test model plan tests: test goals and schedule select and define test cases, test procedures evaluate test results Component engineer: responsible for development of test components that automate some of the test procedures consider independent test

17. Testing Workflow Workers Integration tester: perform integration tests and report defects may be same person as system integrator in implementation workflow System tester: perform system tests and report defects focus on externally observable behavior consider users, beta customers, etc. as testers

18. Testing Workflow – THE PLAN Plan the Tests Strategy and Resources Define Integration Tests Define System Tests Design Regression Tests Define and Mutate Test Cases Implement Tests Evaluate Tests

19. Testing Workflow – THE PLAN Plan the Tests Strategy and Resources Describe testing strategy Inputs: use case model, supplemental requirements, design model Develop test cases and procedures that have a minimum of overlap test the most important test cases test high risk requirement items Estimate resources Estimate human and system resources Schedule the testing effort

20. Testing Workflow – THE PLAN Define Integration Tests Derive test cases from use-case realizations – design (component interaction) Develop tests with minimal overlap that test an interesting path or scenario through use case Primarily consider interaction diagrams Combinations of actor input, output and system start state that lead to interesting scenarios that employ the classes that participate in the diagrams

21. Testing Workflow – THE PLAN Define System Tests Test system as a whole Test combinations of use cases instantiated under different conditions Hardware configurations System loads Number of actors Database size, etc. Prioritize combinations required or are likely to function in parallel likely to influence each other if in parallel involve multiple processes frequently use system resources or use them in complex and unpredictable ways

22. Testing Workflow – THE PLAN Design Regression Tests Modify test cases from earlier builds to be used as regression tests in subsequent builds Make more flexible and resilient to changes to the system under test Specifically address previously found defects Automate where possible

23. Testing Workflow – THE PLAN Define and Mutate Test Cases Work test case by test case and suggest testing procedures for each one Reuse existing test cases May need to modify them to be reusable Align test procedures to subsystems under test Minimize overlap Isolates change to test resulting from change to subsystem

24. Testing Workflow – THE PLAN Implement Tests Create test components from test procedures With test automation tools Perform or specify actions Record actions as a test component When programming explicitly Design and implement using test procedures Provide input and output data generation, visualization, interpretation and management utilities Use spreadsheets and database applications

25. Testing Workflow – THE PLAN Evaluate Tests Perform the integration tests relevant to a build Manually perform test procedures Compare test results with the expected test results Investigate test results that deviate from the expected Faulty system and/or faulty test? Report defects to the component engineer responsible for the components likely to contain the fault Report defects to the test designers for evaluation of overall test results

26. Testing Workflow – THE PLAN Evaluate Tests (cont) Test designers prepare metrics of test acceptance with respect to system quality level Test completeness and coverage Percentage of test cases executed Percentage of code and paths tested Reliability Categorize defect severity Analyze trends of discovered defects Compare with prior experience to predict amount of testing still required to reach an acceptable defect leve

27. Testing Workflow – THE PLAN Evaluate Tests (cont) Suggest actions Suggest further tests Relax test criteria Deliver parts of the system that passed; rework and retest parts that failed Document test completeness, reliability and actions

28. Testing Workflow – SUMMARY Test Model Test cases specify what to test Test procedures specify how to perform tests Test components automate test procedures Test Plan Test Evaluation Defects to be fed into other core work