1 / 37

Testing

Testing. Xiaojun Qi. Testing. There are two basic types of testing Execution-based testing Non-execution-based testing “V & V” Verification: Determine if the workflow was completed correctly Validation: Determine if the product as a whole satisfies its requirements (specification)

hakan
Download Presentation

Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Testing Xiaojun Qi

  2. Testing • There are two basic types of testing • Execution-based testing • Non-execution-based testing • “V & V” • Verification: Determine if the workflow was completed correctly • Validation: Determine if the product as a whole satisfies its requirements (specification) • Term “Testing” is used instead of “V & V”

  3. Software Quality • The quality of software is the extent to which the product satisfies its specifications. • Defect is a generic term for • Fault: It happens when a human makes a mistake • Failure: It is the observed incorrect behavior of the software product as a consequence of a fault • Error: It is the amount by which a result is incorrect. • Every software professional is responsible for ensuring that his or her work is correct • Quality must be built in from the beginning

  4. Quality Issues:Software Quality Assurance • The members of the SQA group must ensure the quality of the software process and thereby ensure the quality of the product. • Test at the end of each workflow • Test when the product is complete • Quality assurance must be applied to the process itself • Develop various standards to which the software must conform • Establish the monitoring procedures for assuring compliance with those standards.

  5. Quality Issues:Managerial Independence • There must be managerial independence between the development group and the SQA group • Neither group should have power over the other • More senior management must decide whether to • Deliver the faulty product on time, or • Test further and deliver the product late • The decision must take into account the interests of the client and the development organization

  6. Non-Execution-Based Testing • Non-execution-based testing means that testing software without running test cases. • Review software (Walkthrough or Inspection) is a common non-execution-based testing method. Its underlying principles are: • We should not review our own work • Group synergy: A document is painstakingly checked by a team of software professionals with a broad range of skills.

  7. Non-Execution-Based Testing: Walkthroughs • A walkthrough team consists of four to six members • It includes representatives of • The team responsible the current workflow • The manager responsible for the current workflow • The team responsible for the next workflow • The SQA group • The clients

  8. Managing Walkthroughs • The walkthrough is preceded by preparation • A list of items • Items not understood • Items that appear to be incorrect • The walkthrough team is chaired by the SQA representative since • The quality of the product is a direct reflection of the professional competence of the SQA group.

  9. Managing Walkthroughs (Cont.) • In a walkthrough we detect faults for later correction, not correct them • A correction produced by a committee is likely to be of low quality • The cost of a committee correction is too high • Not all items flagged as faults are actually incorrect • A walkthrough should not last longer than 2 hours so there is no time to correct faults

  10. Managing Walkthroughs (contd) • Since verbalization leads to fault finding, there are two ways of conducting a walkthrough: • Participant-driven • Document-driven • Document-driven is likely to be more thorough since: • The majority of faults are spontaneously detected by the presenter • A walkthrough should never be used for performance appraisal

  11. Non-Execution-Based Testing: Inspections • An inspection has five formal steps • An overview of the document to be inspected • The preparation: List the statistics of fault types ranked by frequency • The inspection: Find and document the faults • The rework: The individual responsible for the document resolves all faults and problems noted in the written report. • The follow-up: Every issue has been resolved and all fixes must be checked to ensure that no new faults have been introduced.

  12. Inspections (Cont.) • An inspection team has four members • Moderator: Both manager and leader • A member of the team performing the current workflow • A member of the team performing the next workflow • A member of the SQA group • IEEE standard also recommends: • Reader: Leads the team through the design • Recorder: Produce a written report of the detected faults.

  13. An Essential Component of an Inspection: Fault Statistics • Faults are recorded by severity • Major faults: They cause premature termination or damage a database • Minor faults • Faults are recorded by fault type • Examples of design faults: • Not all specification items have been addressed • Interface faults: Actual and formal arguments do not correspond • Logic faults:

  14. Use of Fault Statistics • We compare current fault rates with those of previous products for a given workflow. • We take action if there are a disproportionate number of faults of a particular type in an artifact • Check other code artifacts and take corrective action • Redesigning from scratch is a good alternative • We carry forward fault statistics to the next workflow • We may not detect all faults of a particular type in the current inspection

  15. Successful Stories on Inspections • IBM inspections showed up • 82 percent of all detected faults (1976); 70 percent of all detected faults (1978); 93 percent of all detected faults (1986) • Switching system • 90 percent decrease in the cost of detecting faults (1986) • JPL • Four major faults, 14 minor faults per 2 hours (1990) ; Savings of $25,000 per inspection; The number of faults decreased exponentially by phase (1992) • However, fault statistics should never be used for performance appraisal

  16. Comparison of Inspections and Walkthroughs • Walkthrough: Two-step, informal process • Preparation • Analysis • Inspections: Five-step, formal process • Overview • Preparation • Inspection • Rework • Follow-up

  17. Strengths and Weaknesses of Reviews (Walkthrough or Inspection) • Strengths: • A review is an effective way to detect a fault. • Faults are detected early in the process • Weaknesses: • Reviews are less effective if the process is inadequate. The two solutions to this weakness are: • Large-scale software should consist of smaller, largely independent pieces • The documentation of the previous workflows has to be complete and available online

  18. Metrics for Inspections • Inspection rate: design pages inspected per hour • Fault density: faults per page inspected or faults per 1000 lines of code (KLOC) inspected • Fault detection rate: the number of faults detected per hour • Fault detection efficiency: number of major and minor faults detected per person-hour • Question: Does a 50 percent increase in the fault detection rate mean that • Quality has decreased? Or • The inspection process is more efficient?

  19. Execution-Based Testing • Organizations spend up to 50 percent of their software budget on testing • But delivered software is frequently unreliable • “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” (Dijkstra; 1972) • The only information that can be deduced from the particular test is that the product runs correctly on that particular set of test data.

  20. What Should Be Tested? • Definition of execution-based testing • “The process of inferring certain behavioral properties of the product based, in part, on the results of executing the product in a known environment with selected inputs” • This definition has troubling implications • “Inference”: We have a fault report, the source code, and often nothing else. This makes the inference difficult. • “Known environment”: We never really can know our environment, either the hardware or the software • “Selected inputs”: Sometimes we cannot provide the inputs we want • Solution: • Simulator is needed: A simulator is a working model of the environment in which the product executes.

  21. Items Should be Tested: Utility • The extent to which a user’s needs are met when a correct product is used under conditions permitted by its specifications • Ease of use • Useful functions • Cost effectiveness

  22. Items Should be Tested: Reliability • A measure of the frequency and criticality of product failure • Mean time between failures: How often the product fails • Mean time to repair: How long it takes, on average, to repair the failure • Time (and cost) to repair the results of a failure

  23. Items Should be Tested: Robustness • A function of • The range of operating conditions • The possibility of unacceptable results with valid input • The effect of invalid input

  24. Items Should be Tested: Performance • The extent to which the product meets its constraints with regard to response time or space requirements. • Real-time software is characterized by hard real-time constraints • If data are lost because the system is too slow • There is no way to recover those data

  25. Items Should be Tested: Correctness • A product is correct if it satisfies its output specifications, independent of its use of computing resources, when operated under permitted conditions. • Technically, correctness is • Not necessary • Example: C++ compiler • Not sufficient • Example:trickSort

  26. Importance of the Correctness of Specifications • Incorrect specification for a sort: • Function trickSort which satisfies this specification: Figure 6.1

  27. Importance of the Correctness of Specifications (Cont.) • Corrected specification for the sort:

  28. Testing versus Correctness Proofs • A correctness proof is a mathematical technique for showing that a product is correct, in other words, that it satisfies its specification. • It is an alternative to execution-based testing

  29. Example of a Correctness Proof • Its corresponding flowchart • The code segment to be proven correct

  30. Add the followings: • Input specification • Output specification • Loop invariant • Assertions

  31. Example of a Correctness Proof (Cont.) • Loop Invariant: A mathematical expression that holds irrespective of whether the loop has been executed 0, 1, or many times. • An informal proof (using induction) appears in Section 6.5.1. That is: given the input specification, it can be proven that loop invariant holds. Furthermore, it can be proven that after n iterations the loop terminates and the output specification is satisfied.  The code segment therefore is mathematically proven to be correct.

  32. Correctness Proof Mini Case Study • Dijkstra (1972): • “The programmer should let the program proof and program grow hand in hand” • “Naur text-processing problem” (1969) • Episode 1: Naur constructed a 25-line procedure and informally proved its correctness • Episode 2: Reviewer in Computing Reviews found 1 fault in 1970. • Episode 3: London found 3 more faults in 1971. • Episode 4: Goodenough and Gerhart found 3 further faults in 1975. • Lesson: Even if a product has been proved correct, it must STILL be tested.

  33. Three Myths of Correctness Proving • Software engineers do not have enough mathematics for proofs • Most computer science majors either know or can learn the mathematics needed for proofs • Proving is too expensive to be practical • Economic viability is determined from cost–benefit analysis • Proving is too hard • Many nontrivial products have been successfully proved • Tools like theorem provers can assist us

  34. Difficulties with Correctness Proving • Can we trust a theorem prover? • How do we find input and output specifications, and loop invariants or their equivalents in other logics? • What if the specifications are wrong? • We can never be sure that specifications or a verification system are correct

  35. Correctness Proofs and Software Engineering • Correctness proofs are a vital software engineering tool. They are appropriate: • When human lives are at stake • When indicated by cost–benefit analysis • When the risk of not proving is too great • Also, informal proofs can markedly improve the quality of the product • Use theassertstatement

  36. Who Should Perform Execution-Based Testing? • Programmers should not test their own code artifacts since: • Programming is constructive • Testing is destructive • A successful test finds a fault • Solution: • The programmer debugs the module • The programmer does informal testing • The SQA group then does systematic testing

  37. Test Cases • All test cases must be • Planned beforehand, including the expected output, and • Retained afterwards for regression testing • Testing stops only when the product has been irrevocably discarded

More Related