slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions PowerPoint Presentation
Download Presentation
The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions

Loading in 2 Seconds...

play fullscreen
1 / 46

The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions - PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions. Teresa Xia Cai Group Meeting Feb. 21, 2006. Outline. Testing coverage and testing strategies Research questions Experimental setup Results and analysis

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions' - lani


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

The Effect of Code Coverage on Fault Detection Capability: An Experimental Evaluation and Possible Directions

Teresa Xia Cai

Group Meeting

Feb. 21, 2006

outline
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
  • Discussions and conclusions
introduction
Introduction
  • Test case selection and evaluation is a key issue in software testing
  • Testing strategies aim to select an effective test set to detect as more faults as possible
    • Black-box testing (functional testing)
    • White-box testing (structural testing)
white box testing schemes control data flow coverage
White-box testing schemes:Control/data flow coverage
  • Code coverage - measured as the fraction of program codes that are executed at least once during the test.
    • Block coverage - the portion of basic blocks executed.
    • Decision coverage - the portion of decisions executed
    • C-Use- computational uses of a variable.
    • P-Use- predicate uses of a variable
code coverage an indicator for test effectiveness
Code coverage: an indicator for test effectiveness?
  • Supportive empirical studies
    • high code coverage brings high software reliability and low fault rate
    • both code coverage and fault detected in programs grow over time, as testing progresses.
      • Weyuker et al (1985, 1988, 1990)
      • Horgan, London & Lyu (1994)
      • Wong, Horgan, London & Mathur (1994)
      • Frate, Garg, Mathur & Pasquini (1995)
  • Oppositive empirical studies
    • Can this be attributed to causal dependency between code coverage and defect coverage?
      • Briand & Pfahl (2000)
black box testing schemes testing profiles
Black-box testing schemes:testing profiles
  • Functional testing– based on specified functional requirements
  • Random testing- the structure of input domain based on a predefined distribution function
  • Normal operational testing– based on normal operational system status
  • Exceptional testing- based on exceptional system status

{

{

outline7
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
  • Discussions and conclusions
research questions
Research questions
  • Is code coverage a positive indicator for testing effectiveness?
  • Does such effect vary under various testing profiles?
  • Does such effect vary with different coverage measurements?
  • Is code coverage a good filter to reduce the size of effective test set?
outline9
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
  • Discussions and conclusions
experimental setup
Experimental setup
  • In spring of 2002, 34 teams are formed to develop a critical industry application for a 12-week long project in a software engineering course
  • Each team composed of 4 senior-level undergraduate students with computer science major from the Chinese University of Hong Kong
experimental project description
Experimental project description

Redundant Strapped-Down Inertial Measurement Unit (RSDIMU)

  • Geometry
  • Data flow diagram
software development procedure
Software development procedure
  • Initial design document ( 3 weeks)
  • Final design document (3 weeks)
  • Initial code (1.5 weeks)
  • Code passing unit test (2 weeks)
  • Code passing integration test (1 weeks)
  • Code passing acceptance test (1.5 weeks)
mutant creation
Mutant creation
  • Revision control was applied in the project and code changes were analyzed
  • Faults found during each stage were also identified and injected into the final program of each version to create mutants
  • Each mutant contains one design or programming fault
  • 426 mutants were created for 21 program versions
setup of evaluation test
Setup of evaluation test
  • A test coverage tool was employed to analyze the compare testing coverage
  • 1200 test cases were exercised on 426 mutants
  • All the resulting failures from each mutant were analyzed, their coverage measured, and cross-mutant failure results compared
  • 60 Sun machines running Solaris were involved in the test, where one cycle took 30 hours and a total of 1.6 million files around 20GB were generated
outline18
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
    • Effective of code coverage
    • Under various testing profiles
    • With different coverage measurements
    • Effective test set
  • Discussions and conclusions
the correlation various test regions
The correlation: various test regions
  • Test case coverage contribution on mutant coverage
  • Test case coverage contribution on block coverage

I II III IV V VI

I II III IV V VI

outline27
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
    • Effective of code coverage
    • Under various testing profiles
    • With different coverage measurements
    • Effective test set
  • Discussions and conclusions
in various test regions
In various test regions
  • Linear regression relationship between block coverage and defect coverage in the whole test set
  • Linear modeling fitness in various test case regions
in various test regions cont
In various test regions (cont’)
  • Linear regression relationship between block coverage and defect coverage in region IV
  • Linear regression relationship between block coverage and defect coverage in region VI
in various test regions cont30
In various test regions (cont’)

Observations:

  • Code coverage: a moderate indicator
  • Reasons behind the big variance between region IV and VI
with functional random testing
With functional/random testing
  • Code coverage:

– a moderate indicator

  • Random testing

– a necessary complement to functional testing

    • Similar code coverage
    • Both have high fault detection capability
with functional random testing cont
With functional/random testing (cont’)
  • Failure number of mutants detected only by functional testing or random testing
under normal operational exceptional testing
Under normal operational / exceptional testing
  • The definition of operational status and exceptional status
    • Defined by specification
    • Application-dependent
  • For RSDIMU application
    • Operational status: at most two sensors failed as the input and at most one more sensor failed during the test
    • Exceptional status: all other situations
  • The 1200 test cases are classified to operational and exceptional test cases according to their inputs and outputs
under normal operational exceptional testing cont
Under normal operational / exceptional testing (cont’)
  • Normal operational testing
    • very weak correlation
  • Exceptional testing
    • strong correlation
under normal operational exceptional testing cont35
Under normal operational / exceptional testing (cont’)
  • Normal testing: small coverage range (48%-52%)
  • Exceptional testing: two main clusters
under normal operational exceptional testing cont36
Under normal operational / exceptional testing (cont’)
  • Failure number of mutants detected only by normal operational testing or exceptional testing
under testing profile combinations
Under testing profile combinations
  • Combinations of testing profiles
  • Observations:
    • Combinations containing exceptional testing indicate strong correlations
    • Combinations containing normal testing inherit weak correlations
outline38
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
    • Effective of code coverage
    • Under various testing profiles
    • With different coverage measurements
    • Effective test set
  • Discussions and conclusions
with different coverage measurements
With different coverage measurements
  • Similar patterns as block coverage
  • Insignificant difference under normal testing
  • Decision and P-use have a bit larger correlation, as they relate to change of control flow
outline40
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
    • Effective of code coverage
    • Under various testing profiles
    • With different coverage measurements
    • Effective test set
  • Discussions and conclusions
outline42
Outline
  • Testing coverage and testing strategies
  • Research questions
  • Experimental setup
  • Results and analysis
  • Discussions and conclusions
answers to rqs
Answers to RQs
  • Is code coverage a positive indicator for testing effectiveness?
    • Our answer is supportive
      • At most situations (61.5%), there is an coverage increase when a test case detect additional faults.
      • Under some functional and exceptional testing region, the correlation between code coverage and fault coverage is pretty high
      • When more cumulated code coverage have been achieved, more faults are detected.
answers to rqs cont
Answers to RQs (cont’)
  • Does such effect vary under various testing profiles?
    • A significant correlation exists in exceptional test cases, while no correlation in normal operational test cases.
    • Higher correlation is revealed in functional testing than in random testing, but the difference is insignificant
answers to rqs cont45
Answers to RQs (cont’)
  • Does such effect vary with different coverage measurements?
    • Not obvious with four coverage measurements
  • Is code coverage a good filter to reduce the size of effective test set?
    • Yes, 203 test cases (17% of the original test set) which achieve any coverage increase can detect 98% of the faults.
conclusion
Conclusion
  • Code coverage is a reasonably good indictor for fault detection capability.
  • The strong correlation revealed in exceptional testing implies that coverage works predictably better in certain testing profiles than others.
  • Testing guidelines and strategy can be established for coverage-based testing:
    • For normal operational testing: specification-based, regardless of code coverage
    • For exceptional testing: code coverage is an important metrics for testing capability
  • A quantifiable testing strategy may emerge by combining black-box and white-box testing strategies appropriately.