Theory and practice do they match a case with spectrum based fault localization
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization PowerPoint PPT Presentation


  • 91 Views
  • Uploaded on
  • Presentation posted in: General

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization. Tien-Duy B. Le, Ferdian Thung , and David Lo School of Information Systems Singapore Management University. Spectrum-Based Fault Localization. Locating buggy program elements by

Download Presentation

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Theory and practice do they match a case with spectrum based fault localization

Theory and Practice, Do They Match ?A Case with Spectrum-Based Fault Localization

Tien-Duy B. Le, FerdianThung, and David Lo

School of Information Systems

Singapore Management University


Spectrum based fault localization

Spectrum-Based Fault Localization

  • Locating buggy program elements by

    • Analyzing two sets of execution traces

      • Normal traces and faulty traces

    • Assigning suspiciousness scores to program elements

  • Two well-known SBFL formulas

    • Tarantula

    • Ochiai


Spectrum based fault localization1

Spectrum-Based Fault Localization

  • Xie et al. “A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization” (TOSEM, 2013)

    • Two families of SBFL formulas

      • ER1 and ER5 (5 formulas in total)

      • Theoretically proven to outperform Ochiai and Tarantula

    • Under the assumption: test coverage is 100%


Our goal

Our Goal

Benchmark

Programs

vs.

Theoretically Best SBFL Formula by Xie et. al.

Popular SBFL Formula

?


Notations

Notations


Popular sbfl formulas

Popular SBFL Formulas

  • Tarantula

  • Ochiai


Theoretically best sbfl formulas

Theoretically Best SBFL Formulas


Dataset

Dataset

  • 10 programs, 199 faulty versions

    • Siemens test suite

    • Space, NanoXML, XML-Security

  • Evaluation Metric

    • The lower the EXAM score, the better the performance


Results

Results

  • Ochiai has the lowest EXAM score (21.02%)


Results1

Results

  • Tarantula‘s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas


Results2

Results

  • Wilcoxon signed rank test (significance level of 0.05)

    •  Ochiai is statistically better than ER5a, ER5b, ER5c


100 test coverage assumption

100% Test Coverage Assumption

  • For 135 out of the 199 faulty versions

    • Test coverage < 100%

  • Average test coverage of the 199 versions

    • 84.97%

       Theoretically best SBFL formulas cannot outperform popular SBFL formulas


Conclusion

Conclusion

  • We conduct an empirical study on 10 programs with 199 versions

    • Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai

  • We find that:

    • Ochiai outperforms all theoretically best formulas

    • Tarantula outperforms 4 out of the 5 formulas

    • Assumption of 100% test coverage is not valid in many cases


Future work

Future work

  • In-depth study how test coverage and other factors affect effectiveness of SBFL formulas

  • Theoretically analyze performance of SBFL formulas

    • Assumption: test coverage < 100%


Thank you

Thank you!

Questions? Comments? Advice?

{btdle.2012, ferdiant.2013, [email protected]


  • Login