1 / 26

Covrig : A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software

Covrig : A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software. Paul Marinescu , Petr Hosek , Cristian Cadar Imperial College London. Goal. A nswer questions about software evolution Code quality Test quality Development model

miyo
Download Presentation

Covrig : A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software Paul Marinescu, PetrHosek, CristianCadar Imperial College London

  2. Goal • Answer questions about software evolution • Code quality • Test quality • Development model • Testing improvement opportunities …using software development historical data

  3. Target Audience • Researchers • Hypothesis validation (e.g. are software patches poorly tested?) • Programmers/Project Managers • Assess development quality

  4. Software Metrics • Static • Measured by parsing the software artifacts • Dynamic • Require running the evolving software • More challenging • Very few studies

  5. Example questions • Do executable and test code evolve in sync? • How many patches touch only code/test/none/both? • What is the distribution of patch sizes? • How spread out is each patch through the code? • Is test suite execution deterministic? • How does the overall coverage evolve? • What is the distribution of patch coverage across revisions? • What is the latent patch coverage? • Are bug fixes better covered than other patches? • Is the coverage of buggy code less than average?

  6. Data mining infrastructure Empirical case study

  7. Covrig Overview 2 1 3

  8. Docker Containers • Lightweight, OS-level virtualization • Guest shares kernel with host • Namespace isolation • PID • Network • IPC • Filesystem • Resource limiting • cgroups + Linux Containers + Docker

  9. Docker Containers Features • Isolation • Consistency • Reproducibility • Easy cloud deployment • Performance

  10. Covrig

  11. Challenges Evolving dependencies • Evolving containers • Custom compile flags (-Wno-error)

  12. Challenges Branching development structure • Consider only the ‘main’ branch Alice Bob r1 r1 r2 r3 r3 r4 r2+r4 m1

  13. Challenges Revisions that fail to compile • Accumulate until reaching a compilable revision r1 r2 r3 r1+r2+r3

  14. Data mining infrastructure Empirical case study

  15. Case Study Subjects 1500 revisions and 12 years of development in total

  16. Patch type

  17. Is test suite execution deterministic? FAIL/PASS determinism

  18. Is test suite execution deterministic? Coverage determinism

  19. Test Suite Nondeterminism Causes • Bugs • Race conditions • Hardcoded wall clock timeouts • Incorrect resource consumption expectations • Random test data • Benign race conditions

  20. Are patches properly tested? Sometimes

  21. Patch coverage

  22. Patch coverage 0% 0% 0% 0% 0% 0%

  23. Does covered code contain fewer bugs that not covered code? Not really

  24. Does covered code contain fewer bugs that not covered code? 85 total bugs

  25. Conclusions Dynamic software metrics mining Case study on 6 systems/1500 revisions/12 years of development Open source extensible infrastructure http://srg.doc.ic.ac.uk/projects/covrig/

More Related