1. Bug Driven Bug Finding Chadd Williams

2. Motivation • Finding bugs in software is important • Statically checking code has been effective • finds complex errors • no need to run the code • Many static checkers available • some with specific bug patterns to find • some allow the user to define the patterns • what kinds of bugs are really out there? • Lots of false positive error reports • can we rank the errors better? • can previous bug history help? • Where to start? • Bug reporting databases • CVS commit messages

3. Bug Database • Inspect fixed bugs • review bug discussions • tie fixed bug to source code change • classify the type of the bug • look for bugs that can be found statically Users Developers Bug Database

4. Bug Database: Practical Experience • We inspected the Apache httpd bug database • inspected 200 bug reports marked as fixed • not as helpful as we expected • Only 24% tied directly back to a source code change • bug reports include a discussion of the problem • rarely is a diff or a CVS revision noted • Most are logic errors/feature requests • not the type found by static checkers

5. Bug Database Bug Types • Most classified bugs are logic errors

6. Bug Database: Practical Experience • Most bug reports originate from users • 197 out of 200 • does not capture bugs found by developers • Most bug reports came against a release of the software, not a CVS-HEAD • 198 out of 200 • does not capture bugs between releases • What about the bugs that don’t make it into the release? • they may be in the CVS repository…

7. CVS Repository • Commits may contain useful data • any bug fix must show up in a commit • will commit messages lead us to bug fixes? • Shows bugs fixed between releases • Bugs caught by developers • bugs that could be found by static checking 1.1 1.2 CVS Repository 1.3

8. CVS Repository: Practical Experience • Inspected commit messages • looked for ‘fix’, ‘bug’ or ‘crash’ • ignored those with bug number listed • looked at mature source files • Commit messages are useful • trivially tied to source code change • less logic errors • Common errors found • NULL pointer check • failing to check the return value of a function before use

9. CVS Repository Bug Types • NULL pointer bugs and return value bugs can be found by static analysis

10. Return Value Check Bug • Returning error code and valid data from a function is a common C idiom • the return value should be checked before being used • lint checks for this error • Error types • completely ignored • foo(); • return value used directly as an argument • bar(foo()); • others … int foo(){ … if( error ){ return error_code; } …. return data; } … value = foo(); newPosition + = value; // ???

11. Return Value Checker • Some functions don’t need their return value checked • no error value returned • could lead to many false positives • Naively flagging all unchecked return values leads to many false positives • over 7,000 errors reported for the Apache httpd-2.0 source • Need to determine which are most likely true errors • use historical data • present this data to the user

12. Which return values need checked? • Infer from historical data • look for an add of a check of a return value in a CVS commit • implies the programmer thinks it’s important • Infer from current usage • does the return value of a function get checked in the current version of the software • how often? … value = foo(); if( value != Error) { // Check newPosition + = value; } … Commit Bug Fix … value = foo(); newPosition + = value; // ??? …

13. Our Tool • Static checker that looks for return value check bugs • built on ROSE by Dan Quinlan, et al. • Classify each error by category • ignored return value • return value used as argument, etc. • Produce a ranking of the errors • group errors by called function • rank most promising errors higher • rank functions that most likely need their return value checked higher

14. Return Value Checker: Ranking • Rank errors in two ways • Split functions into two groups • functions flagged with a CVS bug fix commit • at least one CVS commit adds a check of the function’s return value • functions not flagged with CVS bug fix commit • Within each group: • rank by how often the function’s return value is checked in the current software distribution • checked more often means rank higher

15. Case Study • Apache httpd-2.0 on Linux • core system • modules • Apache Runtime Library • Checked all the CVS commits for a return value check bug fix • 6100 commits checked • 2600 commits failed to go through our tool • wrong (too new) version of autoconf • parser problems • compile bugs in the CVS commits

16. Case Study: Results • Our checker marked over 7,000 errors • individual call site for non-void function where the return value is not checked • Too many too look at! • expect many are false positives • Rank errors • inspect CVS bug fix commit flagged functions • inspect functions with return value checked more than 50% of the time in the current source tree value = foo(); // ERROR newPosition + = value; … result = foo(); // ERROR zoo(result);

17. Case Study: Error Breakdown • Inspected 453 errors (of 7,000) • found 98 that may be bugs! • 231 errors associated with a CVS bug fix flagged function • 61 of the 98 bugs found here • false positive rate of 74% • 222 errors associated with a function that has its return value checked > 50% of the time • 37 of the 98 bugs found here • false positive rate of 83%

18. Case Study: A Bug • We investigated an error and found it did crash httpd • error reported near the top of the ranking • The called function builds a filename • arguments represent file and pathname • a char array is returned and directly used as an argument to strcmp() • strcmp(foo()) • NULL return value will cause a seg fault • return value is NULL if the path is too long!

19. Analysis • False positive rate too high! • overall false positive rate: 78% (1-(98/453)) • A false positive rate closer to 50% would be acceptable • the user is likely as not to find a true error • cluster them near the top of the ranking • We did cull 7,000 errors down to 453 • lint would have flagged only the ‘ignored’ errors and not ranked them

20. Conclusion • Bug databases are not useful in understanding much about low-level bugs • good for logic errors • good for misunderstood specifications • CVS commit messages give a better picture of low-level bugs • especially bugs that don’t enter a release • CVS commits can give useful data to help classify error reports

21. Future Work • What other types of bugs are common? • What other checkers can benefit from CVS data? • How can we cut the false positive rate? • Can we dynamically gather data on functions called via function pointers? • many of the error messages involved calls through function pointers • Dyninst will allow us to instrument function pointer call sites and gather data