1 / 27

Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs

Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs. Marjan Aslani, Nga Chung, Jason Doherty, Nichole Stockman, and William Quach Summer Undergraduate Program in Engineering Research at Berkeley (SUPERB) 2008 Team for Research in Ubiquitous Secure Technology. Overview.

trinh
Download Presentation

Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs Marjan Aslani, Nga Chung, Jason Doherty, Nichole Stockman, and William Quach Summer Undergraduate Program in Engineering Research at Berkeley (SUPERB) 2008 Team for Research in Ubiquitous Secure Technology

  2. Overview • Introduction to Fuzz testing • Our research • Result "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  3. What Is Fuzzing? • A method of finding software holes by feeding purposely invalid data as input to a program. – B. Miller et al.; inspired by line noise • Apps: image processors, media players, OS • Fuzz testing is generally automated • Finds many problems related to reliability; many of which are potential security holes. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  4. Types of Fuzz Testing • BlackBox: Randomly generated data is fed to a program as input to see if it crashes. • Does not require knowledge of the program source code/ deep code inspection. • A quick way of finding defects without knowing details of the application. • WhiteBox:Creates test cases considering the target program's logical constraints and data structure. • Requires knowledge of the system and how it uses the data. • Deeper penetration into the program. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  5. Zzuf - Blackbox Fuzzer • Finds bugs in applications by corrupting random bits in user-contributed data. • To make new test cases, Zzuf uses a range of seeds and fuzzing ratios (corruption ratio). "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  6. Catchconv - WhiteBox Fuzzer • To create test cases, CC starts with a valid input, observes the program execution on this input, collects the path condition followed by the program on that sample, and attempts to infer related path conditions that lead to an error, then uses this as the starting point for bug-finding. • CC has has some downtime when it only traces a program and is not generating new fuzzed files. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  7. Valgrind • A tool for detecting memory management errors. • Reports the line number in the code where the program error occurred. • Helped us find and report more errors than we would if we focused solely on segmentation faults. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  8. Types of errors reported by Valgrind By tracking a program’s execution of a file, Valgrind determines the types of errors that occur which may include: • Invalid writes • Double free - Result 256 • Invalid reads • Double free • Uninitialized values • Syscal Pram • Memory leak "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  9. Program run under Valgrind "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  10. Methodology "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  11. All of the test files that triggered bugs were uploaded on Metafuzz.com. The webpage contained: Link to the test file Bug type Program that the bug was found in Stack hash number where the bug was located Metafuzz "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  12. Metafuzz webpage "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  13. Target applications • Mplayer, Antiword, ImageMagick Convert and Adobe Flash Player • MPlayer the promary target: • OS software • Preinstalled on many Linux distributions • Updates available via subversion • Convenient to file a bug report • Developer would get back to us! • Adobe bug reporting protocol requires a certain bug to receive a number of votes form users before it will be looked at by Flash developers. • VLC requires building subversions from nightly shots. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  14. In 6 weeks, generated more than 1.2 million test cases. We used UC Berkeley PSI-cluster of computers, which consists of 81 machines (270 processors). Zzuf, MPlayer, and CC were installed on them. Created a de-duplication script to find the unique bugs. Reported 89 unique bugs; developers have already eliminated 15 of them. Research Highlights "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  15. To provide assessments for the two fuzzers, we gathered several metrics: Number of test cases generated Number of unique test cases generated Total bugs and total unique bugs found by each fuzzer. Result "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  16. Result con’t • Generated 1.2 million test cases • 962,402 by Zzuf. • 279,953 by Catchconv. • From the test cases: • Zzuf found 1,066,000 errors. • Catchconv reported 304,936. • Unique (nonduplicate) errors found: • 456 by Zzuf • 157 by Cachconv "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  17. Result con’t • Zzuf reports a disproportionately larger amount of errors than CC. Is Zzuf better than CC? • No! The two fuzzers generated different numbers of test cases. • How could we have a fair comparison of the fuzzers’ efficiency? • Need to gauge the amount of duplicate work performed by each fuzzer. • Find how many of these test cases were unique. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  18. Average Unique Errors per 100 Unique Test Cases • First, we compared performance of the fuzzers by the average number of unique bugs found per 100 test cases. • Zzuf: 2.69 • CC : 2.63 • Zzuf’s apparent superiority diminishes. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  19. Unique Errors as % of Total Errors • Next, we analyzed fuzzers’ performance basedon the percentage of unique errors found out of the total errors. • Zzuf: .05% • CC: .22% • Less than a quarter percent difference between the fuzzers. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  20. Types of Errors (as % of Total Errors) • Also considered analyzing the fuzzer based on bug types found by the fuzzers. • Zzuf performed better in finding “invalid write”, which is a more important security bug type. • Not an accurate comparison, since we couldn’t tell which bug specifically caused a crash. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  21. Conclusion • We were not able to make a solid conclusion about the superiority of either fuzzer based on the metric we gathered. • Knowing which fuzzer is able to find serious errors more quickly would allow us to make a more informed conclusion about their comparative efficiencies. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  22. Conclusion con’t • Need to record the amount of CPU clock cycles required to execute test cases and find errors. • Unfortunately we did not record this data during our research, we are unable to make such a comparison between the fuzzers. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  23. Guides for Future Research To perform a precise comparison of Zzuf and CC: • The difference between the number of test cases generated by Zzuf and CC for a given seed file and specific time frame should be recorded. • Measure CPU time to compare the number of unique test cases generated by each fuzzer for a given time. • Need a new method to identify unique errors avoid reporting duplicate bugs: • Need to use automatically generate a unique hash for each reported error that can then be used to identify duplicate errors. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  24. Guides for Future Research con’t 4. Use a more robust data collection infrastructure that could accommodate the massive amount of data colected. • Our ISP shut Metafuzz down due to excess server load. • Berkeley storage full. 5.Include an internal issue tracker that keeps track of whether or not a bug has been reported, to avoid reporting duplicate bugs. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  25. WhiteBox or BlackBox?? • With lower budget/ less time: use Blackbox • Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox • In practice, use both. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  26. Acknowledgment • National Science Foundation (NSF) for funding this project through the SUPERB-TRUST (Summer Undergraduate Program in Engineering Research at Berkeley - Team for Research in Ubiquitous Secure Technology) program • Kristen Gates (Executive Director for Education for the TRUST Program) • Faculty advisor David Wagner • Graduate mentors Li-Wen Hsu, David Molner, Edwardo Segura, Alex Fabrikant, and Alvaro Cardenas. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani

  27. Questions? Thank you "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs", Marjan Aslani Thank you! Thank you! Questions? Questions?

More Related