1. By Matt Hargett <matt @ use.net> Software Security Without The Source Code

2. Introduction • Matt Hargett • Security QA Engineer for 7 years • NAI/McAfee, TurboLinux, Cenzic • Discovered many critical vulnerabilities • Created binary code analysis product • BugScan

3. Overview • Why we need to measure software security • Kinds of security policy testing • Whitebox approaches • Blackbox approaches • Effectiveness against real-world exploits • What you can do with this information

4. Why we need to measure • Brand and reputation damage • Proprietary information leakage • Unplanned disruption • Violation of privacy policy • Espionage • Terrorism

5. How do we measure • Whitebox • Manual code inspection • Source code static analysis • Binary static analysis • Runtime analysis • Blackbox • Web Applications • Network protocols • APIs

6. Blackbox Network Testing • Sniff, fuzz, replay • Sniff network traffic • Systematically fuzz the relevant data • Remove delimeters, bitwalk, ... • {admin.command\0::\43\89\42} • Replay fuzzed packets to server • Fuzzing via Proxy • Route traffic through a proxy • Proxy fuzzes data systematically • Fuzzed data gets passed on • Repeat client operation • Protocol-specific fuzzing • Make special client for specific protocol(s)

7. Blackbox Network Testing:In the Real World • Sniff, fuzz, replay • Not stateful • Doesn't work with encryption • Only fuzzes client-side • Fuzzing via Proxy • Lets real client/server handle state • Doesn't work with encryption • Fuzzes client and server data • Protocol-specific fuzzing • Handles real state of client/server • Does encryption itself • Can get great code coverage • Fuzzes client server and data

8. Blackbox Network Testing:General Jeers • Detecting when you've evoked a problem • Measuring code coverage • Slow process • Expensive to scale

9. Blackbox Web Testing • Sniff, fuzz, replay • Auto-crawl or manual clicks • Sniff browser requests • Systematically fuzz the relevant data • Insert SQL Injection, Command Injection, XSS, ... • POST /foo.cgi?name=bob&pass=... • Fuzzing via Proxy • Optionally crawl to generate requests • Send requests through a proxy • Proxy fuzzes requests systematically • Repeat browser operation • GUI Automation tools • Automate real browser interaction • Put bad data into form and cookie fields

10. Blackbox Web Testing:In the Real World • Sniff, fuzz, replay • Server-side state mishaps • Little to no javascript support • No flash, java, web service support • Finds WAY low-hanging fruit • Fuzzing via Proxy • Server-side state mishaps • Must have browser automation anyways • Operations must be self-contained • GUI Automation tools • Maintenance of stored tests • Tests must be self-contained • Can be part of standard QA

11. Blackbox Web Testing:General Jeers • Detecting when you've evoked a problem • Measuring code coverage • Slow process • Expensive to scale • Getting past captchas

12. Whitebox Testing Without Source • Manual review • Call-based static analysis • Pointer and control/data flow static analysis

13. Whitebox Testing Without Source:In the Real World • Manual review • Going through instruction by instruction • Tedious, time consuming, error prone, rare skill • Pair rev-enging, unit testing • Rare skill, unit test exploits may not be real-world • call-based analysis • 42,897 strcpy calls detected!!!!#%* • HttpResponse.GetValue() before Statement.Execute() • Dispose called inside Dispose • Pointer and control/data flow analysis • Must be inter-function to be useful • Must track global/static data • Inter-module tracking also important

14. Whitebox Testing Without Source:General Jeers • People • Hire manual reviewers with a proven track record of real-world exploitable bugs patched by a vendor • Tools • Difficult to use • Poor quality • False positives

15. Whitebox Testing With Source:General Jeers • Worthless when source not available • People • Hire manual reviewers with a proven track record of real-world exploitable bugs patched by a vendor • Tools • There is no free unit test lunch • Demand vendors demonstrate finding novel real-world exploitable bugs their tool finds OOTB • Demand vendors demonstrate finding previously known real-world exploitable bugs their tool finds OOTB • Demand third party vendor-neutral benchmarks • No good visualization/exploration tools for manual reviewers

16. Whitebox Testing:General Jeers, Source or Not • Code is code, period • Most tools are a retarded joke • False positive rates above 10% on large (100KLOC) means a useless tool • High priority report items should be real-world exploitable 95% of the time • Custom signatures shouldn't require extra cost, permission, or license • Tools • Demand vendors demonstrate finding novel real-world exploitable bugs their tool finds OOTB • Demand vendors demonstrate finding previously known real-world exploitable bugs their tool finds OOTB • Demand third party vendor-neutral benchmarks • No good visualization tools for manual reviewers • Zealousy on a given approach

17. Recommendations • Engage a holistic approach • Blackbox and Whitebox • Use multiple vendor tools to cross-check • Source and binary • Runtime and static • Use protocol-specific fuzzers • Ask vendor for code coverage on open source implementation(s) of said protocol(s) • Use UI automation tools for web apps • Any good tool will require tuning

18. Taking Action • If open source, fix the problem yourself • Contact vendor • If vendor cannot supply a fix in 30 days • Escalate the issue • Find a new vendor • If open source, fix the problem yourself • Vendors will string you along

19. Thank You matt @ use . net Questions?