Why Security Testing Is Hard

1. Why Security Testing Is Hard Herbert H. Thompson Presenter: Alicia Young

2. Introduction • Software Testing good at verifying requirements • UML helps move from specification to test cases • Several bugs routinely escape testing • Not specification Violations • Would escape most automated testing • Examine Security bugs to discover why testing can be difficult

3. Side-Effect Behavior • Input A -> result B • What if Input A also resulted in C? • Overt – unexpected dialog box appears • Subtle – writing a file or opening a network port • RDISK utility for Windows • Creates an emergency Repair Disk • Temporary file created with Universal Permissions • During testing, product responds as specified

4. Intended vs. Implemented

5. The State of Security Testing • Exploit Libraries (Librarian Method) • New Products tested with only this library • Finds old vulnerabilities with no hope of finding anything new • Problem is…this strategy actually works! • Developers repeatedly make the same mistakes • Current software is really buggy • Applications will eventually become immune to these test cases

6. The Need for Techniques • Test like detectives • Past bugs teach us how vulnerabilities get into our applications • The key is to learn new techniques of finding bugs • Four General Classes of testing techniques • Dependencies • Unanticipated user input • Techniques to expose Design Vulnerabilities • Techniques to expose implementation vulnerabilities

7. Dependency Insecurities and Failures • Software resides in co-dependent environment • Two Security Concerns • Application may inherit insecurities • External security service resource may fail • Internet Explorer’s Content Advisor • Content advisor password protects classes of sites • If the library fails to load, Internet explorer permits access to any previously blocked site

8. Cause of Dependency Failures • Severely under-applied inputs to software • Error handling code gets little testing scrutiny • These types of failures need to be examined

9. Unanticipated User Input • Inputs that cause undesirable side effects and require special testing • Reserved words • Escape characters • Long strings • Boundary values • Most well known side-effect: Buffer Overflow • Input that can be interpreted as commands

10. Design Insecurities • Many Security Vulnerabilities designed into application • Seeing high-level impact on an application or host is difficult • Test Instrumentation • Many applications shipped with it • Bypassing security controls for ease of testing • Ports left open • Insecure default values and configurations

11. Implementation Insecurities • Perfect design means nothing if Implementation is flawed • Man-in-the-middle attack • Attacker gets between time application checks security and when the application uses information • Xterm – can be exploited to allow a restricted user to append data to the password file

12. Standard Bug-Severity Rankings • Urgent • System crash, Unrecoverable data loss, jeopardizes personnel • High • Impairment of critical system functions and no work-around exists • Medium • Impairment of critical system functions and work-around exists • Low • Inconvenience, annoyance • None • None of the above or an enhancement

13. The Need For Tools • Testers generally rewarded for both quantity and severity of bugs • Side-effect bugs may not get noticed or even dismissed by managers • Equipped with proper tools testers would notice odd behavior • Writing of a temporary file • Sending of extra network packets

14. New Tools • Regmon and Filemon – monitor application interactions with registry and file system • www.sysinternals.com • App-Sight – monitors environmental interactions • www.identify.com • Holodeck – Fine grain control over interactions between application and environment • www.sisecure.com

15. Paper Analysis • Quality Software is Secure Software • Important points made • Better testing techniques • Better testing tools • Design concerns

16. Questions?