Playing Inside the Blackbox: Using Dynamic Instrumentation to Create Security Holes

1. Playing Inside the Blackbox:Using Dynamic Instrumentation to Create Security Holes Barton P. Miller bart@cs.wisc.eduComputer Sciences Department University of Wisconsin Madison, Wisconsin 53705 USA DynInst Security

2. Overview 1. How to easily do dangerous and malicious things to a running program. 2. How to detect when someone does something evil to your program. DynInst Security

3. A New View Running programs are objects to be easily manipulated. Kinds of manipulations might include: • Instrumentation • Optimization • Control DynInst Security

4. The Vehicle:The DynInst API A machine-independent library for machine level code patching. • Eases the task of building new tools. • Provides the basic abstractions to patch code on-the-fly DynInst Security

5. Dynamic Instrumentation • Does not require recompiling or relinking • Saves time: compile and link times are significant in real systems. • Can instrument without the source code (e.g., proprietary libraries). • Can instrument without linking (relinking is not always possible. • Instrument optimized code. DynInst Security

6. Dynamic Instrumentation (con’d) • Only instrument what you need, when you need • No hidden cost of latent instrumentation. • Enables “one pass” tools. • Can instrument running programs (such as Web or database servers) • Production systems. • Embedded systems. • Systems with complex start-up procedures. DynInst Security

7. Pre-Instrumentation Relocated Instruction Post-Instrumentation The Basic Mechanism ApplicationProgram Trampoline Function foo DynInst Security

8. The DynInst Interface • Machine independent representation • Object-based interface to build Abstract Syntax Trees (AST’s) • Write-once, instrument-many (portable) • Hides most of the complexity in the API • Process Hijacker: only 700 lines of user code! • MPI tracer: 250 lines DynInst Security

9. Basic DynInst Operations • Process control: • Attach/create process • Monitor process status changes • Callbacks for fork/exec/exit • Image (executable program) routines: • Find procedures/modules/variables • Call graph (parent/child) queries • Intra-procedural control-flow graph DynInst Security

10. Basic DynInst Operations • Inferior (application processor) operations: • Malloc/free • Allocate heap space in application process • Inferior RPC • Asynchronously execute a function in the application. • Load module • Cause a new .so/.dll to be loaded into the application. DynInst Security

11. Basic DynInst Operations • Inferior operations (continued): • Remove Function Call • Disable an existing function call in the application • Replace Function Call • Redirect a function call to a new function • Replace Function • Redirect all calls (current and future) to a function to a new function. DynInst Security

12. Basic DynInst Operations • Building AST code sequences: • Control structures: if and goto • Arithmetic and Boolean expressions • Get PID/TID operations • Read/write registers and global variables • Read/write parameters and return value • Function call DynInst Security

13. Security Applications of DynInst Lots of tool applications of Dyninst by lots of groups. Here are two security-oriented ones: • License server bypassing • Condor security attacks DynInst Security

14. License Server Program License Data Network Undesired: licensed program refuses to run if license server does not respond. License Server Attack: The Bypass License Server Program License Data Network Normal: licensed program runs after communicates with license server. DynInst Security

15. Example: Adobe FrameMaker Two-step license verification: • retrieve license data from server [once] • check license data for correctness [often] In practice: • allow FM to time-out waiting for server • allow FM to attempt to go into “demo” mode • switch FM back to normal mode • insure that future license checks always succeed DynInst Security

16. Strategies • Complete reverse engineering: • not an option • legal problems • complexity (FrameMaker is a 7 MB binary!) • Focus on certain characteristics: • I/O (network sockets) traffic • execution trace DynInst Security

17. Tools • High-level language translators: • Dyner: interactive, interpreted C subset • Jdyninst: Java to DynInst compiler • Bypasser: an interactive call graph browser • Search and walk application call graph • Resolves function pointers at runtime • Call follow caller or callee paths • Can generate call trace DynInst Security

18. Use • Determining where to apply changes: • get trace for a successful run • get trace for a (forced-)failure run • compare to find differences • repeat as needed DynInst Security

19. Details • FM calls NlOpenlicenses on start up • Contacts license server and caches credential if successful • At end of main, FM calls NluiCheckLicense • If credential is not present, call ChangeProductToDemo (cannot save files) • Frequently, during operation, FM will check for cached credentials. DynInst Security

20. Details • FM calls NlOpenlicenses on start up • Contacts license server and caches credential if successful • Allow this to fail. • At end of main, FM calls NluiCheckLicense • If credential is not present, call ChangeProductToDemo (cannot save files) • Frequently, during operation, FM will check for cached credentials. DynInst Security

21. Details • FM calls NlOpenlicenses on start up • Contacts license server and caches credential if successful • Allow this to fail. • At end of main, FM calls NluiCheckLicense • If credential is not present, call ChangeProductToDemo (cannot save files) • Delete the call to ChangeProductToDemo. • Frequently, during operation, FM will check for cached credentials. DynInst Security

22. Details • FM calls NlOpenlicenses on start up • Contacts license server and caches credential if successful • Allow this to fail. • At end of main, FM calls NluiCheckLicense • If credential is not present, call ChangeProductToDemo (cannot save files) • Delete the call to ChangeProductToDemo. • Frequently, during operation, FM will check for cached credentials. • Change this call to always return “true”. DynInst Security

23. Condor Attack: Lurking Jobs • Condor schedules jobs on idle workstations • In a normal mode, jobs run as a common, low-privilege user ID: “nobody”. • This common user ID provides an opportunity for an evil lurking process to ambush subsequent jobs (from other users): DynInst Security

24. system calls Condor Job Structure Shadow Process User Job Submitting Host Execution Host DynInst Security

25. fork Lurker Process Condor Job Structure Shadow Process Evil User Job system calls Submitting Host Execution Host DynInst Security

26. Condor Job Structure Lurker Process Submitting Host Execution Host DynInst Security

27. system calls Condor Job Structure Shadow Process Innocent User Job Lurker Process Submitting Host Execution Host DynInst Security

28. system calls Condor Job Structure Shadow Process Innocent User Job attach Lurker Process Submitting Host Execution Host DynInst Security

29. system calls Condor Job Structure Shadow Process Innocent User Job rm -rf * Control remote system calls Lurker Process Submitting Host Execution Host DynInst Security

30. DynInst Security

31. Can We Trust a Remote Job? The threats: 1. Cause the job to make improper remote system calls. 2. Cause the job to calculate an incorrect answer. 3. Steal data from the remote job. Threat protection strategies: • File sand-boxing (#1) • System call sand-boxing (#1) • Obscure and encode binary (#1) • Replicate remote job (#2) DynInst Security

32. system calls Sand-Boxing Shadow Process User Job Shadow process selectively rejects system calls: • Restrict access to specific files or directories • Disallow certain system calls • Disallow certain system call parameter values Submitting Host Execution Host DynInst Security

33. Obscuring the Executable User Job Modifier/Obscurer Checking Shadow Modified User Job DynInst Security

34. Obscuring the Executable Goal: Even if an intruder can see, examine, and fully control the remote job, no harm can come to the local machine. DynInst Security

35. Obscuring the Executable Modify the executable: • Replace each system call site with call to a unique random function name This makes understanding the executable harder and makes checking the system call stream easier. • Delete any constant parameters Values like file names or descriptor numbers are hidden. • Pad parameter other lists so all calls have equal number • Permute the order of parameters • Add dummy system calls throughout code DynInst Security

36. Obscuring the Executable Generate a Checking Shadow: • Build a FSM that reflects the inter-procedural control-flow of the job. Note that this FSM has increased accuracy, since each call site has a unique name • Map random function names back to their actual system calls. • Insert parameter values and present in correct order. • Ignore dummy calls. DynInst Security

37. Obscuring the Executable Some notes of interest: • All hiding and replacement is done before the job is submitted to Condor. No knowledge of the encryption or keys is contained in the remote job. • All replacing is done in the Checking Shadow, which only resides on your local machine. • FSM accuracy and computational complexity is simplified by the function renaming. Our early results look good. A tech report will be available by the end of November. DynInst Security

38. How to Get a Copy of DynInst: Release 2.3 (release 3.0 imminent) • Free for research use. • Runs on Solaris (SPARC & x86), Windows NT, AIX/SP2, Linux (x86), Irix (MIPS),Tru64 Unix (Alpha). http://www.paradyn.org http://www.dyninst.org paradyn@cs.wisc.edu DynInst Security