Robust Arbitrary Binary Tester for Dyninst: Enhancing Automated Testing Capabilities

1. parseThat: A Robust Arbitrary-Binary Tester for Dyninst Ray Chen

2. Motivation • Extensive test suite exists • 118 individual sub-tests in 15 major categories • Attach mode flag • Relocate mode flag • Multiple mutatee compilers (GCC, Native) • Multiple mutatee languages (C/C++/Fortran) • Total of over 700 possible individual tests per platform! • Currently 8 platforms tested each night

3. Motivation • Tests written to exercise functionality • Mutatees tailored for mutators • Involve simple/artificial code • Test1.35 involves hand modified assembly output of gcc to trigger function relocation • Poor stress tests of Dyninst performance and capabilities • Mutatees relatively simple compared to real-world applications

4. Motivation • Goals • Allow regular, automated testing on production-scale application binaries • Firefox, GCC, MySQL • Allow a useful way for API users to easily report faults back to developers

5. Monitor Mutator Mutatee Overview • Three-tiered process hierarchy • Testing methodology • Snippets in mutatee kept simple • Mutator wraps API calls and announces results • Monitor records mutator state and watches for abnormal conditions

6. Basic Usage • Operational Modes • parseThat <file_name> <program args> • parseThat <dir_name> • parseThat –c <batch_file> • Parsing Levels • Module • Function • CFG • Instrumentation Levels • No instrumentation • Function Entry • Function Exit • Basic Block • Memory Access

7. Inserted Snippets • Keep instrumentation simple • Goal is to test range of mutatee support • Pre-insertion • Force mutatee to allocate integer from heap • One integer for each level of instrumentation granularity specified per function • Snippet • Increment integer associated with point of insertion, like function entry

8. Mutator Responsibilties • Wrap Dyninst API calls • Before each Dyninst API call • Announce API routine name • After return from API call • Announce result (Success, Warning, Failure) • Report mutatee status • Announce values of inserted counters • Announce abnormal mutatee exit

9. Mutator Output • Modeled after traditional Internet protocols (HTTP, FTP, etc) • Driven by numeric identifiers • Designed to be readable by humans • But also easily parsed by other programs • Transfers data and state information • Message Structure • <Message> ::= <MsgID> <Data> <Description> • <MsgID> ::= <ID><Verbosity><Status>

10. Mutator Output • Example message: 01a73 “” Test Message 3 Digit Hexadecimal Numeric Identifier 1 Digit Severity Identifier 1 Digit Status Identifier Data String (Possibly quoted) Description String

11. Mutator Output • Sample output 01031 sigalrm Attempting to instrument function. 01141 "" Attempting to instrument function entry. 01c61 "" Instructing mutatee to call malloc(). 01c64 "" Success. 02151 "" Inserting counter-increment snippet. 02154 "" Success. 01144 "" Success. 01034 "" Success.

12. Monitor Responsibilities • Interprets mutator output • Provides internal stack for presenting mutator messages to user • Based on verbosity value • User decides how much information they want to see • History records • Provides ability to track changes across runs • Trace pipe • Provides ability to track progress of generated code

13. Monitor Interpreter • Monitor output [ Processing funccall ] =================================== Creating new BPatch object.... done. Requesting notification of mutatee exit.... done. Requestion notification of mutator fork.... done. Forking mutatee process.... Mutatee launched as process 151690. Retrieving BPatch_image object.... done. Parsing image for module information.... Found 20 module(s). Parsing function data from funccall.c module... Found 6 function(s). Parsing CFG data from functions in funccall.c module... 6 warning(s).

14. History Records • Did it work? • Cannot determine success from mutatee alone • Mutatee output? • Mutatee return value? • Track history in logs • Record all output from monitor and mutator • Collect cumulative data for comparison • Report changes detected • Allow user to ultimately determine success

15. History Records • Track mutatees by signature • History records must be locatable by mutatee • Hash of mutatee should provide unique ID • Don’t forget the shared libraries! • An upgraded shared library can significantly alter the output of a mutatee • Mutatee signature • SHA-1 of binary • List of program arguments • SHA-1 of dependant libraries

16. History Records • Each successive run appends output to history record • Differences from latest run displayed to user • Allows us to achieve automated testing on production-scale binaries • Logging diff output over time is a valid regression test

17. Errors In Generated Code • Only API function errors checked so far • Entire class of faults possible after mutatee continues execution • How to report faults of this class? • Almost always results in abnormal mutatee exit • All information is lost when fault detected • Need a method to isolate fault location

18. Errors In Generated Code • Fault Isolation • New userEventCallback feature • Mutatee sends data through inefficient debug channels • Force mutatee to call printf at each instPoint • Additional stream or file to manage • Take best of both worlds: Named Pipe • Written to by mutatee, read by mutator • Mutatee reports at function entry and exit • Mutator forwards data to monitor • Last N trace points stored in monitor

19. Advanced Usage • Advanced Dyninst features available • Attach to existing process • Save-the-World • Enable merge tramps • Enable transactions • Additional parseThat features • Timeout value • Include and exclude functions/modules by regular expression

20. Avalibility • To be released with DyninstAPI 5.0 • Demo tomorrow