Efficient Instrumentation for Code Coverage Testing

1. Efficient Instrumentation forCode Coverage Testing Mustafa M. Tikir Jeffrey K. Hollingsworth

2. Evaluation of Code Coverage • Measuring code coverage is important to • Identify unexecuted program statements • Verify that each path is taken at least once • Requires extensive instrumentation • Must determine if every statement is executed

3. Instrumentation for Code Coverage • Traditional Approach • Static instrumentation using counters • Instrumentation code remains for entire execution • Conservative instrumentation of all possibly needed instrumentation code for all functions • Useless instrumentation wastes time • especially for long running programs such as servers and enterprise software • Full instrumentation increases setup time

4. Our Approach • Insert instrumentation code dynamically • Pre-Instrument all functions at program start • On-Demand instrument at first function call • Periodically remove instrumentation • Use dominator trees • Reduces instrumentation within a function • Use incremental function instrumentation • Insertion of instrumentation on first call • Reduced which function gets instrumented

5. Dyninst API • Modify code in a running program • Implementations available for • Alpha,Sparc,Power,Mips and x86 architectures • A mutator program • Generates machine code from high-level code • Transfers machine code to a running mutatee program Base Tramp Mini Tramp Program • Mutatee is the application being instrumented • Base-trampoline contains • relocated instructions • slots for calling code • Mini-trampoline stores the inserted code Save Registers Pre Setup Args Relocated Instruction Code Snippet Restore Registers Post

6. C B A Using Dominator Trees • Definitions: • A dom B if all paths from entry to basic block B goes through basic block A • A idom B if, for all C, (C != A) and (C dom B) implies (C dom A) • Fact: • If a basic block, B, is executed • all basic blocks along the path from B to the root of dominator tree also execute

7. Leaf Node Instrumentation • Leaf nodes in dominator tree are instrumented • Coverage of internal nodes will be inferred • Coverage information propagated from leaf nodes to entry Dominator Tree Control Flow Graph Entry Entry 1 1 3 2 3 2 5 4 4 5 Exit Exit

8. Non-Leaf Node Instrumentation • Leaf node instrumentation is necessary but not sufficient • Control flow might cause cross edges in dominator tree Control Flow Graph Dominator Tree Entry Entry 2 1 1 2 3 3 Exit Exit • We also instrument basic block, A, if • A has at least one outgoing edge to basic block, B, and • A does not dominate B

9. At Start Create CFG and dominator trees for all functions Instrument the basic blocks selected During Execution Stop the execution at fixed time intervals Delete the executed instrumentation code At Termination Propagate and collect the coverage information At Start Insert breakpoints at each function entry During Execution On breakpoint Identify the function Create CFG and dominator tree Instrument the basic blocks selected At fixed time intervals delete the executed code At Termination Propagate and collect the coverage information Code Coverage Algorithm Pre-Instrumentation On-Demand Instrumentation

10. Reduction In Instrumentation Points • 34-49% with pre-instrumentation • 42-79% with on-demand instrumentation

11. SPEC/compress Coverage Curve • Covers 76% in the first 18% of the execution • Most of the basic blocks that will execute are covered at the beginning of the program • Rest of the run is re-executions

12. SPEC/compress Execution Time • Purecov is a commercial state of art code coverage tool • Our code coverage tools outperform purecov • Significant reduction when dynamic code deletion is enabled • Most of the instrumentation code is deleted at the beginning • Setup time and deletion overhead is insignificant • A few number of basic blocks to instrument and check

13. PostgreSQL Coverage Curve • Wisconsin benchmark queries • Measure the performance of database systems • Executes select/join queries repeatedly

14. PostgreSQL Execution Time Execution Time Setup, Instrumentation and Deletion Time • Using on-demand instrumentation • Our coverage tools outperform purecov almost always • On-demand instrumentation outperforms pre-instrumentation

15. Instrumentation Execution Frequency • Overhead of our dynamic code coverage system is bursty • Running previously unexecuted code results in the execution of a significant amount of instrumentation • Running previously executed code does not result in the execution of any instrumentation

16. Overall Slowdown • Results for our code coverage tool are for 2-second deletion • Slowdown using purecov ranges from 1.83 to 19.78 • Slowdown using our code coverage tools range • From 1.002 to 2.6 for on-demand instrumentation • From 1.002 to 4.96 for pre-instrumentation

17. Dyninst Coverage Tool

18. Conclusions • Dominator trees • Reduce instrumentation points by 34-49% • Plus on-demand instrumentation reduce instrumentation points by 42-79% • Combining dominator trees and on-demand instrumentation reduces • Setup time and deletion interval overhead • Runtime overhead by 38-90% compared to purecov • Dynamic deletion of instrumentation • Computes coverage information faster

19. Conclusions (cont) • Code Coverage overhead reduced to about 10% • Code coverage can now be included as part of production code • Information about the execution of extremely infrequent error cases will be provided • Reduced overhead for residual testing • Dyninst library + Dyninst Coverage tools • http://www.dyninst.org