1 / 68

A Framework for Testing Concurrent Programs

A Framework for Testing Concurrent Programs. MS Thesis Defense Mathias Ricken Rice University June 14, 2007. Organization. Introduction Unit Testing and Concurrency Tractability Contributions Improvements to JUnit Annotations for Concurrency Invariants Bytecode Rewriting Framework

shanna
Download Presentation

A Framework for Testing Concurrent Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A Framework for Testing Concurrent Programs MS Thesis Defense Mathias Ricken Rice University June 14, 2007

  2. Organization • Introduction • Unit Testing and Concurrency • Tractability • Contributions • Improvements to JUnit • Annotations for Concurrency Invariants • Bytecode Rewriting Framework • Other Contributions • Conclusion • Future Work

  3. Unit Testing Program Sub- program Sub- program Sub- program ? ? ? ? Difficult Even less difficult Less difficult

  4. Unit Testing • Unit tests… • Test a part, not the whole program • Occur earlier • Automate testing • Serve as documentation • Prevent bugs from reoccurring • Help keep the shared repository clean • Effective with a single thread of control

  5. Foundation of Unit Testing • Unit tests depend on deterministic behavior • Known input, expected output…Success  correct behaviorFailure  flawed code • Outcome of test is meaningful

  6. Problems Due to Concurrency • Thread scheduling is nondeterministic and machine-dependent • Code may be executed under different schedules • Different schedules may produce different results • Known input, expected output…Success  correct behaviorin this schedule, may be flawed in other scheduleFailure  flawed code • Success of unit test is meaningless

  7. Timeliness of the Problem • Many programs already use concurrency • Often hidden, as part of GUI • Clock speeds have not increased much • Most speed increases are due to multiple cores on one chip • Concurrency must be used to benefit from newer CPUs • Increased use of concurrency in the future

  8. Possible Solutions • Programming Language Features • Race Freedom • Deadlock Freedom, Safe Locking • Atomicity, Transactions • Usually require changes in type system • Fundamental change • C++ standards: 1998, 2003, 200x • Java major changes: 1997 (1.0), 2002 (1.4), 2004 (1.5)

  9. Possible Solutions • Lock-Free Algorithms • Work on copy and assume no concurrency is present (or current thread will finish first) • If there was interference, threads that don’t finish first redo their work • Require some system support (e.g. compare-and-swap), then done in libraries • Not all common data structures are practical and efficient to implement as lock-free

  10. Possible Solutions • Programming Language Features • Ensuring that bad things cannot happen • May restrict programmers • Lock-Free Algorithms • Ensuring that if bad things happen, it’s ok • May limit data structures available • Comprehensive Testing • Testing if bad things happen in any schedule • Does not prevent problems, but does not limit solutions either

  11. Tractability of Comprehensive Testing • Deciding whether any given program contains an error is undecidable • Reduction to the halting problem • Program expected to meet an assertion Requires that the program halts first • Program expected to not halt Requires inverse of halting problem • Does not imply undecidable for all programs

  12. Tractability of Comprehensive Testing • Test all possible schedules • Concurrent unit tests meaningful again • Number of schedules (N) • t: # of threads, s: # of slices per thread detail

  13. Tractability of Comprehensive Testing • If program is race-free, we do not have to simulate all thread switches • Threads interfere only at “critical points”: lock operations, shared or volatile variables, etc. • Code between critical points cannot affect outcome • Simulate all possible arrangements of blocks delimited by critical points • Run dynamic race detection in parallel • Lockset algorithm (e.g. Eraser by Savage et al)

  14. Critical Points Example Local Var 1 All accesses protected by lock lock access unlock lock access unlock Thread 1 Shared Var Lock Local variables don’t need locking All accesses protected by lock All accesses protected by lock Thread 2 lock access unlock Local Var 1

  15. Fewer Schedules • Fewer critical points than thread switches • Reduces number of schedules • Example: Two threads, but no communication N = 1 • Unit tests are small • Reduces number of schedules • Hopefully comprehensive simulation is tractable • If not, heuristics are still better than nothing

  16. Contributions • Improvements to JUnit • Detect exceptions and failed assertions in threads other than the main thread • Annotations for Concurrency Invariants • Express complicated requirements about locks and threads • Tools for Schedule-Based Execution • Record, deadlock monitor • Random delays, random yields

  17. Organization • Introduction • Unit Testing and Concurrency • Tractability • Contributions • Improvements to JUnit • Annotations for Concurrency Invariants • Bytecode Rewriting Framework • Other Contributions • Conclusion • Future Work

  18. Improvements to JUnit • Uncaught exceptions and failed assertions • Not caught in child threads

  19. Sample JUnit Tests publicclass Test extends TestCase { public void testException() { thrownew RuntimeException("booh!"); } public void testAssertion() { assertEquals(0, 1); } } } Both tests fail. Both tests fail. if (0!=1) throw new AssertionFailedError();

  20. end of test spawns Main thread success! uncaught! Child thread Problematic JUnit Tests Main thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread

  21. Problematic JUnit Tests Main thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread Uncaught exception, test should fail but does not!

  22. Improvements to JUnit • Uncaught exceptions and failed assertions • Not caught in child threads • Thread group with exception handler • JUnit test runs in a separate thread, not main thread • Child threads are created in same thread group • When test ends, check if handler was invoked

  23. Thread Group for JUnit Tests Test thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread invokes checks TestGroup’s Uncaught Exception Handler

  24. Thread Group for JUnit Tests Test thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread spawns and waits resumes Main thread failure! check group’s handler spawns end of test Test thread invokes group’s handler uncaught! Child thread

  25. Improvements to JUnit • Uncaught exceptions and failed assertions • Not caught in child threads • Thread group with exception handler • JUnit test runs in a separate thread, not main thread • Child threads are created in same thread group • When test ends, check if handler was invoked • Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end Past tense: occurred!

  26. Child Thread Outlives Parent Test thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread spawns and waits resumes Main thread failure! check group’s handler spawns end of test Test thread invokes group’s handler uncaught! Child thread

  27. Child Thread Outlives Parent Test thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread check group’s handler spawns and waits resumes Main thread success! spawns Too late! Test thread end of test uncaught! invokes group’s handler Child thread

  28. Improvements to JUnit • Child threads are not required to terminate • A test may pass before an error is reached • Detect if any child threads are still alive • Declare failure if test thread has not waited • Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.) • Previous schedule is a test failure • Should be prevented by using Thread.join()

  29. Enforced Join Test thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); … t.join(); } } Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); … t.join(); … thrownew RuntimeException("booh!"); Child thread

  30. Limitations • Improvements only check chosen schedule • A different schedule may still fail • Requires comprehensive testing to be meaningful • May still miss uncaught exceptions • Specify absolute parent thread group, not relative Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) details

  31. Testing ConcJUnit • Replacement for junit.jar or as plugin JAR for JUnit 4.2 • Available as binary and source at http://www.concutest.org/ • Results from DrJava’s unit tests • Child thread for communication with slave VM still alive in test • Several reader and writer threads still alive in low level test (calls to join() missing)

  32. Organization • Introduction • Unit Testing and Concurrency • Tractability • Contributions • Improvements to JUnit • Annotations for Concurrency Invariants • Bytecode Rewriting Framework • Other Contributions • Conclusion • Future Work

  33. Concurrency Invariants • Has to be called in event thread • TableModel, TreeModel • May not be called in event thread • invokeAndWait() • Have to acquire readers/writers lock • AbstractDocument • DrJava’s documents

  34. Invariants Difficult to Determine • May be found in • Javadoc comments • Only in internal comments • Whitepapers • Often not documented at all • Errors not immediately evident • Impossible to check automatically

  35. Java Annotations • Add invariants as annotations@NotEventThreadpublic static void invokeAndWait( Runnable r) { … } • Process class files • Find uses of annotations • Insert bytecode to check invariants at method beginning

  36. Advantages of Annotations • Java Language constructs • Syntax checked by compiler • Easy to apply to part of the program • e.g. when compared to a type system change • Light-weight • Negligible runtime impact if not debugging (slightly bigger class files) • Automatic Checking

  37. Predicate Annotations • In annotation definition, specify static boolean Java method • Method must be callable from every context completely static and public • Data in annotation, method arguments and value of this passed when method invoked

  38. Predicate Annotation Example @PredicateLink(value=Predicates.class, method="example", arguments=true) public @interface ExampleAnnotation { String foo; } Refers to Predicates.example Definition

  39. Predicate Annotation Example Usage public class TestCode { @ExampleAnnotation(foo="test") public void test(int param) { … } } … TestCode t = new TestCode(); t.test(5); Call

  40. Predicate Annotation Example public class Predicates { public static boolean example( Object this0, int param, String foo) { return (foo.length()<param); }

  41. Predicate Annotation Example @PredicateLink(value=Predicates.class, method="example", arguments=true) public @interface ExampleAnnotation { String foo; } public class TestCode { @ExampleAnnotation(foo="test") public void test(int param){…} } … TestCode t = new TestCode(); t.test(5); public class Predicates { public static boolean example( Object this0, intparam, String foo) { return (foo.length()<param); // this0==t, param==5, foo=="test" }

  42. Invariant Annotation Library • @OnlyEventThread, @NotEventThread • @OnlyThreadWithName • @NotNullArgument • @DistinctArguments, @SameArguments • @OnlySynchronizedThis, @NotSynchronizedThis • @OnlySynchronizedArgument, @NotSynchronizedArgument • etc. (ca. 80 annotations)

  43. Problem: Multiple Annotations • Java does not allow the same annotation class multiple times @OnlyThreadWithName("foo") @OnlyThreadWithName("bar") // error void testMethod() { … } • Conjunctions, disjunctions and negations?

  44. Annotation Subclasses? • Let annotation extend a supertype? public @interface Invariant { } public @interface OnlyThreadWithName extends Invariant { String name(); } public @interface And extends Invariant { Invariant[] terms(); } • Subtyping not allowed for annotations

  45. Work-Around • Different meta-annotation, @Combine @Combine(Combine.Mode.AND) public @interface SeveralNames { OnlyThreadWithName[] value(); } @SeveralNames({@OnlyThreadWithName("foo"), @OnlyThreadWithName("bar")}) void testMethod() { … }

  46. Combine Annotations • May only contain invariant annotations • Predicate annotations • Combine annotations • Arrays of the above • Predicate method automatically generated • Calls member predicate methods • Accumulates using AND, OR or NOT • NOT first negates, then uses AND • Default mode is OR • De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)

  47. Invariants As Class Annotation • Short-hand for annotating all methods • What about methods already introduced in a super class, e.g. Object.notify()? • What about methods introduced in subclasses? @Invariant class A { void foo() { … } void bar() { … } } class A { @Invariant void foo() { … } @Invariant void bar() { … } }

  48. Invariants As Class Annotation • Apply invariants on a class only to methods introduced in the class or subclasses class A { public void foo() { … } } @Invariant // only applies to bar() class B extends A { public void bar() { … } }

  49. Invariant Inheritance • Invariants on a method • Apply to the method and all overriding methods in subclasses • Invariants on a class • Apply to all methods introduced in that class or subclasses  Methods can have invariants defined elsewhere • All annotations describe requirements for the client (and, due to subclassing, for subclasses) • Allows frameworks to describe requirements • Description “thread-safe” is often wrong

  50. Invariant Subtyping • To maintain substitutability, subclasses may not strengthen invariants • Invariants can be modeled as special input parameter • Tuple of invariants (“record” in λ calculus [Pierce]) • Subtyping rules for records declare the “wider” record as subtype • In function types, parameter types are contravariant I0 = {}, I1 = {inv1}, I2 = {inv1,inv2}, I2 <: I1 <: I0 F0 = I0→·, F1 = I1→·, F2 = I2→·, F0 <: F1 <: F2

More Related