ConcJUnit: Unit Testing for Concurrent Programs

1. ConcJUnit:Unit Testing for Concurrent Programs COMP 600 Mathias Ricken Rice University August 24, 2009

2. Brian Goetz, Java Concurrency in Practice, Addison-Wesley, 2006

3. Concurrency Practiced Badly [1]

4. Unit Tests… • Occur early • Automate testing • Keep the shared repository clean • Serve as documentation • Prevent bugs from reoccurring • Allow safe refactoring

5. Existing Unit Testing Frameworks • JUnit, TestNG • Don’t detect test failures in child threads • Don’t ensure that child threads terminate • Tests that should fail may succeed

6. ConcJUnit • Replacement for JUnit • Backward compatible • Just replace junit.jar file • Detects failures in all threads • Warns if child threads outlive main thread • Available at www.concutest.org

7. Sample JUnit Tests publicclass Test extends TestCase { public void testException() { thrownew RuntimeException("booh!"); } public void testAssertion() { assertEquals(0, 1); } } Both tests fail. Both tests fail. } if (0!=1) throw new AssertionFailedError();

8. JUnit Test with Child Thread Main thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Child thread end of test spawns Main thread success! uncaught! Child thread

9. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); Uncaught exception, test should fail but does not! • By default, no uncaught exception handler installed for child threads

10. Changes to JUnit • Thread group with exception handler • JUnit test runs in a separate thread, not main thread • Child threads are created in same thread group • When test ends, check if handler was invoked

11. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); invokes checks TestGroup’s Uncaught Exception Handler

12. JUnit Test with Child Thread publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); spawns and joins resumes Main thread failure! check group’s handler end of test Test thread uncaught! invokes group’s handler Child thread

13. Child Thread Outlives Parent publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); check group’s handler Main thread success! Test thread Too late! end of test uncaught! invokes group’s handler Child thread

14. Well-Formed Tests • Uncaught exceptions and failed assertions in all threads cause failure • If the test is declared a success before all child threads have ended, failures may go unnoticed • Therefore, all child threads must terminate before test ends • Check for living child threads after test ends

15. Check for Living Child Threads publicclass Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); } } new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }).start(); thrownew RuntimeException("booh!"); check for living child threads check group’s handler Main thread failure! Test thread end of test uncaught! invokes group’s handler Child thread

16. Correctly Written Test publicclass Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { /* child thread */ } }); t.start(); t.join(); } } Thread t = new Thread(new Runnable() { public void run() { /* child thread */ } }); t.start(); t.join(); // wait until child thread has ended /* child thread */ check for living child threads check group’s handler Main thread success! Test thread end of test Child thread [4]

17. Well-Formed Tests Use Join • Uncaught exceptions and failed assertions in all threads cause failure • Therefore, all child threads must terminate before test ends • Without join() operation, a test may get “lucky” • Require all child threads to be joined

18. Fork/Join Model • Parent thread joins with each of its child threads • May be too limited for a general-purpose programming language Main thread Child thread 1 Child thread 2

19. Join with All Offspring Threads • Main thread joins with all offspring threads, regardless of what thread spawned them Main thread Child thread 1 Child thread 2

20. Join with Last Thread of Chain • Chain of child threads guaranteed to outlive parent • Main thread joins with last thread of chain Main thread Child thread 1 Child thread 2 Child thread 3

21. Generalize to Join Graph • Threads as nodes; edges to joined thread • Test is well-formed as long as all threads are reachable from main thread Main thread MT Child thread 1 CT1 Child thread 2 CT2 Child thread 3 CT3

22. Join Graph Examples Main thread MT Child thread 1 CT1 Child thread 2 CT2 Main thread MT Child thread 1 CT1 Child thread 2 CT2

23. Unreachable Nodes • An unreachable node has not been joined • Child thread may outlive the test Main thread MT Child thread 1 CT1 Child thread 2 CT2

24. Constructing the Graph • In Thread.start()– add node for this • End of Thread.join()– add edge from Thread.currentThread() to this current thread MT this CT [3]

25. Modifying the Java Runtime • Changing Thread.start()and join() • Need to modify Java Runtime Library • Utility to process user’s rt.jar file • Put new jar file on boot classpath:-Xbootclasspath/p:newrt.jar

26. Modifying the Java Runtime • May not always be possible • ~30 MB hard drive space needed • Access to boot classpath option required • ConcJUnit written to work without it • Just no join graph and “lucky” warnings [5]

27. Evaluation • JFreeChart • All tests passed; tests are not concurrent • DrJava: 900 unit tests • Passed: 880 • No join: 1 • Lucky: 18 • Timeout: 1 • Runtime overhead: ~1 percent

28. Limitations • Only checks chosen schedule • A different schedule may still fail • Example: Thread t = new Thread(…);if (nondeterministic()) t.join(); [2]

29. Future Work • Randomly insert sleeps or yields • Example: If a notify() is delayed, a wait() may time out. • Can detect a number of sample problems • Record schedule, replay if test fails • Makes failures reproducible if found [6]

30. Thank you! www.concutest.org

31. Extra Slides

32. Notes • Probably not caused by concurrency problems; just a metaphor. ← • Also cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation) ← • Only add edge if joined thread is really dead; do not add if join ended spuriously. ←

33. Spurious Wakeup publicclass Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } } } Thread t = new Thread(new Runnable() { public void run() { thrownew RuntimeException("booh!"); } }); t.start(); while(t.isAlive()) { try { t.join(); } catch(InterruptedException ie) { } } thrownew RuntimeException("booh!"); Loop since join() may end spuriously ←

34. Thread Creation Context • In Thread.start()– also record stack trace of Thread.currentThread( • Easy to find source code of a child thread that is not joined • Also available for uncaught exception stack traces

35. Creation Context Example class Helper extends Thread { void m() { Assert.fail(); } public void run() { m(); } } class Main { void foo() { // which one? new Helper().start(); new Helper().start(); // ... } } AssertionError: at Helper.m(Helper.java:2) at Helper.run(Helper.java:3) Started at: at Main.foo(Main.java:4) at Main.bar(Main.java:15) at Main.main(Main.java:25) ←

36. Notes • Have not studied probabilities or durations for sleeps/yields:One inserted delay may negatively impact a second inserted delayExample: If both notify() and wait() are delayed. ←

37. Many Thanks To… • My advisor • Corky Cartwright • My committee members • Walid Taha • David Scott • Bill Scherer • NSF and Texas ATP • For providing partial funding

38. Concurrency Problems NotFound During Testing [1]