a framework for testing concurrent programs
Download
Skip this Video
Download Presentation
A Framework for Testing Concurrent Programs

Loading in 2 Seconds...

play fullscreen
1 / 68

A Framework for Testing Concurrent Programs - PowerPoint PPT Presentation


  • 86 Views
  • Uploaded on

A Framework for Testing Concurrent Programs. MS Thesis Defense Mathias Ricken Rice University June 14, 2007. Organization. Introduction Unit Testing and Concurrency Tractability Contributions Improvements to JUnit Annotations for Concurrency Invariants Bytecode Rewriting Framework

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Framework for Testing Concurrent Programs' - shanna


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a framework for testing concurrent programs

A Framework for Testing Concurrent Programs

MS Thesis Defense

Mathias Ricken

Rice University

June 14, 2007

organization
Organization
  • Introduction
    • Unit Testing and Concurrency
    • Tractability
    • Contributions
  • Improvements to JUnit
  • Annotations for Concurrency Invariants
  • Bytecode Rewriting Framework
    • Other Contributions
  • Conclusion
    • Future Work
unit testing
Unit Testing

Program

Sub-

program

Sub-

program

Sub-

program

?

?

?

?

Difficult

Even less difficult

Less difficult

unit testing4
Unit Testing
  • Unit tests…
    • Test a part, not the whole program
    • Occur earlier
    • Automate testing
    • Serve as documentation
    • Prevent bugs from reoccurring
    • Help keep the shared repository clean
  • Effective with a single thread of control
foundation of unit testing
Foundation of Unit Testing
  • Unit tests depend on deterministic behavior
  • Known input, expected output…Success  correct behaviorFailure  flawed code
  • Outcome of test is meaningful
problems due to concurrency
Problems Due to Concurrency
  • Thread scheduling is nondeterministic and machine-dependent
    • Code may be executed under different schedules
    • Different schedules may produce different results
  • Known input, expected output…Success  correct behaviorin this schedule, may be flawed in other scheduleFailure  flawed code
  • Success of unit test is meaningless
timeliness of the problem
Timeliness of the Problem
  • Many programs already use concurrency
    • Often hidden, as part of GUI
  • Clock speeds have not increased much
    • Most speed increases are due to multiple cores on one chip
  • Concurrency must be used to benefit from newer CPUs
    • Increased use of concurrency in the future
possible solutions
Possible Solutions
  • Programming Language Features
    • Race Freedom
    • Deadlock Freedom, Safe Locking
    • Atomicity, Transactions
    • Usually require changes in type system
    • Fundamental change
      • C++ standards: 1998, 2003, 200x
      • Java major changes: 1997 (1.0), 2002 (1.4), 2004 (1.5)
possible solutions9
Possible Solutions
  • Lock-Free Algorithms
    • Work on copy and assume no concurrency is present (or current thread will finish first)
    • If there was interference, threads that don’t finish first redo their work
    • Require some system support (e.g. compare-and-swap), then done in libraries
    • Not all common data structures are practical and efficient to implement as lock-free
possible solutions10
Possible Solutions
  • Programming Language Features
    • Ensuring that bad things cannot happen
    • May restrict programmers
  • Lock-Free Algorithms
    • Ensuring that if bad things happen, it’s ok
    • May limit data structures available
  • Comprehensive Testing
    • Testing if bad things happen in any schedule
    • Does not prevent problems, but does not limit solutions either
slide11

Tractability of Comprehensive Testing

  • Deciding whether any given program contains an error is undecidable
    • Reduction to the halting problem
      • Program expected to meet an assertion Requires that the program halts first
      • Program expected to not halt Requires inverse of halting problem
  • Does not imply undecidable for all programs
tractability of comprehensive testing
Tractability of Comprehensive Testing
  • Test all possible schedules
    • Concurrent unit tests meaningful again
  • Number of schedules (N)
    • t: # of threads, s: # of slices per thread

detail

tractability of comprehensive testing13
Tractability of Comprehensive Testing
  • If program is race-free, we do not have to simulate all thread switches
    • Threads interfere only at “critical points”: lock operations, shared or volatile variables, etc.
    • Code between critical points cannot affect outcome
    • Simulate all possible arrangements of blocks delimited by critical points
  • Run dynamic race detection in parallel
    • Lockset algorithm (e.g. Eraser by Savage et al)
critical points example
Critical Points Example

Local Var 1

All accesses protected by lock

lock access unlock

lock access unlock

Thread 1

Shared Var

Lock

Local variables don’t need locking

All accesses protected by lock

All accesses protected by lock

Thread 2

lock access unlock

Local Var 1

fewer schedules
Fewer Schedules
  • Fewer critical points than thread switches
    • Reduces number of schedules
    • Example: Two threads, but no communication N = 1
  • Unit tests are small
    • Reduces number of schedules
  • Hopefully comprehensive simulation is tractable
    • If not, heuristics are still better than nothing
contributions
Contributions
  • Improvements to JUnit
    • Detect exceptions and failed assertions in threads other than the main thread
  • Annotations for Concurrency Invariants
    • Express complicated requirements about locks and threads
  • Tools for Schedule-Based Execution
    • Record, deadlock monitor
    • Random delays, random yields
organization17
Organization
  • Introduction
    • Unit Testing and Concurrency
    • Tractability
    • Contributions
  • Improvements to JUnit
  • Annotations for Concurrency Invariants
  • Bytecode Rewriting Framework
    • Other Contributions
  • Conclusion
    • Future Work
improvements to junit
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
sample junit tests
Sample JUnit Tests

publicclass Test extends TestCase {

public void testException() {

thrownew RuntimeException("booh!");

}

public void testAssertion() {

assertEquals(0, 1);

}

}

}

Both tests fail.

Both tests fail.

if (0!=1)

throw new AssertionFailedError();

problematic junit tests

end of test

spawns

Main thread

success!

uncaught!

Child thread

Problematic JUnit Tests

Main thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

problematic junit tests21
Problematic JUnit Tests

Main thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

Uncaught exception, test should fail but does not!

improvements to junit22
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
  • Thread group with exception handler
    • JUnit test runs in a separate thread, not main thread
    • Child threads are created in same thread group
    • When test ends, check if handler was invoked
thread group for junit tests
Thread Group for JUnit Tests

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

invokes

checks

TestGroup’s Uncaught Exception Handler

thread group for junit tests24
Thread Group for JUnit Tests

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

spawns and waits

resumes

Main thread

failure!

check group’s handler

spawns

end of test

Test thread

invokes group’s handler

uncaught!

Child thread

improvements to junit25
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
  • Thread group with exception handler
    • JUnit test runs in a separate thread, not main thread
    • Child threads are created in same thread group
    • When test ends, check if handler was invoked
  • Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end

Past tense: occurred!

child thread outlives parent
Child Thread Outlives Parent

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

spawns and waits

resumes

Main thread

failure!

check group’s handler

spawns

end of test

Test thread

invokes group’s handler

uncaught!

Child thread

child thread outlives parent27
Child Thread Outlives Parent

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

check group’s handler

spawns and waits

resumes

Main thread

success!

spawns

Too late!

Test thread

end of test

uncaught!

invokes group’s handler

Child thread

improvements to junit28
Improvements to JUnit
  • Child threads are not required to terminate
    • A test may pass before an error is reached
  • Detect if any child threads are still alive
    • Declare failure if test thread has not waited
    • Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)
  • Previous schedule is a test failure
    • Should be prevented by using Thread.join()
enforced join
Enforced Join

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

});

t.start(); … t.join();

}

}

Thread t = new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

});

t.start(); … t.join(); …

thrownew RuntimeException("booh!");

Child thread

limitations
Limitations
  • Improvements only check chosen schedule
    • A different schedule may still fail
    • Requires comprehensive testing to be meaningful
  • May still miss uncaught exceptions
    • Specify absolute parent thread group, not relative Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)

details

testing concjunit
Testing ConcJUnit
  • Replacement for junit.jar or as plugin JAR for JUnit 4.2
  • Available as binary and source at http://www.concutest.org/
  • Results from DrJava’s unit tests
    • Child thread for communication with slave VM still alive in test
    • Several reader and writer threads still alive in low level test (calls to join() missing)
organization32
Organization
  • Introduction
    • Unit Testing and Concurrency
    • Tractability
    • Contributions
  • Improvements to JUnit
  • Annotations for Concurrency Invariants
  • Bytecode Rewriting Framework
    • Other Contributions
  • Conclusion
    • Future Work
concurrency invariants
Concurrency Invariants
  • Has to be called in event thread
    • TableModel, TreeModel
  • May not be called in event thread
    • invokeAndWait()
  • Have to acquire readers/writers lock
    • AbstractDocument
    • DrJava’s documents
invariants difficult to determine
Invariants Difficult to Determine
  • May be found in
    • Javadoc comments
    • Only in internal comments
    • Whitepapers
  • Often not documented at all
  • Errors not immediately evident
  • Impossible to check automatically
java annotations
Java Annotations
  • Add invariants as [email protected] static void invokeAndWait( Runnable r) { … }
  • Process class files
    • Find uses of annotations
    • Insert bytecode to check invariants at method beginning
advantages of annotations
Advantages of Annotations
  • Java Language constructs
    • Syntax checked by compiler
  • Easy to apply to part of the program
    • e.g. when compared to a type system change
  • Light-weight
    • Negligible runtime impact if not debugging (slightly bigger class files)
  • Automatic Checking
predicate annotations
Predicate Annotations
  • In annotation definition, specify static boolean Java method
    • Method must be callable from every context completely static and public
  • Data in annotation, method arguments and value of this passed when method invoked
predicate annotation example
Predicate Annotation Example

@PredicateLink(value=Predicates.class,

method="example",

arguments=true)

public @interface ExampleAnnotation {

String foo;

}

Refers to Predicates.example

Definition

predicate annotation example39
Predicate Annotation Example

Usage

public class TestCode {

@ExampleAnnotation(foo="test")

public void test(int param) { … }

}

TestCode t = new TestCode(); t.test(5);

Call

predicate annotation example40
Predicate Annotation Example

public class Predicates {

public static boolean example(

Object this0,

int param,

String foo) {

return (foo.length()<param);

}

predicate annotation example41
Predicate Annotation Example

@PredicateLink(value=Predicates.class,

method="example",

arguments=true)

public @interface ExampleAnnotation {

String foo;

}

public class TestCode {

@ExampleAnnotation(foo="test")

public void test(int param){…}

}

TestCode t = new TestCode();

t.test(5);

public class Predicates {

public static boolean example(

Object this0,

intparam,

String foo) {

return (foo.length()<param); // this0==t, param==5, foo=="test"

}

invariant annotation library
Invariant Annotation Library
  • @OnlyEventThread, @NotEventThread
  • @OnlyThreadWithName
  • @NotNullArgument
  • @DistinctArguments, @SameArguments
  • @OnlySynchronizedThis, @NotSynchronizedThis
  • @OnlySynchronizedArgument, @NotSynchronizedArgument
  • etc. (ca. 80 annotations)
problem multiple annotations
Problem: Multiple Annotations
  • Java does not allow the same annotation class multiple times

@OnlyThreadWithName("foo")

@OnlyThreadWithName("bar") // error

void testMethod() { … }

  • Conjunctions, disjunctions and negations?
annotation subclasses
Annotation Subclasses?
  • Let annotation extend a supertype?

public @interface Invariant { }

public @interface OnlyThreadWithName

extends Invariant { String name(); }

public @interface And extends Invariant {

Invariant[] terms();

}

  • Subtyping not allowed for annotations
work around
Work-Around
  • Different meta-annotation, @Combine

@Combine(Combine.Mode.AND)

public @interface SeveralNames {

OnlyThreadWithName[] value();

}

@SeveralNames({@OnlyThreadWithName("foo"),

@OnlyThreadWithName("bar")})

void testMethod() { … }

combine annotations
Combine Annotations
  • May only contain invariant annotations
    • Predicate annotations
    • Combine annotations
    • Arrays of the above
  • Predicate method automatically generated
    • Calls member predicate methods
    • Accumulates using AND, OR or NOT
      • NOT first negates, then uses AND
      • Default mode is OR
      • De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)
invariants as class annotation
Invariants As Class Annotation
  • Short-hand for annotating all methods
  • What about methods already introduced in a super class, e.g. Object.notify()?
  • What about methods introduced in subclasses?

@Invariant

class A {

void foo() { … }

void bar() { … }

}

class A {

@Invariant

void foo() { … }

@Invariant

void bar() { … }

}

invariants as class annotation48
Invariants As Class Annotation
  • Apply invariants on a class only to methods introduced in the class or subclasses

class A {

public void foo() { … }

}

@Invariant // only applies to bar()

class B extends A {

public void bar() { … }

}

invariant inheritance
Invariant Inheritance
  • Invariants on a method
    • Apply to the method and all overriding methods in subclasses
  • Invariants on a class
    • Apply to all methods introduced in that class or subclasses

 Methods can have invariants defined elsewhere

  • All annotations describe requirements for the client (and, due to subclassing, for subclasses)
    • Allows frameworks to describe requirements
    • Description “thread-safe” is often wrong
invariant subtyping
Invariant Subtyping
  • To maintain substitutability, subclasses may not strengthen invariants
  • Invariants can be modeled as special input parameter
    • Tuple of invariants (“record” in λ calculus [Pierce])
    • Subtyping rules for records declare the “wider” record as subtype
    • In function types, parameter types are contravariant

I0 = {}, I1 = {inv1}, I2 = {inv1,inv2}, I2 <: I1 <: I0

F0 = I0→·, F1 = I1→·, F2 = I2→·, F0 <: F1 <: F2

invariant subtyping51
Invariant Subtyping
  • Analyze methods with invariants as parameter
  • Invariants subtyping: A <@ B <@ C

IA = {}, IB = {inv1}, IC = {inv1,inv2}; IC <: IB <: IAFA = IA→·, FB = IB→·, FC = IC→·; FA <: FB <: FC

  • Java subtyping: C <: B <: A

class A {

void f()

{ … };

}

class B

extends A {

@Inv1

void f()

{ … };

}

class C

extends B {

@Inv2

void f()

{ … };

}

detection of subtyping problems
Detection of Subtyping Problems
  • If Java subtyping and invariant subtyping disagree (A <: B but B <@ A)
    • Substitutability not maintained
    • Statically emit warning
  • Detect if client subclasses do not use framework classes as prescribed
    • Safer multithreaded frameworks
java api annotations
Java API Annotations
  • Started to annotate methods in Java API
    • 30 whole classes, 44 individual methods
  • Community project at http://community.concutest.org/
    • Anyone can suggest annotations
    • Vote on suggested annotations
    • Browse by class or annotation type
  • Annotations can be extracted into XML
    • Share annotations
    • Add checks without needing source code
testing invariant checker
Testing Invariant Checker
  • Annotated two DrJava versions
    • 3/26/2004
    • 9/2/2006
  • Ran test suite, logged invariant violations
    • 2004: 18.83% failed
    • 2006: 11.03% failed
  • 2006 version easier to annotate
    • Better documentation of invariants

details

organization55
Organization
  • Introduction
    • Unit Testing and Concurrency
    • Tractability
    • Contributions
  • Improvements to JUnit
  • Annotations for Concurrency Invariants
  • Bytecode Rewriting Framework
    • Other Contributions
  • Conclusion
    • Future Work
bytecode rewriting
Bytecode Rewriting
  • Except for JUnit, all tools use bytecode rewriting
    • Class files easier to parse than source
    • Works for classes without source (Java API)
    • Can perform changes on-the-fly
  • Adding code to detect properties
    • “instrumentation”
  • Offline and on-the-fly
    • Offline: Class file  class file tool
    • On-the-fly: Custom class loader
local vs global
Local vs. Global
  • Changes can be done by
    • modifying one method (local)
    • all code that calls the method (global)
  • Local instrumentation usually better
    • Fewer changes, less bytecode
    • Harder to make a partial instrumentation if global
  • Not all instrumentations can be done locally
    • If method is native, no class file exists
other contributions
Other Contributions
  • Recording schedules
    • thread ID/type
    • thread ID/type/object ID/class/method/PC
  • Deadlock detector
    • thread ID/type/object ID recording required
    • Creates wait graph for each thread and lock
    • Cycle in graph implies deadlock
random sleeps yields
Random Sleeps/Yields
  • Randomly insert sleeps or yield before or after critical pointsExample: If a notify() is delayed, a wait() may time out.
  • Can detect a number of sample problems
  • Have not studied probabilities or durations for sleeps/yields
    • One inserted delay may negatively impact a second inserted delayExample: If both notify() and wait() are delayed.
organization60
Organization
  • Introduction
    • Unit Testing and Concurrency
    • Tractability
    • Contributions
  • Improvements to JUnit
  • Annotations for Concurrency Invariants
  • Bytecode Rewriting Framework
    • Other Contributions
  • Conclusion
    • Future Work
conclusion
Conclusion
  • Improved JUnit now detects problems in other threads
    • Only in chosen schedule
    • Needs schedule-based execution
  • Annotations ease documentation and checking of concurrency invariants
    • Open-source library of Java API invariants
  • Support programs for schedule-based execution
future work
Future Work
  • Schedule-Based Execution
    • Replay given schedule
    • Generate possible schedules
    • Dynamic race detection
    • Probabilities/durations for random yields/sleeps
  • Extend annotations to Floyd-Hoare logic
    • Preconditions, postconditions
    • Representation invariants
many thanks to
Many Thanks To…
  • My advisor
    • Corky Cartwright
  • My committee members
    • Walid Taha
    • Bill Scherer
  • My friends
    • JavaPLT, CS and Rice
  • NFS and Texas ATP
    • For partially providing funding
extra number of schedules
Extra: Number of Schedules

Product of s-combinations

For thread 1: choose s out of ts time slices

For thread 2: choose s out of ts-s time slices

For thread t-1: choose s out of 2s time slices

For thread t-1: choose s out of s time slices

Writing s-combinations using factorial

Cancel out terms in denominator and next numerator

Left with (ts)! in numerator and t numerators with s!

back

extra limitations
Extra: Limitations
  • May still miss uncaught exceptions
    • Specify absolute parent thread group, not relative (rare)
      • Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread
    • Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)
      • Koders.com: 32 method definitions for uncaughtException method

back

generic annotations
Generic Annotations?
  • Write @And as generic annotation?

public @interface And<T> {

T[] terms();

}

public @interface OnlyThreadWithName {

String name();

}

  • Generics not allowed in annotations
extra drjava statistics
Extra: DrJava Statistics

Unit tests

passed

failed

not run

Invariants

met

failed

% failed

KLOC

“event thread”

2004

736

610

36

90

5116

4161

965

18.83%

107

1

2006

881

881

0

0

34412

30616

3796

11.03

129

99

back

ad