a framework for testing concurrent programs l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A Framework for Testing Concurrent Programs PowerPoint Presentation
Download Presentation
A Framework for Testing Concurrent Programs

Loading in 2 Seconds...

play fullscreen
1 / 63

A Framework for Testing Concurrent Programs - PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on

A Framework for Testing Concurrent Programs. COMP 600 Mathias Ricken Rice University August 27, 2007. Unit Testing. Program. Sub- program. Sub- program. Sub- program. ?. ?. ?. ?. ?. Difficult. Even less difficult. Less difficult. Unit Testing. Unit tests…

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Framework for Testing Concurrent Programs' - bernad


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a framework for testing concurrent programs

A Framework for Testing Concurrent Programs

COMP 600

Mathias Ricken

Rice University

August 27, 2007

unit testing
Unit Testing

Program

Sub-

program

Sub-

program

Sub-

program

?

?

?

?

?

Difficult

Even less difficult

Less difficult

unit testing3
Unit Testing
  • Unit tests…
    • Test a part, not the whole program
    • Occur earlier
    • Automate testing
    • Keep the shared repository clean
    • Serve as documentation
    • Prevent bugs from reoccurring
  • Effective with a single thread of control
foundation of unit testing
Foundation of Unit Testing
  • Unit tests depend on deterministic behavior
  • Known input, expected output…Success  correct behaviorFailure  flawed code
  • Outcome of test is meaningful
problems due to concurrency
Problems Due to Concurrency
  • Thread scheduling is nondeterministic and machine-dependent
    • Code may be executed under different schedules
    • Different schedules may produce different results
  • Known input, expected output…Success  correct behaviorin this schedule, may be flawed in other scheduleFailure  flawed code
  • Success of unit test is meaningless
timeliness of the problem
Timeliness of the Problem
  • Many programs already use concurrency
    • Often hidden, as part of GUI
  • Most speed increases are due to multiple cores on one chip
    • Clock speeds have not increased much
  • Increased use of concurrency in the future
possible solutions
Possible Solutions
  • Programming Language Features
    • Ensuring that bad things cannot happen
    • May restrict programmers
  • Lock-Free Algorithms
    • Ensuring that if bad things happen, it’s ok
    • May limit data structures available
  • Comprehensive Testing
    • Testing if bad things happen in any schedule
    • Does not prevent problems, but does not limit solutions either

details

tractability of comprehensive testing
Tractability of Comprehensive Testing
  • Deciding whether any given program contains an error is undecidable
    • Reduction to the halting problem
  • Does not imply undecidable for all programs
number of schedules
Number of Schedules
  • Test all possible schedules
    • Concurrent unit tests meaningful again
  • Number of schedules (N)
    • t: # of threads, s: # of slices per thread

Exponential in both s and t

details

critical points
Critical Points
  • If program is race-free, we do not have to simulate all thread switches
    • Threads interfere only at “critical points”: lock operations, volatile variables, etc.
    • Code between critical points cannot affect outcome
    • Simulate all possible arrangements of blocks delimited by critical points
critical points example
Critical Points Example

Local Var 1

All accesses protected by lock

lock access unlock

lock access unlock

Thread 1

Shared Var

Lock

Local variables don’t need locking

All accesses protected by lock

All accesses protected by lock

Thread 2

lock access unlock

Local Var 1

fewer schedules
Fewer Schedules
  • Fewer critical points than thread switches
    • Reduces number of schedules
    • Example: Two threads, but no communication only one schedule required
  • Unit tests are small
    • Reduces number of schedules
  • Hopefully comprehensive simulation is tractable
    • If not, heuristics are still better than nothing
parts of the framework
Parts of the Framework
  • Improvements to JUnit
    • Detect exceptions and failed assertions in threads other than the main thread
  • Annotations for Concurrency Invariants
    • Express complicated requirements about locks and threads
  • Tools for Schedule-Based Execution
    • Record, deadlock monitor
    • Random delays, random yields
concurrency invariants
Concurrency Invariants
  • Has to be called in event thread
    • TableModel, TreeModel
  • May not be called in event thread
    • invokeAndWait()
  • Have to acquire readers/writers lock
    • AbstractDocument
    • DrJava’s documents
invariants difficult to determine
Invariants Difficult to Determine
  • May be found in
    • Javadoc comments
    • Only in internal comments
    • Whitepapers
  • Often not documented at all
  • Errors not immediately evident
  • Impossible to check automatically
java annotations
Java Annotations
  • Add invariants as annotations@NotEventThreadpublic static void invokeAndWait( Runnable r) { … }
  • Process class files
    • Find uses of annotations
    • Insert bytecode to check invariants at method beginning
advantages of annotations
Advantages of Annotations
  • Java Language constructs
    • Syntax checked by compiler
  • Easy to apply to part of the program
    • e.g. when compared to a type system change
  • Light-weight
    • Negligible runtime impact if not debugging (slightly bigger class files)
  • Automated Checking
predicate annotations
Predicate Annotations
  • In annotation definition, specify static boolean Java method
    • Method must be callable from every context completely static and public
  • Data in annotation, method arguments and value of this passed when method invoked
predicate annotation example
Predicate Annotation Example

@PredicateLink(

value=Predicates.class,

method="example",

arguments=true)

public @interface TestAllowed {

boolean allowed;

}

public class TestCode {

@TestAllowed(allowed=true)

public void test(String param){…}

}

TestCode t = new TestCode();

t.test("xxx");

public class Predicates {

public static boolean example(

Object this0,

String param,

booleanallowed) {

return (allowed)? // this0==t

(param.equals("test")): // param=="xxx"

(!param.equals("test")); // allowed==true

}

zoom

provided annotations
Provided Annotations
  • @NotEventThread
  • @NotThreadWithName
  • @NotSynchronizedThis
  • @NotSynchronizedArgument
  • @NotNullArgument
  • @DistinctArguments
  • Inverses, conjunctions, disjunctions
invariant inheritance
Invariant Inheritance
  • Invariants apply to the method and all overriding methods in subclasses

 Methods can have invariants defined elsewhere

  • All annotations describe requirements for the client (and, due to subclassing, for subclasses)
    • Allows frameworks to describe requirements
invariant subtyping
Invariant Subtyping
  • To maintain substitutability, subclasses may not strengthen invariants
  • Invariants can be modeled as special input parameter
    • Tuple of invariants (“record” in λ calculus [Pierce])
    • Subtyping rules for records declare the “wider” record as subtype
    • In function types, parameter types are contravariant
invariant subtyping23
Invariant Subtyping
  • Analyze methods with invariants
  • Invariants subtyping: A <@ B <@ C

IA = {}, IB = {inv1}, IC = {inv1,inv2}; IC <: IB <: IAFA = IA→·, FB = IB→·, FC = IC→·; FA <: FB <: FC

  • Java subtyping: C <: B <: A

class A {

void f()

{ … };

}

class B

extends A {

@Inv1

void f()

{ … };

}

class C

extends B {

@Inv2

void f()

{ … };

}

detection of subtyping problems
Detection of Subtyping Problems
  • If Java subtyping and invariant subtyping disagree (A <: B but B <@ A)
    • Substitutability not maintained
    • Statically emit warning
  • Detect if client subclasses do not use framework classes as prescribed
    • Safer multithreaded frameworks
java api annotations
Java API Annotations
  • Started to annotate methods in Java API
  • Community project at http://community.concutest.org/
    • Browse and suggest annotations
  • Annotations can be extracted into XML
    • Share annotations
    • Add checks without needing source code
testing invariant checker
Testing Invariant Checker
  • Annotated two DrJava versions
    • 3/26/2004
    • 9/2/2006
  • Ran test suite, logged invariant violations
    • 2004: 18.83% failed
    • 2006: 11.03% failed
  • 2006 version easier to annotate
    • Better documentation of invariants

details

conclusion
Conclusion
  • Improved JUnit now detects problems in other threads
  • Annotations ease documentation and checking of concurrency invariants
  • Support programs for schedule-based execution
future work
Future Work
  • Schedule-Based Execution
    • Replay given schedule
    • Generate possible schedules
    • Dynamic race detection
    • Probabilities/durations for random yields/sleeps
  • Extend annotations to Floyd-Hoare logic
    • Preconditions, postconditions
    • Representation invariants
many thanks to
Many Thanks To…
  • My advisor
    • Corky Cartwright
  • My committee members
    • Walid Taha
    • Bill Scherer
  • My friends
    • JavaPLT, CS and Rice
  • NFS and Texas ATP
    • For partially providing funding
possible solutions31
Possible Solutions
  • Programming Language Features
    • Race Freedom
    • Deadlock Freedom, Safe Locking
    • Atomicity, Transactions
    • Usually require changes in type system
    • Fundamental change
      • C++ standards: 1998, 2003, 200x
      • Java major changes: 1997 (1.0), 2002 (1.4), 2004 (1.5)

back

possible solutions32
Possible Solutions
  • Lock-Free Algorithms
    • Work on copy and assume no concurrency is present (or current thread will finish first)
    • If there was interference, threads that don’t finish first redo their work
    • Require some system support (e.g. compare-and-swap), then done in libraries
    • Not all common data structures are practical and efficient to implement as lock-free

back

extra number of schedules
Extra: Number of Schedules

Product of s-combinations

For thread 1: choose s out of ts time slices

For thread 2: choose s out of ts-s time slices

For thread t-1: choose s out of 2s time slices

For thread t-1: choose s out of s time slices

Writing s-combinations using factorial

Cancel out terms in denominator and next numerator

Left with (ts)! in numerator and t numerators with s!

back

improvements to junit
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
sample junit tests
Sample JUnit Tests

publicclass Test extends TestCase {

public void testException() {

thrownew RuntimeException("booh!");

}

public void testAssertion() {

assertEquals(0, 1);

}

}

}

Both tests fail.

Both tests fail.

if (0!=1)

throw new AssertionFailedError();

problematic junit tests

end of test

spawns

Main thread

success!

uncaught!

Child thread

Problematic JUnit Tests

Main thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

problematic junit tests37
Problematic JUnit Tests

Main thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

Uncaught exception, test should fail but does not!

improvements to junit38
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
  • Thread group with exception handler
    • JUnit test runs in a separate thread, not main thread
    • Child threads are created in same thread group
    • When test ends, check if handler was invoked
thread group for junit tests
Thread Group for JUnit Tests

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

invokes

checks

TestGroup’s Uncaught Exception Handler

thread group for junit tests40
Thread Group for JUnit Tests

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

spawns and waits

resumes

Main thread

failure!

check group’s handler

spawns

end of test

Test thread

invokes group’s handler

uncaught!

Child thread

improvements to junit41
Improvements to JUnit
  • Uncaught exceptions and failed assertions
    • Not caught in child threads
  • Thread group with exception handler
    • JUnit test runs in a separate thread, not main thread
    • Child threads are created in same thread group
    • When test ends, check if handler was invoked
  • Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end
child thread outlives parent
Child Thread Outlives Parent

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

spawns and waits

resumes

Main thread

failure!

check group’s handler

spawns

end of test

Test thread

invokes group’s handler

uncaught!

Child thread

child thread outlives parent43
Child Thread Outlives Parent

Test thread

publicclass Test extends TestCase {

public void testException() {

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

}

}

new Thread(new Runnable() {

public void run() {

thrownew RuntimeException("booh!");

}

}).start();

thrownew RuntimeException("booh!");

Child thread

check group’s handler

spawns and waits

resumes

Main thread

success!

spawns

Too late!

Test thread

end of test

uncaught!

invokes group’s handler

Child thread

improvements to junit44
Improvements to JUnit
  • Child threads are not required to terminate
    • A test may pass before an error is reached
  • Detect if any child threads are still alive
    • Declare failure if test thread has not waited
    • Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)
  • Previous schedule is a test failure
    • Should be prevented by using Thread.join()
testing concjunit
Testing ConcJUnit
  • Replacement for junit.jar or as plugin JAR for JUnit 4.2
  • Available as binary and source at http://www.concutest.org/
  • Results from DrJava’s unit tests
    • Child thread for communication with slave VM still alive in test
    • Several reader and writer threads still alive in low level test (calls to join() missing)
limitations
Limitations
  • Improvements only check chosen schedule
    • A different schedule may still fail
    • Requires comprehensive testing to be meaningful
  • May still miss uncaught exceptions
    • Specify absolute parent thread group, not relative Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)

details

extra limitations
Extra: Limitations
  • May still miss uncaught exceptions
    • Specify absolute parent thread group, not relative (rare)
      • Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread
    • Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)
      • Koders.com: 32 method definitions for uncaughtException method

back

generic annotations
Generic Annotations?
  • Write @And as generic annotation?

public @interface And<T> {

T[] terms();

}

public @interface OnlyThreadWithName {

String name();

}

  • Generics not allowed in annotations
extra drjava statistics
Extra: DrJava Statistics

Unit tests

passed

failed

not run

Invariants

met

failed

% failed

KLOC

“event thread”

2004

736

610

36

90

5116

4161

965

18.83%

107

1

2006

881

881

0

0

34412

30616

3796

11.03

129

99

back

predicate annotation example50
Predicate Annotation Example

@PredicateLink(value=Predicates.class,

method="example",

arguments=true)

public @interface TestAllowed {

boolean allowed;

}

Refers to Predicates.example

Definition

back

predicate annotation example51
Predicate Annotation Example

Usage

public class TestCode {

@TestAllowed(allowed=true)

public void test(String param) { … }

}

TestCode t = new TestCode(); t.test("xxx");

Call

back

predicate annotation example52
Predicate Annotation Example

Predicate Method

public class Predicates {

public static boolean example(

Object this0,

String param,

boolean allowed) {

return (allowed)? (param.equals("test")):

(!param.equals("test"));

}

back

problem multiple annotations
Problem: Multiple Annotations
  • Java does not allow the same annotation class multiple times

@NotThreadWithName("foo")

@NotThreadWithName("bar") // error

void testMethod() { … }

  • Conjunctions, disjunctions and negations?
annotation subclasses
Annotation Subclasses?
  • Let annotation extend a supertype?

public @interface Invariant { }

public @interface OnlyThreadWithName

extends Invariant { String name(); }

public @interface And extends Invariant {

Invariant[] terms();

}

  • Subtyping not allowed for annotations
work around
Work-Around
  • Different meta-annotation, @Combine

@Combine(Combine.Mode.AND)

public @interface SeveralNames {

OnlyThreadWithName[] value();

}

@SeveralNames({@NotThreadWithName("foo"),

@NotThreadWithName("bar")})

void testMethod() { … }

combine annotations
Combine Annotations
  • May only contain invariant annotations
    • Predicate annotations
    • Combine annotations
    • Arrays of the above
  • Predicate method automatically generated
    • Calls member predicate methods
    • Performs AND, OR, XOR, NOT or IMPLIES
invariants as class annotation
Invariants As Class Annotation
  • Short-hand for annotating all methods
  • What about methods already introduced in a super class, e.g. Object.notify()?
  • What about methods introduced in subclasses?

@Invariant

class A {

void foo() { … }

void bar() { … }

}

class A {

@Invariant

void foo() { … }

@Invariant

void bar() { … }

}

invariants as class annotation59
Invariants As Class Annotation
  • Apply invariants on a class only to methods introduced in the class or subclasses

class A {

public void foo() { … }

}

@Invariant // only applies to bar()

class B extends A {

public void bar() { … }

}

bytecode rewriting
Bytecode Rewriting
  • Except for JUnit, all tools use bytecode rewriting
    • Class files easier to parse than source
    • Works for classes without source (Java API)
    • Can perform changes on-the-fly
  • Adding code to detect properties
    • “instrumentation”
  • Offline and on-the-fly
    • Offline: Class file  class file tool
    • On-the-fly: Custom class loader
local vs global
Local vs. Global
  • Changes can be done by
    • modifying one method (local)
    • all code that calls the method (global)
  • Local instrumentation usually better
    • Fewer changes, less bytecode
    • Harder to make a partial instrumentation if global
  • Not all instrumentations can be done locally
    • If method is native, no class file exists
other contributions
Other Contributions
  • Recording schedules
    • thread ID/type
    • thread ID/type/object ID/class/method/PC
  • Deadlock detector
    • thread ID/type/object ID recording required
    • Creates wait graph for each thread and lock
    • Cycle in graph implies deadlock
random sleeps yields
Random Sleeps/Yields
  • Randomly insert sleeps or yield before or after critical pointsExample: If a notify() is delayed, a wait() may time out.
  • Can detect a number of sample problems
  • Have not studied probabilities or durations for sleeps/yields
    • One inserted delay may negatively impact a second inserted delayExample: If both notify() and wait() are delayed.