cs294 32 dynamic data race detection l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
CS294-32: Dynamic Data Race Detection PowerPoint Presentation
Download Presentation
CS294-32: Dynamic Data Race Detection

Loading in 2 Seconds...

play fullscreen
1 / 101

CS294-32: Dynamic Data Race Detection - PowerPoint PPT Presentation


  • 120 Views
  • Uploaded on

CS294-32: Dynamic Data Race Detection. Koushik Sen UC Berkeley. Race Conditions. class Ref { int i; void inc() { int t = i + 1; i = t; } } . Courtesy Cormac Flanagan. Race Conditions. class Ref { int i; void inc() { int t = i + 1; i = t; } }

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'CS294-32: Dynamic Data Race Detection' - gallia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
race conditions
Race Conditions

class Ref {

int i;

void inc() {

int t = i + 1;

i = t;

}

}

Courtesy Cormac Flanagan

race conditions3
Race Conditions

class Ref {

int i;

void inc() {

int t = i + 1;

i = t;

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

  • A race condition occurs if
  • two threads access a shared variable at the same time without synchronization
  • at least one of those accesses is a write
race conditions4
Race Conditions

t1

class Ref {

int i;

void inc() {

int t = i + 1;

i = t;

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

t2

RD(i)

RD(i)

WR(i)

WR(i)

lock based synchronization
Lock-Based Synchronization

class Ref {

int i; // guarded by this

void inc() {

synchronized (this) {

int t = i + 1;

i = t;

}

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

  • Field guarded by a lock
  • Lock acquired before accessing field
  • Ensures race freedom
dynamic race detection
Dynamic Race Detection
  • Happens Before [Dinning and Schonberg 1991]
  • Lockset:
    • Eraser [Savage et al. 1997]
    • Precise Lockset [Choi et al. 2002]
  • Hybrid [O'Callahan and Choi 2003]
dynamic race detection7
Dynamic Race Detection
  • Advantages
    • Precise knowledge of the execution
      • No False positive [Happens Before]
      • Unless you try to predict data races [Lockset]
  • Disadvantages
    • Produce false negatives
      • because they only consider a subset of possible program executions.
what we are going to analyze
What we are going to analyze?
  • A trace representing an actual execution of a program
  • Trace is sequence of events:
    • MEM(m,a,t): thread t accessed memory local m, where the access a 2 {RD,WR}
      • m can be o.f, C.f, a[i]
    • ACQ(l,t): thread t acquires lock l
      • Ignore re-acquire of locks. l can be o.
    • REL(l,t): thread t releases lock l
    • SND(g,t): thread t sends message g
    • RCV(g,t): thread t receive message g
      • If t1 calls t2.start(), then generate SND(g,t1) and RCV(g,t2)
      • If t1 calls t2.join(), then generate SND(g,t2) and RCV(g,t1)
how to generate a trace
How to generate a trace?

class Ref {

int i; // guarded by this

void inc() {

print(“ACQ(“+id(this)+”,”+thisThread+”)”);

synchronized (this) {

print(“MEM(“+id(this)+”.i,RD,”+thisThread+”)”);

int t = i + 1;

print(“MEM(“+id(this)+”.i,WR,”+thisThread+”)”);

i = t;

}

print(“REL(“+id(this)+”,”+thisThread+”)”);

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

Instrument a Program

class Ref {

int i; // guarded by this

void inc() {

synchronized (this) {

int t = i + 1;

i = t;

}

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

sample trace
Sample Trace

class Ref {

int i; // guarded by this

void inc() {

print(“ACQ(“+id(this)+”,”+thisThread+”)”);

synchronized (this) {

print(“MEM(“+id(this)+”.i,RD,”+thisThread+”)”);

print(“MEM(“+id(this)+”.i,WR,”+thisThread+”)”);

i = i + 1;

}

print(“REL(“+id(this)+”,”+thisThread+”)”);

}

}

Ref x = new Ref(0);

parallel {

x.inc(); // two calls happen

x.inc(); // in parallel

}

assert x.i == 2;

Sample Trace

ACQ(4365,t1);

MEM(4365.i,RD,t1)

MEM(4365.i,WR,t1)

REL(4365,t1);

ACQ(4365,t2);

MEM(4365.i,RD,t2)

MEM(4365.i,WR,t2)

REL(4365,t2);

compute locks held by a thread
Compute Locks Held by a Thread

L(t) = locks held by thread t. How do we compute L(t)?

Locks Held

Sample Trace

L(t1)={}, L(t2)={}

L(t1)={4365}, L(t2)={}

L(t1)={4365}, L(t2)={}

L(t1)={4365}, L(t2)={}

L(t1)={}, L(t2)={}

L(t1)={}, L(t2)={4365}

L(t1)={}, L(t2)={4365}

L(t1)={}, L(t2)={4365}

L(t1)={}, L(t2)={}

ACQ(4365,t1);

MEM(4365.i,RD,t1)

MEM(4365.i,WR,t1)

REL(4365,t1);

ACQ(4365,t2);

MEM(4365.i,RD,t2)

MEM(4365.i,WR,t2)

REL(4365,t2);

let us now analyze a trace
Let us now analyze a trace
  • Instrument Program
  • Run Program => A Trace File
  • Analyze Trace File
happens before relation
Happens-before relation
  • [Dinning and Schonberg 1991]
  • Idea: Infer a happens-before relation Á between events in a trace
  • We say e1Á e2
    • If e1 and e2 are events from the same thread and e1 appears before e2 in the trace
    • If e1 = SND(g,t) and e2 = RCV(g,t’)
    • If there is a e’ such that e1Á e’ and e’ Á e2
  • REL(l,t) and ACQ(g,t’) generates SND(g,t) and RCV(g,t’)
  • We say e1 and e2 are in race, if
    • e1 and e2 are not related by Á,
    • e1 and e2 are from different threads
    • e1 and e2 access the same memory location and one of the accesses is a write
happens before example 1
Happens-before: example 1

Thread 1

Thread 2

x := x + 1

ACQ(mutex)

v := v + 1

REL(mutex)

Any two accesses of shared variables are in the relation happens-before

ACQ(mutex)

v := v + 1

REL(mutex)

x := x + 1

happens before example 2
Happens-before: example 2

Thread 1

Thread 2

ACQ(mutex)

v := v + 1

REL(mutex)

x := x + 1

Therefore, only this second execution reveals the existing datarace!!

x := x + 1

ACQ(mutex)

v := v + 1

REL(mutex)

eraser lockset
Eraser Lockset
  • Savage,Burrows,Nelson,Sobalvarro,Anderson
  • Assume a database D storing tuples

(m,L)

where:

    • m is a memory location
    • L is a set of locks that protectm
  • Initially D contains a tuple (m,U) for each memory location m, where U is the universal set
how it works
How it works?
  • For an event MEM(m,a,t) generate the tuple (m,L(t))
  • Let (m, L’) be the tuple present in D
    • Report race over memory location m if L(t) Å L’ = empty set
  • Replace (m, L’) by (m, L(t) Å L’) in D
eraser example 1

L(t2) = {mutex}

(v,{mutex}) 2D

L(t1)={mutex}

(v,{mutex}) 2D

Eraser: Example 1

Thread 1

Thread 2

ACQ(mutex)

v := v + 1

REL(mutex)

ACQ(mutex)

v := v + 1

REL(mutex)

eraser example 2

L(t2) = {mutex2}

(v,{}) 2D

L(t1) = {mutex1}

(v,{mutex1}) 2D

Eraser: Example 2

Thread 1

Thread 2

ACQ(mutex1)

v := v + 1

REL(mutex1)

ACQ(mutex2)

v := v + 1

REL(mutex2)

Warning!!

lockset
Lockset

any threadr/w

Shared-read/write

Track lockset

Shared-exclusive

Track lockset

race condition!

extending lockset thread local data
Extending Lockset (Thread Local Data)

first threadr/w

Thread

Local

second

threadr/w

any threadr/w

Shared-read/write

Track lockset

Shared-exclusive

Track lockset

race condition!

extending lockset read shared data
Extending Lockset (Read Shared Data)

first threadr/w

Thread

Local

second

threadread

second

threadwrite

any threadr/w

Shared-read/write

Track lockset

Shared-exclusive

Track lockset

Read

Shared

any threadread

any threadwrite

race condition!

eraser problem

T1(L1,L2)

v

  • false alarm

T2(L2,L3)

T3(L1,L3)

Eraser: Problem

Thread 1

Thread 2

Thread 3

ACQ(L2,L3)

ACQ(L3,L1)

ACQ(L1,L2)

v := v + 1

v := v + 1

v := v + 1

REL(L3,L2)

REL(L1,L3)

REL(L2,L1)

precise lockset
Precise Lockset
  • Choi, Lee, Loginov, O'Callahan, Sarkar, Sridharan
  • Assume a database D storing tuples

(m,t,L,a)

where:

    • m is a memory location
    • t is a thread accessing m
    • L is a set of locks held by t while accessingm
    • a is the type of access (read or write)
  • Initially D is empty
how it works25
How it works?
  • For an event MEM(m,a,t) generate the tuple (m,a,L(t),t)
  • If there is a tuple (m’,a’,L’,t’) in D such that
    • m = m’,
    • (a = WR) Ç (a’=WR)
    • L(t) Å L’ = empty set
    • t  t’
    • Report race over memory location m
  • Add (m,a,L(t),t) in D
optimizations
Optimizations
  • Stop adding tuples on m once a race on m is detected
  • Do not add (m,a,L,t) to D if (m,a,L’,t) is already in D and L’ µ L
  • Many more …
precise lockset27
Precise Lockset

Thread 1

Thread 2

x := x + 1

ACQ(mutex)

ACQ(mutex)

v := v + 1

v := v + 1

REL(mutex)

REL(mutex)

x := x + 1

D

(x, RD,{},t1)

(v,RD,{mutex},t2)

(x,WR,{},t1)

(v,WR,{mutex},t2)

(v,RD,{mutex},t1)

Conflict

detected!

(x,RD,{},t2)

(v,WR,{mutex},t1)

(x,WR,{},t2)

precise lockset28
Precise Lockset

Thread 1

Thread 2

Thread 3

ACQ(m2,m3)

ACQ(m3,m1)

ACQ(m1,m2)

v := v + 1

v := v + 1

v := v + 1

REL(m3,m2)

REL(m1,m3)

REL(m2,m1)

D

(v,RD,{m1,m2},t1)

(v,RD,{m2,m3},t2)

(v,RD,{m1,m3},t3)

(v,WR,{m1,m2},t1)

(v,WR,{m2,m3},t2)

(v,WR,{m1,m3},t3)

No conflicts detected!

precise lockset not so precise
Precise Lockset: Not so Precise

Thread 1

Thread 2

x := x + 1

t2.start()

Precise Lockset gives Warning: But no warning with Happens-Before

x := x + 1

hybrid dynamic data race detection
Hybrid Dynamic Data Race Detection
  • Relax Happens Before
    • No happens before relation between REL(l,t) and a subsequent ACQ(l,t)
  • Maintain Precise Lockset along with Relaxed Happens-Before
hybrid
Hybrid
  • O'Callahan and Choi
  • Assume a database D storing tuples

(m,t,L,a,e)

where:

    • m is a memory location
    • t is a thread accessing m
    • L is a set of locks held by t while accessingm
    • a is the type of access (read or write)
    • e is the event associated with the access
  • Initially D is empty
how it works32
How it works?
  • For an event e = MEM(m,a,t) generate the tuple (m,a,L(t),t,e)
  • If there is a tuple (m’,a’,L’,t’,e’) in D such that
    • m = m’,
    • (a = WR) Ç (a’=WR)
    • L(t) Å L’ = empty set
    • t  t’
    • e and e’ are not related by the happens before relation, i.e., :(e Á e’) Æ:(e’ Á e)
    • Report race over memory location m
  • Add (m,a,L(t),t,e) in D
hybrid dynamic data race detection33
Hybrid Dynamic Data Race Detection

Thread 1

Thread 2

x := x + 1

Precise Lockset detects a data race, but e2 Á e3. Therefore, hybrid technique gives no warning

t2.start()

x := x + 1

D

(x,RD,{},t1,e1)

(x,RD,{},t2,e3)

(x,WR,{},t1,e2)

(x,WR,{},t2,e4)

distributed computation

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Distributed Computation
  • A set of processes: {p1,p2,…,pn}
  • No shared memory
    • Communication through messages
  • Each process executes a sequence of events
    • send(m) : sends a message with content m
    • receive(m): receives a message with content m
    • Internal event: changes local state of a process
  • ith event from process pj is denoted by eij
distributed computation as a partial order

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Distributed Computation as a Partial Order
  • Distributed Computation defines a partial order on the events
    • e ! e’
      • e and e’ are events from the same process and e executes before e’
      • e is the send of a message and e’ is the receive of the same message
      • there is a e’’ such that e ! e’’ and e’’ ! e’
distributed computation as a partial order36

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Distributed Computation as a Partial Order
  • Problem: An external process or observer wants to infer the partial order or the computation for debugging
    • No global clock
    • At each event a process can send a message to the observer to inform about the event
    • Message delay is unbounded

Observer

can we infer the partial order
Can we infer the partial order?
  • From the observation:
  • Can we associate a suitable value with every event such that
    • V(e) < V(e’) , e ! e’
  • We need the notion of clock (logical)

e12 e13 e11 e21 e23 e43 e33 e31 e32 e22

lamport s logical time
Lamport’s Logical Time
  • All processes use a counter (clock) with initial value of zero
  • The counter is incremented by and assigned to each event, as its timestamp
  • A send (message) event carries its timestamp
  • For a receive (message) event the counter is updated by
    • Max(receiver-counter, message-timestamp) + 1
  • Send the counter value along with an event to the observer
example

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Example

1

2

3

4

5

1

6

2

1

3

example40

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Example
  • Problem with Lamport’s logical clock:
    • e ! e’ ) C(e) < C(e’)
    • C(e) < C(e’) ) e ! e’

X

1

2

3

4

5

1

6

2

1

3

example41

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Example
  • Problem with Lamport’s logical clock:
    • e ! e’ ) C(e) < C(e’)
    • C(e) < C(e’) ) e ! e’

X

1

2

3

4

5

1

6

2

1

3

vector clock
Vector Clock
  • Vector Clock: Process ! Nat
  • V: P ! N
  • Associate a vector clock Vp with every process p
  • Update vector clock with every event as follows:
    • Internal event at p_i:
      • Vp(p) := Vp(p) + 1
    • Send Message from p:
      • Vp(p) := Vp(p) + 1
      • Send Vp with message
    • Receive message m at p:
      • Vp(i) := max(Vp(i),Vm(i)) for all i 2 P, where Vm is the vector clock sent with the message m
      • Vp(p) := Vp(p) + 1
example43

p3

e13

e23

e33

e43

m1

m4

m2

e22

e32

e12

p2

m3

p1

e11

e21

e31

Physical Time

Example

V = (a,b,c) means V(p1)=a, V(p2)=b, and V(p3)=c

(0,0,1)

(0,1,2)

(2,1,3)

(2,1,4)

(2,2,4)

(0,1,0)

(2,3,4)

(1,0,0)

(2,0,0)

(3,0,0)

intuitive meaning of a vector clock
Intuitive Meaning of a Vector Clock
  • If Vp = (a,b,c) after some event then
    • p is affected by the ath event from p1
    • p is affected by the bth event from p2
    • p is affected by the cth event from p3
comparing vector clocks
Comparing Vector Clocks
  • V · V’ iff for all p 2 P, V(p) · V’(p)
  • V = V’ iff for all p 2 P, V(p) = V’(p)
  • V < V’ iff V · V’ and V  V’
  • Theorem: Ve < Ve’ iff e ! e’
  • Send an event along with its vector clock to the observer
definition of data race
Definition of Data Race
  • Traditional Definition (Netzer and Miller 1992)

x=1

if (x==1) …

definition of data race47
Definition of Data Race
  • Traditional Definition (Netzer and Miller 1992)

x=1

X

send(m)

receive(m)

if (x==1) …

operational definition of data race
Operational Definition of Data Race
  • We say that the execution of two statements are in race if they could be executed by different threads temporally next to each other and both access the same memory location and at least one of the accesses is a write

x=1

x=1

X

send(m)

receive(m)

if (x==1) …

if (x==1) …

Temporally next

to each other

race directed random testing racefuzzer
Race Directed Random Testing: RACEFUZZER
  • RaceFuzzer: Race directed random testing
  • STEP1: Use an existing technique to find set of pairs of state transitions that could potentially race
    • We use hybrid dynamic race detection
    • Static race detection can also be used
    • Transitions are approximated using program statements
race directed random testing racefuzzer50
Race Directed Random Testing: RACEFUZZER
  • RaceFuzzer: Race directed random testing
  • STEP1: Use an existing technique to find set of pairs of state transitions that could potentially race
    • We use hybrid dynamic race detection
    • Static race detection can also be used
    • Transitions are approximated using program statements
  • STEP2: Bias a random scheduler so that two transitions under race can be executed temporally next to each other
racefuzzer using an example
RACEFUZZER using an example

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Run ERASER: Statement pair (s5,s6) are in race

racefuzzer using an example52
RACEFUZZER using an example

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Run ERASER: Statement pair (s5,s6) are in race

racefuzzer using an example53
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Goal: Create a trace exhibiting the race

racefuzzer using an example54
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Example Trace:

s1: g1();

s2: g2();

s3: g3();

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: o1.f = 1;

s6: if (o1.f==1)

s7: ERROR;

s4: g4();

s5: o2.f = 1;

Racing Statements

Temporally Adjacent

Goal: Create a trace exhibiting the race

racefuzzer using an example55
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

racefuzzer using an example56
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

racefuzzer using an example57
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1();

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

racefuzzer using an example58
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

racefuzzer using an example59
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

racefuzzer using an example60
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s6: if (o1.f==1)

racefuzzer using an example61
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s6: if (o1.f==1)

racefuzzer using an example62
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s6: if (o1.f==1)

Postponed = { }

racefuzzer using an example63
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s6: if (o1.f==1)

Do not postpone

if there is a deadlock

Postponed = {}

s6: if (o1.f==1)

racefuzzer using an example64
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

racefuzzer using an example65
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example66
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example67
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example68
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example69
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example70
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example71
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s5: o2.f = 1;

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example72
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s5: o2.f = 1;

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example73
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s5: o2.f = 1;

Race?

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example74
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s5: o2.f = 1;

Race?

NO

o1.f ≠ o2.f

Postponed = {s6: if (o1.f==1) }

racefuzzer using an example75
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s5: o2.f = 1;

Postponed = {s6: if (o1.f==1), }

s5: o2.f = 1;

racefuzzer using an example76
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

Postponed = {s6: if (o1.f==1), s5: o2.f = 1;}

racefuzzer using an example77
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

Postponed = {s6: if (o1.f==1), s5: o2.f = 1;}

racefuzzer using an example78
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

Postponed = {s6: if (o1.f==1), s5: o2.f = 1;}

racefuzzer using an example79
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

Postponed = {s6: if (o1.f==1), s5: o2.f = 1;}

racefuzzer using an example80
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

Race?

YES

o1.f = o1.f

Postponed = {s6: if (o1.f==1), s5: o2.f = 1;}

racefuzzer using an example81
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s6: if (o1.f==1) s5: o1.f = 1;

Postponed = {s5: o2.f = 1;}

racefuzzer using an example82
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

s6: if (o1.f==1)

Postponed = {s5: o2.f = 1;}

racefuzzer using an example83
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

s6: if (o1.f==1)

Racing Statements

Temporally Adjacent

Postponed = {s5: o2.f = 1;}

racefuzzer using an example84
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

s6: if (o1.f==1)

s7: ERROR;

Racing Statements

Temporally Adjacent

Postponed = {s5: o2.f = 1;}

racefuzzer using an example85
RACEFUZZER using an example

(s5,s6) in race

Thread1

foo(o1);

sync foo(C x) {

s1: g1()

s2: g2();

s3: g3();

s4: g4();

s5: x.f = 1;

}

Thread2

bar(o1);

bar(C y) {

s6: if (y.f==1)

s7: ERROR;

}

Thread3

foo(o2);

Execution:

s1: g1();

s1: g1();

s2: g2();

s2: g2();

s3: g3();

s3: g3();

s4: g4();

s4: g4();

s5: o1.f = 1;

s6: if (o1.f==1)

s7: ERROR;

s5: o2.f = 1;

Racing Statements

Temporally Adjacent

Postponed = { }

another example
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

another example87
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

Race

Racing Pair: (8,10)

another example88
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

Racing Pair: (8,10) Postponed Set = {Thread2}

another example89
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

another example90
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

another example91
Another Example

Thread1{

1: lock(L);

2: f1();

3: f2();

4: f3();

5: f4();

6: f5();

7: unlock(L);

8: if (x==0)

9: ERROR;

}

Thread2{

10: x = 1;

11: lock(L);

12: f6();

13: unlock(L);

}

Hit error with 0.5 probability

implementation
Implementation
  • RaceFuzzer: Part of CalFuzzer tool suite
  • Instrument using SOOT compiler framework
  • Instrumentations are used to “hijack” the scheduler
    • Implement a custom scheduler
    • Run one thread at a time
    • Use semaphores to control threads
  • Deadlock detector
    • Because we cannot instrument native method calls

lock(L1);

X=1;

unlock(L1);

lock(L2);

Y=2;

unlock(L2);

implementation93
Implementation
  • RaceFuzzer: Part of CalFuzzer tool suite
  • Instrument using SOOT compiler framework
  • Instrumentations are used to “hijack” the scheduler
    • Implement a custom scheduler
    • Run one thread at a time
    • Use semaphores to control threads
  • Deadlock detector
    • Because we cannot instrument native method calls

ins_lock(L1);

lock(L1);

ins_write(&X);

X=1;

unlock(L1);

ins_unlock(L1);

ins_lock(L1);

lock(L2);

Y=2;

unlock(L2);

ins_unlock(L1);

Custom

Scheduler

racefuzzer useful features
RACEFUZZER: Useful Features
  • Classify real races from false alarms
  • Inexpensive replay of a concurrent execution exhibiting a real race
  • Separate some harmful races from benign races
  • No false warning
  • Very efficient
    • We instrument at most two memory access statements and all synchronization statements
  • Embarrassingly parallel
racefuzzer limitations
RACEFUZZER: Limitations
  • Not complete: can miss a real race
    • Can only detect races that happen on the given test suite on some schedule
  • May not be able to separate all real races from false warnings
    • Being random in nature
  • May not be able to separate harmful races from benign races
    • If a harmful race does not cause in a program crash
  • Each test run is sequential
summary
Summary
  • Claim: testing (a.k.a verification in industry) is the most practical way to find software bugs
    • We need to make software testing systematic and rigorous
  • Random testing works amazingly well in practice
    • Randomizing a scheduler is more effective than randomizing inputs
    • We need to make random testing smarter and closer to verification
      • Bias random testing
    • Prioritize random testing
      • Find interesting preemption points in the programs
      • Randomly preempt threads at these interesting points
java lang stringbuffer
java.lang.StringBuffer

/**

... used by the compiler to implement the binary string concatenation operator ...

String buffers are safe for use by multiple threads. The methods are synchronized so that all the operations on any particular instance behave as if they occur in some serial order that is consistent with the order of the method calls made by each of the individual threads involved.

*/

/*# atomic */ public class StringBuffer { ... }

java lang stringbuffer99

sb.length() acquires lock on sb, gets length, and releases lock

other threads can change sb

use of stale len may yield StringIndexOutOfBoundsException

inside getChars(...)

java.lang.StringBuffer

public class StringBuffer {

private int count;

public synchronized int length() { return count; }

public synchronized void getChars(...) { ... }

/*# atomic */

public synchronized void append(StringBuffer sb){

int len = sb.length();

...

...

...

sb.getChars(...,len,...);

...

}

}

java lang stringbuffer100
java.lang.StringBuffer

public class StringBuffer {

private int count;

public synchronized int length() { return count; }

public synchronized void getChars(...) { ... }

/*# atomic */

public synchronized void append(StringBuffer sb){

int len = sb.length();

...

...

...

sb.getChars(...,len,...);

...

}

}

StringBuffer.append is not atomic:

Start:

at StringBuffer.append(StringBuffer.java:701)

at Thread1.run(Example.java:17)

Commit: Lock Release

at StringBuffer.length(StringBuffer.java:221)

at StringBuffer.append(StringBuffer.java:702)

at Thread1.run(Example.java:17)

Error: Lock Acquire

at StringBuffer.getChars(StringBuffer.java:245)

at StringBuffer.append(StringBuffer.java:710)

at Thread1.run(Example.java:17)

related work
Related work
  • Stoller et al. and Edelstein et al. [ConTest]
    • Inserts yield() and sleep() randomly in Java code
  • Parallel randomized depth-first search by Dwyer et al.
    • Modifies search strategy in Java Pathfinder by Visser et al.
  • Iterative context bounding (Musuvathi and Qadeer)
    • Systematic testing with bounded context switches
  • Satish Narayanasamy, Zhenghao Wang, Jordan Tigani, Andrew Edwards and Brad Calder “Automatically Classifying Benign and Harmful Data Races Using Replay Analysis”, PLDI 07