anomaly based bug prediction isolation and validation an automated approach for software debugging n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging PowerPoint Presentation
Download Presentation
Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging

Loading in 2 Seconds...

play fullscreen
1 / 52

Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging - PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on

Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging. Martin Dimitrov Huiyang Zhou. Background: Terminology. Defect : Erroneous piece of code (a bug)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging' - soledad


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
anomaly based bug prediction isolation and validation an automated approach for software debugging

Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging

Martin Dimitrov Huiyang Zhou

background terminology
Background: Terminology
  • Defect: Erroneous piece of code (a bug)
  • Infection: The defect is triggered => the program state differs from what the programmer intended.
  • Failure: An observable error (crash, hang, wrong results) in program behavior.

Terminologies are based on the book “Why Programs Fail” by Andreas Zeller.

University of Central Florida

background from defects to failures
Background: From Defects to Failures

Erroneous code

Variable and input values

Observer sees failure

Figure from the book

“Why Programs Fail” by A. Zeller

Sane state

Infected state

Program execution

University of Central Florida

motivation
Motivation
  • The typical process of software debugging involves:
    • Examine the point of program failure and reason backwards about the possible causes.
    • Create a hypothesis of what could be the root cause.
    • Modify the program to verify the hypothesis.
    • If the failure is still there, the search resumes.
  • Software debugging is tedious and time consuming !
  • In this work we propose an approach to automate the debugging effort and pinpoint the failure root cause.

University of Central Florida

presentation outline
Presentation Outline
  • Motivation
  • Proposed approach
  • Detecting anomalies (step 1)
  • Isolating relevant anomalies (step 2)
  • Validating anomalies (step 3)
  • Experimental methodology
  • Experimental results
  • Conclusions

University of Central Florida

proposed approach
Proposed Approach

Dynamic Instruction Stream

mov ...

cmp ...

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

Current instruction

time

University of Central Florida

proposed approach1
Proposed Approach

Dynamic Instruction Stream

cmp ...

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

Current instruction

jr ...

time

University of Central Florida

proposed approach2
Proposed Approach

Dynamic Instruction Stream

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

Current instruction

movl ...

time

University of Central Florida

proposed approach3
Proposed Approach

Dynamic Instruction Stream

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

Current instruction

inc ...

time

University of Central Florida

proposed approach4
Proposed Approach

Dynamic Instruction Stream

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

Current instruction

cmp ...

time

University of Central Florida

proposed approach5
Proposed Approach

Dynamic Instruction Stream

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

Current instruction

jl ...

time

University of Central Florida

proposed approach6
Proposed Approach

Dynamic Instruction Stream

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

Current instruction

test ...

time

University of Central Florida

proposed approach7
Proposed Approach

Dynamic Instruction Stream

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

Current instruction

jne ...

time

University of Central Florida

proposed approach8
Proposed Approach

Dynamic Instruction Stream

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

Current instruction

mov ...

time

University of Central Florida

proposed approach9
Proposed Approach

Dynamic Instruction Stream

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

Current instruction

call ...

time

University of Central Florida

proposed approach10
Proposed Approach
  • A program failure is observed:
    • Crash
    • Hang
    • Incorrect results, etc.
  • Start the automated debugging process
  • The output of our approach is a ranked list of instructions (the possible root-cause of failure)

Dynamic Instruction Stream

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

call ...

Failure point

mov ...

Failure

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

mov ...

cmp ...

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

Current instruction

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution1
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

cmp ...

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

Current instruction

jr ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution2
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

jge ...

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

Current instruction

movl ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution3
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

mov ...

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

Current instruction

inc ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution4
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

mov ...

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

Current instruction

cmp ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution5
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

lea ...

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

Current instruction

jl ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution6
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

movl ...

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

Current instruction

test ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution7
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

inc ...

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

Current instruction

jne ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution8
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

cmp ...

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

Current instruction

mov ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution9
Proposed ApproachStep 1: Detect anomalies in program execution

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

jl ...

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

Current instruction

call ...

time

University of Central Florida

proposed approach step 1 detect anomalies in program execution10
Proposed ApproachStep 1: Detect anomalies in program execution
  • Each anomaly constitutes a hypothesis for the root cause of program failure.

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

call ...

Failure point

mov ...

Failure

time

University of Central Florida

proposed approach step 2 isolate the relevant anomalies
Proposed ApproachStep 2: Isolate the relevant anomalies

Dynamic Instruction Stream

anomaly

anomaly

anomaly

anomaly

movl ...

  • Create dynamic forward slices from the anomalies to the failure point.
  • Discard anomalies which do not lead to the failure point.

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

call ...

Failure point

mov ...

Failure

time

University of Central Florida

proposed approach step 3 validate the isolated anomalies
Proposed Approach Step 3: Validate the isolated anomalies
  • Automatically “fix” the anomaly and observe if the program still fails.

Dynamic Instruction Stream

anomaly

anomaly

movl ...

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

call ...

Failure point

mov ...

Failure

time

University of Central Florida

proposed approach step 3 validate the isolated anomalies1
Proposed ApproachStep 3: Validate the isolated anomalies

Dynamic Instruction Stream

movl ...

  • If the failure disappears we have a high confidence the root cause have been pinpointed.

inc ...

cmp ...

jr ...

movl ...

inc ...

cmp ...

jl ...

test ...

jne ...

mov ...

call ...

No failure

mov ...

Success

time

University of Central Florida

detecting program anomalies step 1
Detecting Program Anomalies (Step 1)
  • When infected by a software bug the program is likely to misbehave:
    • Out-of-bounds addresses and values
    • Unusual control paths
    • Page faults
    • Redundant computations, etc.
    • Anomaly detection: Infer program specifications from passing runs and turn them into ‘soft’ assertions.
    • Learn program invariants during passing runs

(e.g. variable “i” is always between 0 and 100)

    • Flag violated invariants during the failing run

(e.g. Report anomaly if variable “i” is 101)

University of Central Florida

detecting program anomalies
Detecting Program Anomalies
  • We use several anomaly detectors to monitor a large spectrum of program invariants and catch more bugs.
  • DIDUCE [S. Hangal et al. , ICSE 2002]
    • Instructions tent to produce values/addresses within a certain range (e.g. 0 <= i <= 100). Detect violations of these invariants.
  • AccMon [P. Zhou et al. , MICRO-37 2004]
    • Only a few static instructions access a given memory location (load/store set locality). Signal an anomaly if memory access does not belong to the load/store set.
  • LoopCount
    • Detect abnormal number of loop iterations.

University of Central Florida

detecting program anomalies1
Detecting Program Anomalies

void more_arrays () {

. . .

a_count += STORE_INCR;

/* Copy the old arrays. */

for (indx = 1; indx < old_count; indx++)

arrays[indx] = old_ary[indx];

/* Initialize the new elements. */

for (; indx < v_count; indx++){/* defect*/

arrays[indx] = NULL; /*infection: */

/* Free the old elements. */

if (old_count != 0){

free (old_ary); /*crash*/

}

}

Heap Memory

data

data

data

data

data

size

a_count 

size

data

data

v_count 

data

data

University of Central Florida

detecting program anomalies2
Detecting Program Anomalies

void more_arrays () {

. . .

a_count += STORE_INCR;

/* Copy the old arrays. */

for (indx = 1; indx < old_count; indx++)

arrays[indx] = old_ary[indx];

/* Initialize the new elements. */

for (; indx < v_count; indx++){/* defect*/

arrays[indx] = NULL; /*infection: */

/* Free the old elements. */

if (old_count != 0){

free (old_ary); /*crash*/

}

}

LoopCount: Loop iterates more times than usual.

LoopCount: Loop iterates more times than usual. (false – positive)

AccMon: store instruction is not in store set of this memory location.

AccMon: store instruction is not in store set of this memory location. (false – positive)

DIDUCE: Address of store instruction is out of normal range.

DIDUCE: Address of store instruction is out of normal range. (false – positive)

University of Central Florida

detecting program anomalies architectural support
Detecting Program Anomalies: Architectural Support
  • DIDUCE and AccMon capture invariants using limited size caches structures, as proposed in previous work
  • LoopCount utilizes the existing loop-branch predictor to detect anomalies.
  • Advantages and disadvantages of hardware support:
    • Performance efficiency
    • Portability
    • Efficient ways to change or invalidate dynamic instructions
    • Limited hardware resource may become a concern

University of Central Florida

isolating relevant anomalies step 2
Isolating Relevant Anomalies (Step 2)
  • Anomaly detectors alone are NOT effective for debugging:
    • May signal too many anomalies / false positives
    • Tradeoff between bug coverage and number of false positives
  • Our solution:
    • Allow aggressive anomaly detection for maximum bug coverage
    • Automatically isolate only the relevant anomalies by constructing dynamic forwards slices from the anomaly to the failure point

University of Central Florida

isolating relevant anomalies architectural support
Isolating Relevant Anomalies: Architectural Support
  • Add token(s) to each register and memory word.
  • Detected anomalies set a token associated with the destination memory word or register.
  • Tokens propagate based data dependencies.
  • When the program fails, examine the point of failure for token.

University of Central Florida

isolating relevant anomalies architectural support1
Isolating Relevant Anomalies: Architectural Support

void more_arrays () {

. . .

/* Copy the old arrays. */

for (indx = 1; indx < old_count; indx++)

arrays[indx] = old_ary[indx];

/* Initialize the new elements. */

for (; indx < v_count; indx++){/* defect */

arrays[indx] = NULL; /*infection*/

/* Free the old elements. */

if (old_count != 0){

free (old_ary); /*crash */

}

}

Token

Memory

. . .

Failure

mov %ebx,0xc(%edx)

University of Central Florida

isolating relevant anomalies architectural support2
Isolating Relevant Anomalies: Architectural Support
  • Problem: Many tokens for each memory location/ register
  • Solution:
    • We leverage tagged architectures for information flow tracking.
    • Use only one token (1 bit) (i.e., shared by all anomalies )
    • We leverage delta debugging [A. Zeller, FSE 1999] to isolate the relevant anomalies automatically.

Number of Initial Anomalies

Number of Isolated Anomalies

University of Central Florida

delta debugging
Delta-Debugging

University of Central Florida

validating isolated anomalies step 3
Validating Isolated Anomalies (Step 3)
  • Validate the remaining anomalies by applying a “fix” and observing if the program failure disappears.
  • Our “fix” is to nullify the anomalous instruction (turn it into no-op)
  • If the program succeeds, we have a high confidence we have found the root cause (or at least broken the infection chain)

University of Central Florida

validating isolated anomalies
Validating Isolated Anomalies

void more_arrays () {

. . .

/* Initialize the new elements. */

for (; indx < v_count; indx++){/* defect */

arrays[indx] = NULL; /*infection*/

/* Free the old elements. */

if (old_count != 0){

free (old_ary); /*crash */

}

}

Token

Memory

data

data

0x0

0x0

0x0

data

data

data

0x0

0x0

0x0

size

size

data

Success

data

data

data

  • The “size” information is not corrupted and the program terminates successfully.

University of Central Florida

validating isolated anomalies1
Validating Isolated Anomalies
  • Four possible outcomes of our validation step:

> ... (hang)

> Segmentation Fault

> Hello World !

> Hello Martin 

Failure

Success

Unknown

No crash

Program fails the same way as before.

Program produces expected output.

Program fails in an unexpected manner.

Program does not crash, but produces unexpected results.

  • Rank isolated anomalies based on the outcome:
  • succeed (highest) , no crash, unknown, failure (lowest)
  • In our running example the root-cause is ranked #1.

University of Central Florida

experimental methodology
Experimental Methodology
  • Implemented a working debugging tool using binary instrumentation (PIN).
  • Evaluated applications from BugBench [S. Lu et al., Bugs 2005] and gcc compiler.

University of Central Florida

experimental results
Experimental Results

University of Central Florida

case study gcc
Case Study: GCC

University of Central Florida

gcc defect
GCC Defect

University of Central Florida

gcc fix
GCC Fix

University of Central Florida

limitations
Limitations
  • No failure, no bug detection
    • Un-triggered bugs
    • Bugs are triggered but output is correct
  • Target at bugs in sequential programs

University of Central Florida

conclusions
Conclusions
  • We present a novel automated approach to pinpoint the root causes of software failures:
    • Detect anomalies during program execution.
    • Isolate only the relevant anomalies.
    • Validate isolated anomalies by nullifying execution results
  • Our experimental results demonstrate that we accurately pinpoint the defect even for large programs such as gcc.

University of Central Florida

questions
Questions
  • The tool is available for download at:

http://csl.cs.ucf.edu/debugging

University of Central Florida