Simplify testing using fault injection
Download
1 / 43

Simplify Testing using Fault Injection - PowerPoint PPT Presentation


  • 649 Views
  • Updated On :

Simplify Testing using Fault Injection. Rob Grzywinski [email protected] Motivation. Writing and testing an IO-intensive application Robust, asynchronous event (message) collector / router NIO (early 1.4 days) How to best simulate and test various failure modes?

Related searches for Simplify Testing using Fault Injection

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Simplify Testing using Fault Injection' - Jeffrey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Motivation l.jpg
Motivation

  • Writing and testing an IO-intensive application

    • Robust, asynchronous event (message) collector / router

    • NIO (early 1.4 days)

  • How to best simulate and test various failure modes?

    • NIO buggy as all get-out

    • Networks are inherently flaky and problematic

  • Didn’t want to have to obfuscate or drastically change code in order to test it


Agenda l.jpg
Agenda

  • Quick Introduction to Fault Injection

  • Why Use Fault Injection

  • Fault Injection Examples

  • Beyond Fault Injection

  • Quick Introduction to AOP

  • Fault Injection in Java

    • AspectJ

    • Javassist

  • Wrap Up


Fault injection 101 l.jpg
Fault Injection 101

  • A fault is an abnormal condition or defect which may lead to a failure[10]

  • Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response [9]

    • The goal isn’t to recreate the conditions that produced the fault


Fault injection 101 cont l.jpg
Fault Injection 101 (cont)

There are two primary steps to Fault Injection:

  • To identify the set of faults that can occur within an application, module, class, method, etc.

    • e.g. if the application does not use the network then there’s no point in injecting network faults

    • In practice this isn’t as easy as it sounds

      e.g. 3rd party libs

  • To exercise those faults to evaluate how the application responds

    • Does the application detect the fault, is it isolated and does the application recover from it? [8]


Example l.jpg
Example

byte[]readFile() throws IOException {

...

final InputStream is = new FileInputStream(…);

...

while((offset < bytes.length) &&

(numRead = is.read(bytes, offset, (bytes.length - offset))) >= 0)

offset += numRead;

...

is.close();

return bytes;

}

(from http://javaalmanac.com/egs/java.io/File2ByteArray.html)

What could go wrong with this code?

  • new FileInputStream() can throw FileNotFoundException

  • InputStream.read() can throw IOException and IndexOutOfBoundsException and can return -1 for end of file

  • is.close() can throw IOException

How do we test how the application responds to one of these situations?

Specifically, how do we inject these faults so that we can test how the application responds?


Possible techniques l.jpg
Possible Techniques

  • Force the situations at the OS level

    • Quite hairy to reproduce reliably when they’re needed

  • Refactor the snot out of it

    • Replace the call to InputStream.read() to some local instrumented method

    • Create our own instrumented InputStream subclass possibly using mock objects

      • Inject the subclass via IoC (requires some framework such as PicoContainer or Spring)

  • Just comment out the code and replace with throw new IOException()

    • Egad!

      We need a way to inject a fault without changing the code!


Why use fault injection l.jpg
Why Use Fault Injection

  • Fault occurred in a production application and it is determined that that is a valid fault (i.e. it is expected to occur)

    • Inject the fault in a testing environment without having to reproduce the conditions that actually caused the fault

  • Any fault that is difficult or inconvenient to reproduce in testing:

    • Out of memory

    • Disk full

    • Database crash

    • 3rd party library / Java bug

    • Deadlock

    • Resource contention

    • Dumb users

    • Invalid / stale sessions

    • Corner cases


Why use fault injection cont l.jpg
Why Use Fault Injection (cont)

  • You publish an API and you want to be tolerant of user input or response

  • It is inconvenient or undesirable to refactor code to facilitate testing

    • e.g. 3rd party library

  • You believe that testing code coverage is a good metric to measure for code quality [11]

    • Injecting faults forces exception paths to be followed

      More to come …


Reminder l.jpg
Reminder

You don’t have to recreate the conditions that caused the fault …

… you only have to recreate the fault

e.g. if an OhNoException crashed your application because some little old lady drove her 1966 Buick Skylark into your datacenter -- you don’t need the Skylark or the little old lady to fix the application


Quick aside l.jpg
Quick Aside

Traditionally, Fault Injection is centers around faults derived from hardware:

  • Disk faults such as file missing and corrupt file

  • Network faults such as host not found, host inaccessible, and high latency

  • Memory faults such as corrupt memory

  • Operating system faults such as unexpected hardware interrupts

    Traditionally used in mission-critical fault tolerant environments


Quick aside cont l.jpg
Quick Aside (cont)

Fault Injection comes in two favors:

  • Hardware-based

    • Typically requires specialized hardware

  • Software-based

    • Traditionally attempts to mimic hardware-based fault injection and typically involves direct interaction with the operating system

    • More recently expanded to include any fault that can occur in software such as:

      • Mutated bytecode

      • Exceptions

      • Invalid or mutated input / output values

      • Deadlock / resource contention

      • Application Security

      • etc.


Example13 l.jpg
Example

Creating a directory and temporary file

final File directory = new File(…);

final boolean success = directory.mkdirs();

if(success) {

final File file = File.createTempFile(…, …, directory);

...

}

Faults:

  • Making the parent directory(s) fails (returns false)

  • Temp file creation failure (throws IOException)

    • Permissions

    • Parent directory(s) deleted

  • Injection of a specific temp file


Example cont l.jpg
Example (cont)

Great uses of Fault Injection that I can’t easily show in examples:

  • Deeply nested exceptions

  • Effects of unchecked exceptions

  • Consequences of using catch(Exception e)

    i.e.NullPointerExceptions that go unnoticed

  • Complex / real-world cases


Intelligent fault injection l.jpg
Intelligent Fault Injection

Fault injection doesn’t have to be all on or all off

Logic can be coded around injected faults

e.g.InputStream.read()

  • Throw IOException after n bytes are read

  • Return -1 (EOF) one byte before the actual EOF

  • Sporadically mutate the read bytes


Beyond fault injection l.jpg
Beyond Fault Injection

The techniques that I’m going to demonstrate aren’t specific to injecting faults (i.e. conditions that may lead toa failure)

  • “What if” analysis

  • Test difficult corner cases

    e.g.some optimization routine runs when certain conditions are met

  • Initialize application state to some desired value

  • etc.

    The techniques are similar to mock objects or IoC but much more granular


Summary l.jpg
Summary

Intercept operations and inject code to:

  • Perform a custom function

  • Return a custom value

  • Modify a input parameter

  • Throw an exception

  • Set the state of an object to a specific value


Checkpoint l.jpg
Checkpoint

Done:

  • Quick Introduction to Fault Injection

  • Why Use Fault Injection

  • Fault Injection Examples

  • Beyond Fault Injection

    Next:

  • Quick Introduction to AOP

  • Fault Injection in Java

    • AspectJ

    • Javassist

  • Wrap Up


Checkpoint cont l.jpg
Checkpoint (cont)

Questions?


Aspect oriented programming 101 l.jpg
Aspect Oriented Programming 101

Join Point

a point in the flow of a running program [14]

It is the “where” in AOP

  • Not all AO environments support all join points

    e.g. Spring AOP only supports method-based join points


Aspect oriented programming 101 cont l.jpg
Aspect Oriented Programming 101 (cont)

Join point examples

  • reading or writing a field

  • calling or executing a method or constructor

  • catching or throwing an exception

    A “join point shadow” is the location of a join point in the source code or bytecode of the program [5]

    We will use “join point” and “join point shadow” interchangeably


Aspect oriented programming 101 cont22 l.jpg
Aspect Oriented Programming 101 (cont)

Pointcut

a set of join points [6]

Effectively a query where the join points are the data that is being queried – the “which”


Aspect oriented programming 101 cont23 l.jpg
Aspect Oriented Programming 101 (cont)

Advice

code that executes at each join point picked out by a pointcut[7]

Advice is the code that you’re injecting -- the “what”


Aspect oriented programming 101 cont24 l.jpg
Aspect Oriented Programming 101 (cont)

Aspect

a logical collection of advice

An aspect is to AO what an object is to OO


Aspect oriented programming 101 cont25 l.jpg
Aspect Oriented Programming 101 (cont)

Introduction (aka Mixin or inter-type declaration)

advice that provides additional functionality to a class

e.g. methods or member variables can be added via an Introduction to the “interface” of a class


Aspect oriented programming 101 cont26 l.jpg
Aspect Oriented Programming 101 (cont)

Weaving

the process by which specific join points are identified by pointcuts and advice is injected into bytecode


How aop works l.jpg
How AOP Works

  • Compile-time

    a specialized compiler is used to weave the source code and advice together to produce bytecode

  • Post-compile (aka binary weaving)

    weaves advice into existing class files modifying them as necessary

  • Load-time

    a classloader performs binary weaving as the class files are loaded into the JVM or a Java agent is used via JVMTI (JVM Tool Interface)

  • Run-time

    classes that are already loaded by the JVM are woven with aspects via JPDA (Java Platform Debugger Architecture) to replace method bodies. Dynamic proxies can also be used.


Load time weaving l.jpg
Load-Time Weaving

We’re going to focus on load-time weaving (LTW)

  • Advantange (over compile-time or post-compile): does not modify classes on disk

  • Disadvantage: increases application startup time


Java aop l.jpg
Java AOP

  • AspectJ (http://www.eclipse.org/aspectj/): Excellent all-around solution.

  • dynaop (https://dynaop.dev.java.net/) Bob Lee’s stripped down AOP: Requires proxy creation in code (i.e. not 100% transparent).

  • CGLIB (http://cglib.sourceforge.net/): a roll-your-own solution using proxies. Requires proxy creation in code (i.e. not 100% transparent). For load-time weaving a custom class loader would be needed.

  • BCEL (http://jakarta.apache.org/bcel/): roll-your-own solution requiring knowledge of Java byte code to write advice. For load-time weaving a custom class loader would be needed.

  • Javassist (http://www.csg.is.titech.ac.jp/~chiba/javassist/): A large step above CGLIB / BCEL in that it includes a stripped down compiler and modifies the advised class on the fly. Load-time weaving is supported.

  • Spring AOP (http://www.springframework.org/docs/reference/aop.html): Limited to method-based joinpoints. If using the @AspectJ annotations then it effectively becomes AspectJ otherwise standard Spring-style XML configuration. JDK dynamic proxies or CGLIB for implementation. If you’re already using Spring and only need to advise coarse-grained objects (i.e. the objects that you’re IoC’ing) and method-based jointpoints are suitable then this is a good solution.

  • JMangler (http://roots.iai.uni-bonn.de/research/jmangler/): A step above BCEL / CGLIB but a step below Javassist. Requires knowledge of Java byte code (it uses the BCEL API for advice). Load-time weaving is supported.

  • JBoss-AOP (http://labs.jboss.com/portal/jbossaop/): Effectively on par with AspectJ. If you’re using JBoss then this would be the best solution. Even has support in their IDE.



Aspectj l.jpg
AspectJ

  • AspectJ

    http://www.eclipse.org/aspectj/

  • Documentation

    http://www.eclipse.org/aspectj/doc/released/

  • AspectJ FAQ

    http://www.eclipse.org/aspectj/doc/next/faq.html

  • AspectJ Development Environment Guide

    http://www.eclipse.org/aspectj/doc/next/devguide/index.html

  • AspectJ Programming Guide

    http://www.eclipse.org/aspectj/doc/released/progguide/index.html

  • AspectJ Quick Reference

    http://www.eclipse.org/aspectj/doc/released/quick5.pdf


Load time weaving aspectj l.jpg
Load-Time Weaving AspectJ

Load-time Weaving can be accomplished via [13]:

  • -javaagent:pathto/aspectjweaver.jar

  • aj.bat

  • Custom classloader (WeavingURLClassLoader)

    Both require a META-INF/aop.xml to be located in the classpath


Aspectj examples l.jpg
AspectJ Examples

There are three primary classes in the example:

  • Main

    • Creates and calls the other classes

  • ReadFile

    • Accepts a File on construction

    • displayExistence() displays if the file exists

    • byte[] getBytesFromFile() returns the contents of the file as a byte array

  • TemporaryFile

    • readWriteTemporaryFile()

      • Creates a temp file

      • Writes data into it

      • Reads contents line-by-line

      • Reads contents byte-by-byte

      • Deletes the temp file


Aspectj pros l.jpg
AspectJ Pros

  • Quite mature

  • Integration with IDEs (e.g. AJDT)

  • Responsive forums

  • Does 95% of what you need


Aspectj cons l.jpg
AspectJ Cons

  • Expect to burn a lot of time figuring out the difference between call and execute, within and withincode, etc.

  • Can add quite a bit of time to incremental compiles (Also see bugs #146071 and #145438)

  • It’s very easy to confuse the IDE

  • No access to local variables

  • You have to be careful about other errors in your application. A compile error in some completely unrelated file may cause errors in aspects.

  • Brittle with respect to refactoring especially in the case of load-time weaving. If using AJDT with compile-time weaving then refactorings apply across aspects.


Javassist l.jpg
Javassist

  • Javassist

    http://www.csg.is.titech.ac.jp/~chiba/javassist/

  • Javadoc

    http://www.csg.is.titech.ac.jp/~chiba/javassist/html/index.html

  • Tutorial

    http://www.csg.is.titech.ac.jp/~chiba/javassist/tutorial/tutorial.html



Javassist pros l.jpg
Javassist Pros

  • Provides a relatively simple interface to do fault injection at the level of the bytecode

  • Provides a limited compiler to take away the need to work directly in bytecode (e.g. BCEL)


Javassist cons l.jpg
Javassist Cons

  • Limited compiler

  • There’s much that you need to roll in order to use it

  • Confusing code fragment language


Future work l.jpg
Future Work

  • Simplify fault injection + JUnit

    e.g. run a set of test cases with different faults

  • Library of faults for java.* and javax.*

  • Profiler for java.* and javax.* faults

  • Encourage fault libraries and profilers for common 3rd party libs



References l.jpg
References

[1] Harbulot, B. & Gurd, J. (2005). A join point for loops in AspectJ.

http://www.cs.iastate.edu/~leavens/FOAL/papers-2005/harbulot-gurd.pdf

[2] http://www.eclipse.org/aspectj/doc/released/faq.html#q:joinpoints

[3] http://www.aspectprogrammer.org/blogs/adrian/2004/06/go_from_stateme.html

[4] Ossher, H. & Tarr, P. Operation-Level Composition: A Case in (Join) Point

http://www.ccs.neu.edu/research/demeter/SOP/Tarr.pdf

[5] Hilsdale, E. & Hugunin, J. Advice Weaving in AspectJ

http://hugunin.net/papers/aosd-2004-cameraReady.pdf

[6] http://en.wikipedia.org/wiki/Pointcut

[7] http://www.eclipse.org/aspectj/doc/released/faq.html#q:advice

[8] Hsueh, M, Tasi, T, & Iyer, R. Fault InjectionTechniques and Tools.

IEEE CS Press, 1997, pp. 75-82

[9] Clark, J. & Pradhan, D. Fault Injection. IEEE CS Press, 1995, pp. 47-56

[10] http://en.wikipedia.org/wiki/Fault_%28technology%29

[11] http://blogs.msdn.com/bwill/archive/2003/09/23/51169.aspx

[12] Espak, M. Improving Efficiency by Weaving at Run-time

http://www.old.netobjectdays.org/pdf/03/papers/ws-yrw/415.pdf

[13] http://www.eclipse.org/aspectj/doc/released/devguide/ltw-configuration.html

[14] http://en.wikipedia.org/wiki/Join_point


Thank you l.jpg
Thank You

Rob Grzywinski

http://www.realityinteractive.com/rgrzywinski/

[email protected]


ad