1 / 43

Simplify Testing using Fault Injection

Simplify Testing using Fault Injection. Rob Grzywinski rgrzywinski@yahoo.com. Motivation. Writing and testing an IO-intensive application Robust, asynchronous event (message) collector / router NIO (early 1.4 days) How to best simulate and test various failure modes?

danielreed
Download Presentation

Simplify Testing using Fault Injection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Simplify Testing using Fault Injection Rob Grzywinski rgrzywinski@yahoo.com

  2. Motivation • Writing and testing an IO-intensive application • Robust, asynchronous event (message) collector / router • NIO (early 1.4 days) • How to best simulate and test various failure modes? • NIO buggy as all get-out • Networks are inherently flaky and problematic • Didn’t want to have to obfuscate or drastically change code in order to test it

  3. Agenda • Quick Introduction to Fault Injection • Why Use Fault Injection • Fault Injection Examples • Beyond Fault Injection • Quick Introduction to AOP • Fault Injection in Java • AspectJ • Javassist • Wrap Up

  4. Fault Injection 101 • A fault is an abnormal condition or defect which may lead to a failure[10] • Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response [9] • The goal isn’t to recreate the conditions that produced the fault

  5. Fault Injection 101 (cont) There are two primary steps to Fault Injection: • To identify the set of faults that can occur within an application, module, class, method, etc. • e.g. if the application does not use the network then there’s no point in injecting network faults • In practice this isn’t as easy as it sounds e.g. 3rd party libs • To exercise those faults to evaluate how the application responds • Does the application detect the fault, is it isolated and does the application recover from it? [8]

  6. Example byte[]readFile() throws IOException { ... final InputStream is = new FileInputStream(…); ... while((offset < bytes.length) && (numRead = is.read(bytes, offset, (bytes.length - offset))) >= 0) offset += numRead; ... is.close(); return bytes; } (from http://javaalmanac.com/egs/java.io/File2ByteArray.html) What could go wrong with this code? • new FileInputStream() can throw FileNotFoundException • InputStream.read() can throw IOException and IndexOutOfBoundsException and can return -1 for end of file • is.close() can throw IOException How do we test how the application responds to one of these situations? Specifically, how do we inject these faults so that we can test how the application responds?

  7. Possible Techniques • Force the situations at the OS level • Quite hairy to reproduce reliably when they’re needed • Refactor the snot out of it • Replace the call to InputStream.read() to some local instrumented method • Create our own instrumented InputStream subclass possibly using mock objects • Inject the subclass via IoC (requires some framework such as PicoContainer or Spring) • Just comment out the code and replace with throw new IOException() • Egad! We need a way to inject a fault without changing the code!

  8. Why Use Fault Injection • Fault occurred in a production application and it is determined that that is a valid fault (i.e. it is expected to occur) • Inject the fault in a testing environment without having to reproduce the conditions that actually caused the fault • Any fault that is difficult or inconvenient to reproduce in testing: • Out of memory • Disk full • Database crash • 3rd party library / Java bug • Deadlock • Resource contention • Dumb users • Invalid / stale sessions • Corner cases 

  9. Why Use Fault Injection (cont) • You publish an API and you want to be tolerant of user input or response • It is inconvenient or undesirable to refactor code to facilitate testing • e.g. 3rd party library • You believe that testing code coverage is a good metric to measure for code quality [11] • Injecting faults forces exception paths to be followed More to come …

  10. Reminder You don’t have to recreate the conditions that caused the fault … … you only have to recreate the fault e.g. if an OhNoException crashed your application because some little old lady drove her 1966 Buick Skylark into your datacenter -- you don’t need the Skylark or the little old lady to fix the application

  11. Quick Aside Traditionally, Fault Injection is centers around faults derived from hardware: • Disk faults such as file missing and corrupt file • Network faults such as host not found, host inaccessible, and high latency • Memory faults such as corrupt memory • Operating system faults such as unexpected hardware interrupts Traditionally used in mission-critical fault tolerant environments

  12. Quick Aside (cont) Fault Injection comes in two favors: • Hardware-based • Typically requires specialized hardware • Software-based • Traditionally attempts to mimic hardware-based fault injection and typically involves direct interaction with the operating system • More recently expanded to include any fault that can occur in software such as: • Mutated bytecode • Exceptions • Invalid or mutated input / output values • Deadlock / resource contention • Application Security • etc.

  13. Example Creating a directory and temporary file final File directory = new File(…); final boolean success = directory.mkdirs(); if(success) { final File file = File.createTempFile(…, …, directory); ... } Faults: • Making the parent directory(s) fails (returns false) • Temp file creation failure (throws IOException) • Permissions • Parent directory(s) deleted • Injection of a specific temp file

  14. Example (cont) Great uses of Fault Injection that I can’t easily show in examples: • Deeply nested exceptions • Effects of unchecked exceptions • Consequences of using catch(Exception e) i.e.NullPointerExceptions that go unnoticed • Complex / real-world cases

  15. Intelligent Fault Injection Fault injection doesn’t have to be all on or all off Logic can be coded around injected faults e.g.InputStream.read() • Throw IOException after n bytes are read • Return -1 (EOF) one byte before the actual EOF • Sporadically mutate the read bytes

  16. Beyond Fault Injection The techniques that I’m going to demonstrate aren’t specific to injecting faults (i.e. conditions that may lead toa failure) • “What if” analysis • Test difficult corner cases e.g.some optimization routine runs when certain conditions are met • Initialize application state to some desired value • etc. The techniques are similar to mock objects or IoC but much more granular

  17. Summary Intercept operations and inject code to: • Perform a custom function • Return a custom value • Modify a input parameter • Throw an exception • Set the state of an object to a specific value

  18. Checkpoint Done: • Quick Introduction to Fault Injection • Why Use Fault Injection • Fault Injection Examples • Beyond Fault Injection Next: • Quick Introduction to AOP • Fault Injection in Java • AspectJ • Javassist • Wrap Up

  19. Checkpoint (cont) Questions?

  20. Aspect Oriented Programming 101 Join Point a point in the flow of a running program [14] It is the “where” in AOP • Not all AO environments support all join points e.g. Spring AOP only supports method-based join points

  21. Aspect Oriented Programming 101 (cont) Join point examples • reading or writing a field • calling or executing a method or constructor • catching or throwing an exception A “join point shadow” is the location of a join point in the source code or bytecode of the program [5] We will use “join point” and “join point shadow” interchangeably

  22. Aspect Oriented Programming 101 (cont) Pointcut a set of join points [6] Effectively a query where the join points are the data that is being queried – the “which”

  23. Aspect Oriented Programming 101 (cont) Advice code that executes at each join point picked out by a pointcut[7] Advice is the code that you’re injecting -- the “what”

  24. Aspect Oriented Programming 101 (cont) Aspect a logical collection of advice An aspect is to AO what an object is to OO

  25. Aspect Oriented Programming 101 (cont) Introduction (aka Mixin or inter-type declaration) advice that provides additional functionality to a class e.g. methods or member variables can be added via an Introduction to the “interface” of a class

  26. Aspect Oriented Programming 101 (cont) Weaving the process by which specific join points are identified by pointcuts and advice is injected into bytecode

  27. How AOP Works • Compile-time a specialized compiler is used to weave the source code and advice together to produce bytecode • Post-compile (aka binary weaving) weaves advice into existing class files modifying them as necessary • Load-time a classloader performs binary weaving as the class files are loaded into the JVM or a Java agent is used via JVMTI (JVM Tool Interface) • Run-time classes that are already loaded by the JVM are woven with aspects via JPDA (Java Platform Debugger Architecture) to replace method bodies. Dynamic proxies can also be used.

  28. Load-Time Weaving We’re going to focus on load-time weaving (LTW) • Advantange (over compile-time or post-compile): does not modify classes on disk • Disadvantage: increases application startup time

  29. Java AOP • AspectJ (http://www.eclipse.org/aspectj/): Excellent all-around solution. • dynaop (https://dynaop.dev.java.net/) Bob Lee’s stripped down AOP: Requires proxy creation in code (i.e. not 100% transparent). • CGLIB (http://cglib.sourceforge.net/): a roll-your-own solution using proxies. Requires proxy creation in code (i.e. not 100% transparent). For load-time weaving a custom class loader would be needed. • BCEL (http://jakarta.apache.org/bcel/): roll-your-own solution requiring knowledge of Java byte code to write advice. For load-time weaving a custom class loader would be needed. • Javassist (http://www.csg.is.titech.ac.jp/~chiba/javassist/): A large step above CGLIB / BCEL in that it includes a stripped down compiler and modifies the advised class on the fly. Load-time weaving is supported. • Spring AOP (http://www.springframework.org/docs/reference/aop.html): Limited to method-based joinpoints. If using the @AspectJ annotations then it effectively becomes AspectJ otherwise standard Spring-style XML configuration. JDK dynamic proxies or CGLIB for implementation. If you’re already using Spring and only need to advise coarse-grained objects (i.e. the objects that you’re IoC’ing) and method-based jointpoints are suitable then this is a good solution. • JMangler (http://roots.iai.uni-bonn.de/research/jmangler/): A step above BCEL / CGLIB but a step below Javassist. Requires knowledge of Java byte code (it uses the BCEL API for advice). Load-time weaving is supported. • JBoss-AOP (http://labs.jboss.com/portal/jbossaop/): Effectively on par with AspectJ. If you’re using JBoss then this would be the best solution. Even has support in their IDE.

  30. Questions?

  31. AspectJ • AspectJ http://www.eclipse.org/aspectj/ • Documentation http://www.eclipse.org/aspectj/doc/released/ • AspectJ FAQ http://www.eclipse.org/aspectj/doc/next/faq.html • AspectJ Development Environment Guide http://www.eclipse.org/aspectj/doc/next/devguide/index.html • AspectJ Programming Guide http://www.eclipse.org/aspectj/doc/released/progguide/index.html • AspectJ Quick Reference http://www.eclipse.org/aspectj/doc/released/quick5.pdf

  32. Load-Time Weaving AspectJ Load-time Weaving can be accomplished via [13]: • -javaagent:pathto/aspectjweaver.jar • aj.bat • Custom classloader (WeavingURLClassLoader) Both require a META-INF/aop.xml to be located in the classpath

  33. AspectJ Examples There are three primary classes in the example: • Main • Creates and calls the other classes • ReadFile • Accepts a File on construction • displayExistence() displays if the file exists • byte[] getBytesFromFile() returns the contents of the file as a byte array • TemporaryFile • readWriteTemporaryFile() • Creates a temp file • Writes data into it • Reads contents line-by-line • Reads contents byte-by-byte • Deletes the temp file

  34. AspectJ Pros • Quite mature • Integration with IDEs (e.g. AJDT) • Responsive forums • Does 95% of what you need

  35. AspectJ Cons • Expect to burn a lot of time figuring out the difference between call and execute, within and withincode, etc. • Can add quite a bit of time to incremental compiles (Also see bugs #146071 and #145438) • It’s very easy to confuse the IDE • No access to local variables • You have to be careful about other errors in your application. A compile error in some completely unrelated file may cause errors in aspects. • Brittle with respect to refactoring especially in the case of load-time weaving. If using AJDT with compile-time weaving then refactorings apply across aspects.

  36. Javassist • Javassist http://www.csg.is.titech.ac.jp/~chiba/javassist/ • Javadoc http://www.csg.is.titech.ac.jp/~chiba/javassist/html/index.html • Tutorial http://www.csg.is.titech.ac.jp/~chiba/javassist/tutorial/tutorial.html

  37. Javassist Example

  38. Javassist Pros • Provides a relatively simple interface to do fault injection at the level of the bytecode • Provides a limited compiler to take away the need to work directly in bytecode (e.g. BCEL)

  39. Javassist Cons • Limited compiler • There’s much that you need to roll in order to use it • Confusing code fragment language

  40. Future Work • Simplify fault injection + JUnit e.g. run a set of test cases with different faults • Library of faults for java.* and javax.* • Profiler for java.* and javax.* faults • Encourage fault libraries and profilers for common 3rd party libs

  41. Questions?

  42. References [1] Harbulot, B. & Gurd, J. (2005). A join point for loops in AspectJ. http://www.cs.iastate.edu/~leavens/FOAL/papers-2005/harbulot-gurd.pdf [2] http://www.eclipse.org/aspectj/doc/released/faq.html#q:joinpoints [3] http://www.aspectprogrammer.org/blogs/adrian/2004/06/go_from_stateme.html [4] Ossher, H. & Tarr, P. Operation-Level Composition: A Case in (Join) Point http://www.ccs.neu.edu/research/demeter/SOP/Tarr.pdf [5] Hilsdale, E. & Hugunin, J. Advice Weaving in AspectJ http://hugunin.net/papers/aosd-2004-cameraReady.pdf [6] http://en.wikipedia.org/wiki/Pointcut [7] http://www.eclipse.org/aspectj/doc/released/faq.html#q:advice [8] Hsueh, M, Tasi, T, & Iyer, R. Fault InjectionTechniques and Tools. IEEE CS Press, 1997, pp. 75-82 [9] Clark, J. & Pradhan, D. Fault Injection. IEEE CS Press, 1995, pp. 47-56 [10] http://en.wikipedia.org/wiki/Fault_%28technology%29 [11] http://blogs.msdn.com/bwill/archive/2003/09/23/51169.aspx [12] Espak, M. Improving Efficiency by Weaving at Run-time http://www.old.netobjectdays.org/pdf/03/papers/ws-yrw/415.pdf [13] http://www.eclipse.org/aspectj/doc/released/devguide/ltw-configuration.html [14] http://en.wikipedia.org/wiki/Join_point

  43. Thank You Rob Grzywinski http://www.realityinteractive.com/rgrzywinski/ rgrzywinski@yahoo.com

More Related