1 / 21

FIG: A Prototype Tool for On-Line Verification of Recovery Mechanisms

FIG: A Prototype Tool for On-Line Verification of Recovery Mechanisms. Naveen Sastry, Pete Broadwell, Jonathan Traupman, David Patterson University of California, Berkeley. Presentation Outline. Introduction Objective/Motivation Background Methods Implementation Test setup Evaluation

katima
Download Presentation

FIG: A Prototype Tool for On-Line Verification of Recovery Mechanisms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. FIG: A Prototype Tool for On-Line Verification of Recovery Mechanisms Naveen Sastry, Pete Broadwell,Jonathan Traupman, David Patterson University of California, Berkeley

  2. Presentation Outline • Introduction • Objective/Motivation • Background • Methods • Implementation • Test setup • Evaluation • Test results • Conclusions

  3. The Berkeley/Stanford ROC Project • Purpose: investigating novel techniques for building highly-dependable Internet services • Example techniques: • Advanced support for operator undo • Stability through targeted restarts • Integrated root cause analysis • Online verification of recovery mechanisms

  4. FIG Project Objective/Motivation Objective: • Develop a lightweight, extensible tool for injecting errors to test recovery code/mechanisms Motivation: • Testing and production environments are always different • Large systems will require recovery code, which should be tested as part of normal operation

  5. “Software’s Invisible Users” User Input User interface Application Other libraries Other apps System libraries (libc) OS Concept: Jim Whittaker Florida Institute of Technology

  6. Related Testing Methods • Ballista (DeVale, Koopman, Siewiorek) • “Top-down” testing of POSIX-compliant OS and library interfaces • Fuzz (Miller, Fredriksen, So) • Tested UNIX applications by feeding them random input streams • Holodeck (Whittaker et al.) • Similar approach to ours, but only for Windows 2000/XP

  7. Normal call path Injected fault FIG Implementation Application • Thin stub library between app & libraries • Traps API calls • Logs them • Inserts faults • Can be inserted into any app without modification • Uses LD_PRELOAD libfig.so libc.so, other libs OS

  8. API stubs are automatically generated Very easy to add new APIs to log Fault injection is under script control Can simulate multiple fault models (e.g., memory pressure) MALLOC_INDEX interval 82 to infinity return 0 errno ENOMEM probability 0.03 OPEN_INDEX // device out of space. interval 100 to infinity return –1 errno ENOSPC probability 0.001 // kernel out of memory. interval 100 to 120 return –1 errno ENOMEM probability 0.1 // too many files open. callnumber 108 return -1 errno EMFILE probability 1.0 Extensibility Sample control file:

  9. Test Setup: Applications • GNU file utilities (ls, mv, etc.) • Emacs 20.7.1 – with and without X • Apache 1.3.22 • Berkeley DB 4.0.14 • Netscape Navigator 4.76 • MySQL server 3.23.36

  10. Test Setup:Instrumented Calls & Their Errors • malloc() – memory exhaustion • read() – I/O error, system call was interrupted • write() – I/O error, no space left on device, call interrupted • open() – memory exhaustion, no space on device, too many files open • select() – memory exhaustion

  11. Test Results: Client Apps

  12. Test Results: Server Apps

  13. Netscape Reacts

  14. Test Results: Overhead • Note: FIG communicates with a separate logging daemon through shared memory to reduce logging overhead. Timing using Berkeley DB (non-transactional) to read, sort and write one million words.

  15. Strategies forReliable Services: • Intelligent retry • ls: “bounded retry” of malloc() • Resource preallocation • Apache: allocates buffer pool at startup • Degraded service • Apache: deactivates logging if disk full • Process pools • Apache and MySQL

  16. FIG as a Prototype for Online Error Injection • Low run-time overhead • Easy to enable/disable • Easy to configure • Extensible • Can simulate multiple fault models

  17. A Case for OnlineError Injection • Recovery code is not usually exercised during normal operation • Deployed environments tend to differ from testing environments • Can run error injection tests on a subset of deployed systems • FIG can simulate common environmental errors

  18. Conclusions • FIG exposed a variety of deficiencies in how our test applications handled environmental errors • Server apps are generally more robust than client applications • FIG exhibits low overhead • FIG is suitable for online error injection

  19. Future Directions • Limitations of FIG: • Only for UNIX-like OSes • Limited to app/library interface (proxy for app/OS interaction) • Make FIG part of a larger test suite • Include clock time and event based error triggers • Greater flexibility in configuration file

  20. Other Related Work • Xept (Vo et al.) • Instruments object code to ensure that error handling code exists • Processor & memory errors • DOCTOR, HYBRID, DEFINE • Process memory corruption • FERRARI, DEFINE

More Related