1 / 72

Automated Tools for Software Reliability

Automated Tools for Software Reliability. Suhabe Bugrara suhabe@stanford.edu Stanford University. Problem. 80% of development cost on identifying and correcting defects Software errors cost US economy $60 billion annually (0.6% of GDP). Manual Testing.

triage
Download Presentation

Automated Tools for Software Reliability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automated Tools for Software Reliability Suhabe Bugrara suhabe@stanford.edu Stanford University

  2. Problem • 80% of development cost on identifying and correcting defects • Software errors cost US economy $60 billion annually (0.6% of GDP)

  3. Manual Testing • Traditional approach to quality assurance • Expensive • Time consuming • Not systematic • Difficult to quantify effectiveness of test suite • Cannot make any guarantees about reliability • Insufficient for safety critical systems

  4. Automated Tools • Programs to find defects in programs • Automated • Systematic • Easy to quantify effectiveness • Provide guarantees about reliability • Sometimes expensive (for now…) • Sometimes time consuming (for now…)

  5. Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms

  6. Static Driver Verifier • Program analyzer for API usage rules • Developed by Microsoft Research • Applied to device drivers in Windows • Sound: reports all possible errors • Incomplete: may report false alarms

  7. SDV: Overview • Write API usage rule specification • Instrument program with usage checks • Abstract program • Check abstraction for errors • If error found, see if error is false alarm • If false alarm, refine abstraction • If not false alarm, report error as bug

  8. API Usage Rules • Ex. locks are alternatingly acquired and released

  9. API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine • States = {locked, unlocked,error} • Transitions = {acquire(), release()}

  10. API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine • States = {locked, unlocked,error} • Transitions = {acquire(), release()} acquire(); unlocked locked release(); release(); acquire(); error

  11. state { enum { Unlocked=0; Locked=1} state = Unlocked; } KeAcquireSpinLock.return { if (state == Locked) error(); else state = Locked; } KeReleaseSpinLock.return { if (!(state == Locked)) error(); else state = Unlocked; }

  12. enum {Unlocked=0, Locked=1} state = Unlocked voidKeAcquireSpinLock_return() { if (state == Locked) error(); else state = Locked; } voidKeReleaseSpinLock_return() { if (!(state == Locked)) error(); else state = Unlocked; }

  13. 1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: 26: }

  14. enum {Unlocked=0, Locked=1} state = Unlocked voidKeAcquireSpinLock_return() { if (state == Locked) error(); else state = Locked; } voidKeReleaseSpinLock_return() { if (!(state == Locked)) error(); else state = Unlocked; }

  15. 1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: KeAcquireSpinLock_return(); 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: KeReleaseSpinLock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: KeReleaseSpinLock_return(); 26: } Program A

  16. SDV: Abstraction • Construct abstractionB of original program A • Over-approximates reachability • If error() is reachable in A, then it is also reachable in B • This characteristic makes SDV sound • If error() is reachable in B, then it may not be reachable in A • This characteristic makes SDV incomplete • Check abstraction B for any errors

  17. Reachable States Abstraction B real bug! error Original A Sound: If A has error, then B has error

  18. Reachable States false alarm! Abstraction B error Original A Incomplete: If B has error, then A may not have error

  19. bool b1; b1 = false; Abstract state == Locked withb1 voidKeAcquireSpinLock_return() { if (b1) error(); else b1 = true; } voidKeReleaseSpinLock_return() { if (!(b1)) error(); else b1 = false; }

  20. 1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: ; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } 23: } while (SdvMakeChoice()); 24: ; 25: KeReleaseSpinLock_return(); 26: } Program B

  21. 1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: ; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } 23: } while (SdvMakeChoice()); 24: ; 25: KeReleaseSpinLock_return(); 26: } Error trace found!

  22. 1: voidexample() { 2: do { 3: KeAcquireSpinLock(); 4: KeAcquireSpinLock_return(); 5: nPacketsOld = nPackets; 6: req = devExt->WLHV 7: if (req && req->status) { 8: devExt->WLHV = req->Next 9: KeReleaseSpinLock(); 10: KeReleaseSpinLock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->IoS.Status = SUCCCESS; 14: irp->IoS.Info = req->Status; 15: } else { 16: irp->IoS.Status = FAIL; 17: irp->IoS.Info = req->Status; 18: } 19: SmartDevFreeBlock(req); 20: IoCompleteRequest(irp); 21: nPackets++; 22: } 23: } while (nPackets!=nPacketsOld); 24: KeReleaseSpinLock(); 25: KeReleaseSpinLock_return(); 26: } But, no bug in original program!

  23. 1: voidexample() { 2: do { 3: ; 4: KeAcquireSpinLock_return(); 5: b2 = false; 6: ; 7: if (SdvMakeChoice()) { 8: ; 9: ; 10: KeReleaseSpinLock_return(); 11: ; 12: if (SdvMakeChoice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: b2 = !b2 ? true : SdvMakeChoice(); 22: } 23: } while (b2); 24: ; 25: KeReleaseSpinLock_return(); 26: } ProgramC

  24. Reachable States Abstraction B error Refined C Original A false alarm no longer reported!

  25. SDV: Summary • Write API usage rule specification • Instrument program with usage checks • Abstract program • Check abstraction for errors • If error found, see if error is false alarm • If false alarm, refine abstraction • If not false alarm, report error as bug

  26. Soundness • Assume memory safety • No buffer/integer overflows • Safe memory management • No null pointer dereferences • Oversimplified harness • Use stubs to model calls into OS procedures • Stubs may not represent all behavior

  27. Research Challenges in Verification • Eliminate assumption of memory safety • Eliminate false alarms • Scale to the entire operating system • Verify more complicated properties • prove consistency of file system data structures

  28. Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms

  29. EXE • Automatically generate test cases that explore important program paths • Developed by Dawson Engler’s group • Bug finding tool • Unsound: may not report all errors • Complete: never reports false alarms

  30. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }

  31. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }

  32. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) find a solution using an automatic constraint solver… x = -1

  33. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) && (ret = -x) find a solution using an automatic constraint solver… x = 12345678

  34. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) && (ret = x) find a solution using an automatic constraint solver… x = 4

  35. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } EXE automatically generated test cases for each path… x = -1 x = 12345678 x = 4

  36. intbad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }

  37. 1: intsymbolic_bad_abs (int x) { 2: add_constraints(x >= INT_MIN, x <= INT_MAX); 3: ret = new symbol; 4: 5: if (fork() == child) { 6: add_constraints(x < 0, ret = -x); 7: return ret; 8://(x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) 9: } else 10: add_constraints(x >= 0); 11: 12: if (fork() == child) { 13: add_constraints(x = 12345678, ret = -x); 14: return ret; 15: //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) 16:// && (ret = -x) 17: } else 18: add_constraints(x != 12345678); 19: 20: add_constraints(ret = x); 21: return ret; 22: //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) 23:&& (ret = x) 24:}

  38. 1: intmain (void) { 2: unsigned i, t, a[4] = { 1, 3, 5, 2}; 3: make_symbolic(&i); 4: 5: if (i >= 4) 6: exit(0); 7: 8: char *p = (char *) a + i * 4; 9: *p = *p – 1; 10: 11: t = a[*p]; 12: 13: t = t / a[i]; 14: 15: if (t == 2) 16: assert(i == 1); 17: else 18: assert(i == 3); 19: }

  39. Review • Why does SDV produce false alarms and EXE doesn’t? • Why use SDV, then?

  40. Saturn • Large-scale program verification • Developed by Alex Aiken’s group • Sound: reports all errors • Incomplete: may report false alarms • Gives guarantees of reliability on systems as large as the Linux kernel with over 6.2 million lines of code

  41. Program Analyzers Complete Incomplete Undecidable Decidable Sound • Reports all errors • Reports no false alarms • Reports all errors • May report false alarms Decidable Decidable Unsound • May not report all errors • Reports no false alarms • May not report all errors • May report false alarms

  42. Unchecked User Pointer Dereferences • Security property of operating systems • Two types of pointers in operating systems • kernel pointer: pointer created by the operating system • user pointer: pointer created by a user application and passed to the operating system via an entry point such as a system call • Must check that a user pointer points into userspace before dereferencing it

More Related