1 / 40

Getting Results From Testing

Getting Results From Testing. Laura K. Dillon Software Engineering and Network Systems Lab Michigan State University www.cse.msu.edu/sens. Overview. Background: Testing The Problem: Temporal Behavior Validation A Solution: Graphical Interval Logic-Based Oracles. PREMISES.

shiela
Download Presentation

Getting Results From Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Getting Results FromTesting Laura K. Dillon Software Engineering and Network Systems Lab Michigan State University www.cse.msu.edu/sens Software Engineering & Network Systems Laboratory Michigan State University

  2. Overview • Background: Testing • The Problem: Temporal Behavior Validation • A Solution: Graphical Interval Logic-Based Oracles Software Engineering & Network Systems Laboratory Michigan State University

  3. PREMISES • Software intensive systems must be tested extensively • Testing should be doneearly and often Software Engineering & Network Systems Laboratory Michigan State University

  4. Evaluate adequacy of tests Create test plans Run tests Formal Specifications Evaluate correctness of tests Reqs/ Design/ Code Reqs/ Design/ Code Reqs/ Design/ Code Testing is more than running tests! Test plans Test results Software Engineering & Network Systems Laboratory Michigan State University

  5. Formal Specifications Evaluate correctness of tests A specification-based test oracle Checks that behaviors exhibited during testing satisfy the specifications Test results Software Engineering & Network Systems Laboratory Michigan State University

  6. Overview • Background: Testing • The Problem: Temporal Behavior Validation • A Solution: Graphical Interval Logic-Based Oracles Software Engineering & Network Systems Laboratory Michigan State University

  7. The Oracle Problem • Answers the question: Were any of the test executions erroneous? • Concurrency adds a new dimension of complexity to this problem Software Engineering & Network Systems Laboratory Michigan State University

  8. A concurrent system … • Consists of asynchronous communicating threads of control: Tasks / Processes / Threads • Threads synchronize/communicate by sending messages or through sharing data • Temporal ordering matters Software Engineering & Network Systems Laboratory Michigan State University

  9. Cus(1) Cus(N) Change Change Prepay Start Finish Charge Act Operator Pump GAS STATION EXAMPLE . . . [Helmbold, Luckham: 1985] Software Engineering & Network Systems Laboratory Michigan State University

  10. Example Temporal Properties • Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas • If Cus(1) prepays before Cus(2), then Cus(1) gets to pump gas before Cus(2) gets to pump gas Software Engineering & Network Systems Laboratory Michigan State University

  11. When validating behaviors of concurrent software systems . . . • Large number of behaviors can be produced • Behaviors can be long • Behaviors that violate necessary temporal constraints are easily overlooked Software Engineering & Network Systems Laboratory Michigan State University

  12. Overview • Background: Testing • The Oracle Problem: Behavior Validation • A Solution: Graphical Interval Logic-Based Oracles Software Engineering & Network Systems Laboratory Michigan State University

  13. Overview of GIL Oracles • Desired temporal properties of execution traces are expressed formally in Graphical Interval Logic (GIL) • Execution traces are generated in testing • Oracle checks that execution traces satisfy GIL specifications • Displays anomalies to help user identify faults Software Engineering & Network Systems Laboratory Michigan State University

  14. Cus(N) Cus(1) Change Change Programmer defines “conditions” to trace Pump1 Pump1 Pay1 PayN Pay1 PayN Prepay Start Finish Charge Act Operator Pump Software Engineering & Network Systems Laboratory Michigan State University

  15. Formal comments define Pay1, . . ., PayN --%pay$n, for $n in ID_RANGE: Condition is --%triggered by --%begin accept Cus($n):Operator.prepay --%cancelled by --%begin activation Gas_Simulation --%end accept Operator:Cus($n).change Similarly: Pump1, . . ., PumpN Software Engineering & Network Systems Laboratory Michigan State University

  16. init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fini acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start An execution trace induces a state sequence . . . . . . Pay1 Pump1 Pay2 Pump2 . . . . . . . . . Software Engineering & Network Systems Laboratory Michigan State University

  17. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas Software Engineering & Network Systems Laboratory Michigan State University

  18. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) | | Pay1 Pay1 | Pay1 [ ) Pump1 Software Engineering & Network Systems Laboratory Michigan State University

  19. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) | | Pay1 Pay1 | Pay1 [ ) Pump1 Software Engineering & Network Systems Laboratory Michigan State University

  20. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) | | Pay1 Pay1 | Pay1 [ ) Pump1 Software Engineering & Network Systems Laboratory Michigan State University

  21. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) | | Pay1 Pay1 | Pay1 [ ) Pump1 Software Engineering & Network Systems Laboratory Michigan State University

  22. DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas [ ) | | Pay1 Pay1 | Pay1 [ ) Pump1 Software Engineering & Network Systems Laboratory Michigan State University

  23. Fair1.2: If Cus(1) prepays before Cus(2), then Cus(1) gets to pump before Cus(2) gets to pump Software Engineering & Network Systems Laboratory Michigan State University

  24. Fair1.2: [ ) | | Pay1 Pay1 Pay2 | | Pay1 Pay1 | Pump1 [ ) Pump2 Software Engineering & Network Systems Laboratory Michigan State University

  25. | | Pay1 Pay1 Pay2 Fair1.2: [ ) | | Pay1 Pay1 | Pump1 [ ) Pump2 Software Engineering & Network Systems Laboratory Michigan State University

  26. | | Pay1 Pay1 Pay2 Fair1.2: [ ) | | Pay1 Pay1 | Pump1 [ ) Pump2 Software Engineering & Network Systems Laboratory Michigan State University

  27. Fair1.2: [ ) | | Pay1 Pay1 Pay2 | | Pay1 Pay1 | Pump1 [ ) Pump2 Software Engineering & Network Systems Laboratory Michigan State University

  28. | | Pay1 Pay1 Pay2 Fair1.2: [ ) | | Pay1 Pay1 | Pump1 [ ) Pump2 Software Engineering & Network Systems Laboratory Michigan State University

  29. init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fin acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start Pay1 Pump1 Pay2 Pump2 [ ) | | Violates DoesPump1 Pay1 Pay1 [ ) Pay1 Software Engineering & Network Systems Laboratory Michigan State University

  30. s fif and only ifAccept(s, Df ) A GIL formula,f, determines a deterministic finite state automaton, Df , such that, for every finite state sequence, s, . . . Software Engineering & Network Systems Laboratory Michigan State University

  31. DFA for DoesPump1 P1 = Pay1 M1 = Pump1 P1 0 1 P1 P1 P1 2 P1 P1 P1 M1 5 P1 M1 P1 4 3 P1 M1 true P1 P1 M1 Software Engineering & Network Systems Laboratory Michigan State University

  32. init acc C(1):Op.pre acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start end C(1):Pp.fini acc C(2):Pp.start acc Op:C(1).chan end C(2):Pp.fin acc C(1):Op.pre acc C(3):Pp.start acc Op:C(2).chan end C(3):Pp.fin acc C(2):Op.pre acc C(3):Op.pre acc C(1):Pp.start Checking an execution trace . . . . . . Pay1 Pump1 Pay2 Pump2 . . . . . . . . . . . . 0 2 3 3 4 4 4 4 2 2 3 3 3 3 3 3 Software Engineering & Network Systems Laboratory Michigan State University

  33. Summary • Testing concurrent systems is complicated by the need to check temporal properties • Temporal oracle can be produced from GIL specifications • Checks that execution traces satisfy GIL specifications • Displays anomalies to help user identify faults Software Engineering & Network Systems Laboratory Michigan State University

  34. Yet to do • Refine heuristics for displaying failures • Conduct case studies/user studies • Integrate oracles into a proactive debugger Software Engineering & Network Systems Laboratory Michigan State University

  35. Acknowledgements Thanks! This work was supported in part by • NSF grants EIA-0000433, CDA-9617310, and CCR-9896190 • Department of the Navy, Office of Naval Research under grant N00014-01-1-0744 Software Engineering & Network Systems Laboratory Michigan State University

  36. Meridian: An Integrated Toolkit for Developing Interactive Distributed Applications • NSF Experimental Partnership Program • Betty Cheng, Laura Dillon, Phillip McKinley, Kurt Stirewalt Software Engineering & Network Systems Laboratory Michigan State University

  37. Interactive Distributed Applications (IDAs) Interact with users. Processing/data distributed across a network. • Examples: • On-board driver/pilot navigation systems. • Computer-supported collaborative work environments. • Distributed interactive simulation. Software Engineering & Network Systems Laboratory Michigan State University

  38. Characteristics of IDAs • Interactivity: • Must interact with one or more human users. • Design requires prototyping and experimentation. • Concurrency: • Comprise levels of communicating, concurrent components. • Analysis requires formal reasoning. • Reuse: • IDAs built primarily from reusable components. • E.g., comm. protocols, resource managers, data displays. • Design involves selecting/specializing components. Software Engineering & Network Systems Laboratory Michigan State University

  39. Meridian goals • Advance development technology for IDAs. • Formalize development artifacts and relationships between them • Libraries of reusable components • Provide end-to-end automation techniques. • Impact practice: • Must complement existing design methods and notations, e.g.UML Software Engineering & Network Systems Laboratory Michigan State University

  40. IDA External Parameters IDA Reuse Repository IDA Interface Requirements IDA Models IDA Constraints Test Cases, Oracles Meridian Vision Refined Specifications Specifications Design Processing Specification Analysis Testing/ Simulation Model Editing Code Requirements Feedback User Software Engineering & Network Systems Laboratory Michigan State University

More Related