1 / 25

MADARA KATS: Automated Testing with Distributed Knowledge and Reasoning

MADARA KATS: Automated Testing with Distributed Knowledge and Reasoning. James Edmondson Vanderbilt Institute for Software Integrated Systems Mail: james.r.edmondson@vanderbilt.edu Project site: http://madara.googlecode.com. KATS: Table of Contents. Scenario. Overview. Modeling.

yair
Download Presentation

MADARA KATS: Automated Testing with Distributed Knowledge and Reasoning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MADARA KATS:Automated Testing with Distributed Knowledge and Reasoning James Edmondson Vanderbilt Institute for Software Integrated Systems Mail: james.r.edmondson@vanderbilt.edu Project site: http://madara.googlecode.com

  2. KATS: Table of Contents Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Motivating scenario • Solution overview • Modeling tests • Disseminating and evaluating test knowledge • Automating Android smartphone tests with MAML • Demonstration • Conclusions

  3. KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Three computers and two smart phones connected to a network • The two smart phones are connected to two of the computers network

  4. KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • The phones communicate with system services on the server • The phones must wait for the system services to warm up before connecting (5 seconds) server System Services network

  5. KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • A race condition exists if App 2 is started immediately after App 1 • How do we automatedly test for this type of scenario? server Phone App 1 System Services network Phone App 2

  6. KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • If the test fails, we may have to run a separate test on another host server Phone App 1 System Services network Process Results Process Results Phone App 2 Phone App 3 Process Results Process Results • After the tests are performed, each host must process its log results

  7. KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Scenario Requirements: • Ability to sequence tests, preferably without a need for a centralized server (single point of failure) • Ability to sequence different types of programs (e.g. Python results parsers, C++ services, and Java smart phone applications) • Ability to disseminate knowledge of test failure and success • Configurable knowledge reasoning service to make sense of the incoming information • Portable to Windows, Linux and other operating systems

  8. MADARA: Architectural Overview Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • EXTERNAL DEPENDENCIES • C++ for speed and portability • ACE for portable threads and OS calls • DDS for transport of knowledge • Android SDK for Android smartphone dev • Monkeyrunner for sending events to phones KATS MAML KaRL Monkeyrunner ACE DDS Android SDK C++ Python Java • DEVELOPED TECHNOLOGIES • MAMLfor instrumentation of phones • KaRLfor knowledge and reasoning • KATS for automated testing

  9. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The barrier condition is met when all potentially distributed entities participating in a test have reached the first phase of their life cycle. Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  10. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The precondition is a knowledge requirement that must be true before proceeding (e.g. ServicesStarted for Phone App 1) Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  11. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The temporal delay is a platform-neutral sleep of at least the specified number of seconds Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  12. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postdelay is a knowledge expression to evaluate after the temporal delay has finished. Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  13. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The application launch creates a heavy weight process to run the user-specific executable Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  14. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postlaunch phase begins with the evaluation of user-provided knowledge expressions and extends until the application exits or is killed Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  15. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postcondition is evaluated after the application has exited. A return value is made available to the user-specified knowledge expression Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  16. KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • During the exit phase, the testing entity cleans up and ends its life cycle Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.

  17. KATS: Testing Entity Modeling Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • A GME DSML paradigm is provided for modeling all testing entities: • Visual interface for configuring entities and their life cycles • The modeled entity attributes are generated into an XML file • Recursive includes are allowed to chain entity groups and provide for intricate testing sequences • Provides specialized KATS processes like the KATS Observer, which allows for knowledge monitoring of active tests

  18. KATS: Disseminating Knowledge Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KATS uses the MADARA KaRL reasoning engine • Portable reasoning engine built for distributed real-time knowledge services • Microsecond latency on evaluations • Capable of evaluating millions of knowledge rules per second • Built in C++ on top of ACE • Provides network transports for two DDS vendors • Open Splice Community Edition (open source) • NDDS (licensed and closed source)

  19. KATS: Disseminating Knowledge Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KaRL provides an intuitive multimodal logic • Supports declarative or rule-based expressions • .kats.result == 2 => PhoneApp1.failed = 1 • PhoneApp1.failed && Services.running && (StartTest2 = 1) • Supports local variables (variable starts with a period) which are not disseminated • Global variable values will be disseminated to all entities in the domain when changed • First class support for 264 knowledge values per variable • No available ontologies for non-integer types but can be supported

  20. KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • So… what does this have to do with Smart Phones? • MADARA contains a Python library called MAML which allows for instrumenting an Android smart phone from a connected host machine • Uses a combination of Android Monkeyrunner and the Android Debug Bridge • Can also run instrumentation apks (installed programs that manipulate a different installed program) and JUnit tests • The MAML library is open source and may be downloaded separately from the rest of the MADARA source code

  21. KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • So… what the heck is Monkeyrunner? • The functional parts of Monkeyrunnerallow for sending KeyEvents (like mimicking a user clicking a button, etc.) to the smart phone • Also allows for taking snapshots of the screen and saving it to disk • Wait… so what does this MADARA MAML thingy do? • Encapsulates the working Monkeyrunner functionality • Fixes the broken parts of the API • Provides mechanisms for directly using the Android Debug Bridge • Provides convenience functions for instrumentinglogcat from Python • Provides a much more user friendly library of functions to send KeyEventsthan Monkeyrunner does

  22. KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Okay… but what does MAML have to do with KATS? • Python scripts that use the MAML library may be fitted with the testing entity life cycle of KATS to instrument smart phones • KATS are MAMLs…

  23. KATS: Demo the thing already! Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Why are you looking at this screen? • Our powerpoint instrumentation feature is broken…

  24. KATS: Conclusions Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KATS features a flexible test entity life cycle model that provides fine-grained sequencing support for automating almost any test, including most that require microsecond precision to replicate race conditions between hosts • The KATS paradigm for GME provides an intuitive interface for configuring all aspects of tests and their sequencing • The KaRL engine provides a dynamic knowledge and reasoning service that is built into appropriate KATS life cycle phases • The MAML Python library provides a extensible interface for instrumentingAndroid Smart Phone applications • One day, these technologies may contribute to a distributed, automated deployment that walks your dog and waters your plants while you are playing with robots at work.

  25. KATS: Conclusions Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Email • James Edmondson (james.r.edmondson@vanderbilt.edu) • Websites • MADARA project site (http://madara.googlecode.com) • Distributed Reasoner blog (http://distributedreasoner.blogspot.com)

More Related