Pluggable And Code-free Automation System

1. Pluggable And Code-free Automation System Jeffrey Bian, 2007 @ IBM

2. PACAS is a complex transformation system • As designer, developer and team lead • Over 20K (line of code), Java 6.0 • OSS libraries imported: • Tcljava and jacl, Apache Xerces XML package, JACOB • Tech & Tools: • Java, XML/XSLT/XPath, TCL, Java COM bridge • Logo created with Photoshop

3. Agenda • Introduction to Test Automation • PACAS Overview & Design • PACAS Details • In the Whole Quality Process • Practice in Real Project

4. Test Automation in Agile Process • Agile Process • Test Driven • Test automation is a must • Fast iteration • Test automation is a must • Test automation in this documentation targets ‘functional testing’.

5. Traditional Test Automation Headaches • System Under Test always changes or evolves • As we saw in Agile process in previous slides • Hard to maintain scripts (Maintenance cost)

6. Traditional Test Automation Headaches • Coupling with the tools or platforms • May impose great migration workload when tools or platform changes • Nowadays popular test automation tools are IBM Rational Functional Tester and HP WinRunner

7. Scripts Tool D Tool C Library B Library A Traditional Test Automation Headaches • Call for programming skills • Training cost • Debugging cost Oh, the testers still need more time learning to use tool C, tool D, Lib A and Lib B.

8. Unexpected results with test automation • The headaches listed above make test automation less efficient • Even worse in some situations • Low ROI • Let’s try to make things better!

9. Dissecting a functional test • Test Case • Each test case is divided into steps • Each step is further divided into actions (verifications are special actions) • Supporting information • Test data • Test object : A test object is an object or target we operate during the testing. Such as a button on a HTML page if we are to test this page.

10. Come to conclusion • To build a system that • Implements the action-based test style • Presents data, object mapping and test case in a uniform manner • Screens the details of the underlying tools and other structures • Constrains the testing engineers with some well defined format to describe the test cases • Must be flexible and extensible

11. Agenda • Introduction to Test Automation • PACAS Overview & Design • PACAS Details • In the Whole Quality Process • Practice in Real Project

12. Hello, PACAS • PACAS is a transformation system • It transforms XML based test case description into scripts under some tools • In context of PACAS, “test case” and “test script” are interchangeable • Test scripts are in XML format and self-contained. • Our beliefs • Testers are not coders (T.A.N.C.) • Testers need not do real “programming” • Test script should be easy to read and maintain

13. Hello, PACAS • Code-free • Under PACAS test scripts are the same as test cases, which are described in an well-defined XML format. • Test engineers think of the test logic that reflects the business requirement instead of coding logic • They do not have to map test logic again to code logic • Write down test scripts in a more readable way

14. Hello, PACAS • Pluggable • The design model of PACAS is layered • Test Logic Layer -> Test Transformation Layer -> Test Execution Layer • Further, Test Transformation Layer is divided into Parser Sub Layer and Transform Sub Layer • Plenty of interfaces for users to extend • Custom parsers and transformers could be used Parser Sub Layer Transform Sub Layer

15. Hello, PACAS • The Magic! Translate With little efforts Click the link “Quote Order”. Snippet from test case document Snippet from PACAS test script Feed in PACAS And transform Snippet from PACAS test script Snippet from IBM RFT script ( in Java Language)

16. A Scenario Jeff wrote a test case in XML format. Jeff wrote his data and test object description. Jeff calls org.pacas.ScriptFactory to transform his test assets. ScriptFactory will launch data translator and object mapping translator to translate the data.xml and maps.xml if necessary ScriptFactory parses the testMyExcel.xml, translates it into real platform-specific scripts.

17. PACAS is Different … • Presents Another Point of View • Unlike most other framework / system, which is bound to a specific tool and must be programmed, PACAS presents a ‘top-down’ view over tests • Thinking at a Higher Level • To make your testing scripts tool-independent • Action-based • Writing actions to indicate user operations, which is a more natural way for describing tests. • E.g., Click web:myButton. • Looks like COBOL or TCL

18. What PACAS Brings Along… • Separate test logic from coding logic • Shadow the underlying mechanisms • Low training cost • Even no debugging, because the runnable code are generated automatically! • Provide a platform-independent test descriptive method • We only need to switch a set of configuration to generate scripts for different tools.

19. A Closer Look Test Logic Layer Case Information Policies Rules External Tools Data File Suite Policies Case Rules Mapping Generator Case Policies Case Description Object Mapping Eclipse Policy Rules 3 Transformation Layer Parsers Transformation Engine Customization Data Step Code Parser Step Transformer Platform Definition Default Parsers File Composer Action Definition 2 Test Execution Layer Interfaces Platform Specific Data Presentation Platform Specific Object Mapping Presentation Platform Libraries 1 Execution Engine (RFT, WinRunner)

20. How it works Test Transformation Layer Test Logic Layer Test Execution Layer Predefined specific test control library Case Description Action Definition List Data User-defined specific test control library Object Mapping Other supporting components Tool specific configuration set XML Schema for Test cases Data Object Mapping Customized translator XML Schema for Action Definition Transformation Engine Real scripts Real scripts Driver XML to transform test case to code XSLT to transform data / map xmls to tool-specific format Configuration for user project structure

21. Agenda • Introduction to Test Automation • PACAS Overview & Design • PACAS Details • In the Whole Quality Process • Practice in Real Project

22. Basic Structure of Test Cases Some unimportant parts omitted.

23. Basic Structure of Test Cases (Cont.)

24. Within Each ‘Step‘ - Actions • The text section of a step node contains actions • Syntax: actionNameOrAlias param1 param2 … paramN. actionNameOrAlias param1 param2 … paramN -> var. Return Value Evaluation Directive Implicit variable declaration

25. Within Each ‘Step‘ - Actions • There are branch and loop support • With Sequence / Branch / Loop structures we could express most our logic • begin loop … end loop • If … then … else … end if

26. Within Each ‘Step‘ - Verify • At the end there are ‘verify’ nodes • This implies that you have to do some verification ‘after’ some operations • Verifications and actions within one step cannot appear in mixed order.

27. Within Each ‘Step‘ - Verify • Verifications should be predefined • Verifications differ by the ‘type’ attribute • ‘!’ prefix means ‘negative’ verification • Other attributes will have different meanings for different type.

28. Rationale • A. For each test domain, the basic operations (actions) are finite set. • They are defined in some action define xml. • B. For each test domain and specific project, verifications are a finite set. • They are defined in some action define xml.

29. Agenda • Introduction to Test Automation • PACAS Overview & Design • PACAS Details • In the Whole Quality Process • Practice in Real Project

30. From BA to TA • More concerns about test automation in the whole quality process • Business Analyst • How to trace business requirements throughout the test? • Test Architect • How to generate scripts efficiently and effectively?

31. Generate Test Cases from Models ? • For End to End testing scenario • Tests must reflect the business logic. • If we could translate business models into related test scenarios. • “Archetest” story from IBM Watson Lab • Archetest is designed as a system that generates test cases from business requirement description. • Still with lots of difficulties. • Not applicable yet.

32. Generate Test Scripts from Test Cases ? • On the other hand • Test cases could be quite detailed and if we have a uniform schema for authoring test cases • Recall that PACAS scripts are just test cases! • This is where PACAS fits!

33. Quality Process Manager Test Process Definition Test Policy Definition Reporting and Metrics Test Preparation Test Construction Setup and Deploy Test Environment Execute Test Problem Determination And Results Analysis Test Sizing Test Plan created High level plan approved Assign test environments Assign authors of tests Test Case definition Test Design Test Script Authoring Extendibility Deployment of builds Test lab management Hardware/Software Resource Scheduling Running tests Defect submission Make/modify team’s execution assignments Post-mortem Review criteria Integrate to RQM? RQM is a new product from Rational. It is a quality management platform. Unofficially PACAS was once considered as candidate for the ‘Test Construction’ component. Discussed with Lucy Li about the integration thing, still in discussion.

34. Agenda • Introduction to Test Automation • PACAS Overview & Design • PACAS Details • In the Whole Quality Process • Practice in Real Project

35. Improve Productivity • Test scripts generation saved cost • Testers focused on the case and did not spend much time on debugging the scripts! • Data below came from real DSW testing projects The framework here refers to PACAS

36. The Effect • Now PACAS is used by • DSW Testing team • DSW is a very big web application with more than 40 sub applications • FastPass application regression test • Automation POC team initiated by DSW team

37. The Status • Promotion • PACAS is selected as session for 2nd IBM Adoption of Technology Conference • Known by VIP client *X Bank through Lab Base Service team. The client has started a project to adapt PACAS and asked for further sessions. (In Plan) • Highly valued by stakeholder • Sessions are made to many teams throughout IBM China Development Lab * I hide the name of the client here. It is one of the biggest banks in China.

38. The Status • Improvements (to do) • Now designing a GUI tool for PACAS based on Eclipse to enable ‘drag and drop’ to generate test cases • A new version of PACAS v3, planning to rewrite some of the code of the core transformation engine in C++ to accelerate the running. • Support ‘pure’ XML format. Make actions also XML elements, namely ‘<action>’! • Support repository retrieving. To retrieve data and object map files from a remote repository.

39. More… • ATM as a standalone application within PACAS system • ATM stands for Automated Test Modeling • It tries to translate test scripts and related resources under one tool into another • It is an innovative project and we are in halfway of the prototyping • As designer and team lead, I am leveraging virtual team to help on the project. • Added this ATM tool into PACAS toolpak

40. Reference • Lily Zhao, Automation in Agile Testing (PowerPoint) • Lucy Li, IBM Rational Automated Software Quality Products (PowerPoint) • Carl J. Nagle, ‘Data Drive Test Automation Frameworks’ • ‘STAF (Software Testing Automation Framework)’ project, SourceForge • ‘STAX (Xml support for Software Testing Automation Framework)’ project, SourceForge • ‘SAFS (Software Automation Framework Support)’ project, SourceForge • Documentation on AOP, ‘Spring’ project, SourceForge • Documentation and source code for ‘Tcl/Tk’ project, SourceForge • Lloyd H. Nakatani & Mark A. Jones, ‘Jargons and Info centrism’ • Thomas Crutcher Dunnavant Jr., ‘A Collection of Commodity XML Techniques For DSL Tool Development', University of Alabama • ‘Tcl-Java’ project, SourceForge • Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, ‘Design Patterns’ • ‘JavaComilerCompiler’ project, SourceForge • IBM, ‘iTcl Framework’ and corresponding courses • ‘Eli-project’, SourceForge • ‘Stratego/XT’ project, www.stratego-language.org • Selenium project • Watr, Rubyforge • Using XML and XSL for code generation， DeveloperWorks. • XML Path Language, http://www.w3.org/TR/xpath, W3C Recommendation • XML Transformations (XSLT), http://www.w3.org/TR/xslt, W3C Recommendation • ANTLR Project, ANTLR v3 Reference • Huawei Tech., ATF Guide • Summary of IEEE Std 829 • Harold Abelson and Gerald Jay Sussman with Julie Sussman, Structure.And.Interpretation.Of.Computer.Programs,.2nd.Edition • IEEE Std 829-1998 test • Martin Nally (IBM Fellow), Rational Technical Strategy (PowerPoint)