…b uilding your own AT f ramework … 21 November, 2003 Se a n Hanly shanly@exoftware

1. …building your own AT framework…21 November, 2003Sean Hanlyshanly@exoftware.com

2. Acceptance Tests… anything else… • - Story of a “Story” • AT Demonstration • Pattern • Anti-Patterns • AT Framework Patterns • Process Patterns • Interactive Session

3. The Story of a "Story”

4. XP Process Overview Project Stories Years Release Stories Months Iteration Story Story Story Story Weeks Tasks Implementation Days Test Cases

5. this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Release Planning customer + developers + tester + interaction designer V(elocity) = 20

6. this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement 1 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Bucket 2 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement 6 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Release Plan 5 5 5 5 … 5 5 5 5 5 5 5 5 5 5 5 5

7. 1 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Iteration Planning customer + developers + tester + interaction designer AutomatedAcceptanceTests + 5

8. this is a short description of a customer’s requirement Test Refactor Code Implementation Planning failing acceptancetest acceptance tests 5 passing acceptancetest

9. The Story

10. customer wants the ability to have a rules engine that allows rules to be evalauted against a clients financial position • A good user story: • IIndependent  • NNegotiable • VValuable / Vertical • EEstimable  • SSmall  • TTestable Initial Story Presentation(Release Plan)

11. customer wants the ability to • analyse the amount of cash • a client is currently holding • too little • too much • just right • (depends on lifestyle cost and attitude to risk) • A good user story: • IIndependent  • NNegotiable • VValuable / Vertical  • EEstimable  • SSmall  • TTestable New Story 25

12. 2 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement 1 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement customer wants the ability to analyse the amount of cash a client is currently holding this is a short description of a customer’s requirement this is a short description of a customer’s requirement 6 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Bucket this is a short description of a customer’s requirement Release Plan 15 15 10 15 … 15 25 25 25 25 10 15 5 10 15 15 10 V(elocity) = 65

13. this is a short description of a customer’s requirement this is a short description of a customer’s requirement 2 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement 1 this is a short description of a customer’s requirement customer wants the ability to analyse the amount of cash a client is currently holding Bucket this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description. of a customer’s reuqirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement 6 this is a short description of a customer’s requirement this is a short description of a customer’s requirement this is a short description of a customer’s requirement Release Plan 15 15 10 15 15 25 25 25 25 10 15 5 10 15 15 10 V(elocity) = 65

14. + • customer wants the ability to • analyse the amount of cash • a client is currently holding • too little • too much • just right • (depends on lifestyle cost and attitude to risk) screen design + 25 interaction tests + engine tests Iteration Two

15. Screen Design

16. Interaction Tests (example) //Overview “Analyses a clients cash balance to see if they are holding too much cash given their attitude to risk and lifestyle cost…” //setup client data UserClicksMainMenu MenuFinancialObjectives UserInputsText FinancialObjectivesAttitudeToRisk "3 - Low Return - Long " "Term Investment" UserClicksMainMenu MenuCurrentBalanceSheet UserInputsText CurrentBalanceSheetTotalCash 30000 UserClicksMainMenu MenuFinancialObjectives UserInputsText FinancialObjectivesLifestyleCost 25000 //cash rule TestValueOfText AnalyseObservation "Given your attitude to risk, you " "should maintain a cash balance no " "greater than €12,500." TestValueOfText AnalyseRecommendation "Consider moving €17,500 from cash " "accounts to investable assets." TestValueOfText AnalyseDestination "Go to Invest Capital Sum and move " "excess cash to cash deposit unless " "holding cash to buy an asset." //hyperlink UserClicksControl AnalyseDestination TestValueOfLabel WorkAreaTitle "Invest Capital Sum"

17. Engine Test (example)

18. Demonstration

19. Automated Acceptance Tests • Unambiguous, Transparent • Controls Scope Creep • No Assumptions • “When do you stop?” • Surfaces and defines requirements • Executable Requirements Documentation • Executable – Asset, Regression Suite • Test Infecting your Customer

21. AT Mini-Patterns

22. Anti-Patterns

23. LunaticsRunningTheAsylum • Problem • The developers create an AT framework that suits the way they think and the technologies they like or are currently in vogue. As an example we delivered a XML, Cocoon, Java Reflection based framework to our client!!! • Solution • Eat our own dogfood, i.e. listen to your customer whether they be a business user, tester, interaction designer. They know what they want to achieve and how they would like it to work, not you! • Do give them alternatives and possibilites. They may not always be aware of what is possible.

24. what happens when the lunatics run the asylum!!

25. RecordWhatAndPlaybackWhat • Problem • Trying to use a GUI based record and playback tool means that • AT’s cannot be created upfront and • where they are, they the tend to be extremely fragile due to change i.e. constant re-recording and consequent frustration. • Solution • Create an AT framework that allows upfront executable requirements definition e.g. ACT and FIT. • However, record and playback can give you a very cheap regression suite for legacy code bases.

26. MetaLanguageMadness(JAct, Tython ) • Problem • In the effort to create an AT frameowrk the “lunatics” invent a whole new programming language. This results in un-necessary AT complexity for the customer and the temptation to overload the tests with actual logic, breaking the DRY principle. • Solution • Remember who the framework will be used by and what the goal of the framework is. • Create a definition language not a programming language.

27. Other anti-patterns • Test the system not the OS i.e. figure out the boundaries • AllOrNothing – 80% is ok • Not being subtle enough in setup and teardown • Using a copy of the complete database • Testers/Customers will not write code • Allow for some verbosity e.g. FIT • Overly generic e.g. a GUI scripting language

28. Patterns

29. CreateADomainLanguage(AddACommandLine) • Problem • When creating an AT framework it is difficult to see how to expose the system to testing in a manner that allows for definition of requirements and ease of testing. Here intimacy and testability always seem to be the problem. Additionally the framework has to talk to the customer/tester and reflect the domain. Importantly this often means allowing for scripting of actual recognizable usage scenarios. • Solution • Create a simple verb based, (similar to OS commands), scripting language for the application • Create high level domain verbs, e.g. CreateCustomer, not low level calls, e.g. CallSessionBean “jndi::/Customer” createCustomer • Hook directly into the environment • Verbs should act as client/actor to all parts of the system and be conversational in nature

30. DomainLanguageInsights • Improves Metaphor • In the beginning there was the command line – intimacy • Reusable • Improves Design • Becomes a way of interacting with the system • Leverages existing test code

31. Example do batch work SetPretendTime “2003,7,24,13,5,6” RunBatchJob //check files, database and account balances CompareFiles expected.out actual.out TestCustomerRecordEquals “1,10,…” TestCustomerRecordEquals “2,10,…” TestCustomerRecordEquals “3,10,…” CheckPhoneBalance 353879793001 1000 CheckPhoneBalance 353879793002 1000 CheckPhoneBalance 353879793003 0985 //setup SetPretendTime “2003,7,23,13,5,6” DeleteAllRecordsInTheDatabase SetPhoneBalance 353879793001 1000 SetPhoneBalance 353879793002 1000 SetPhoneBalance 353879793003 1000 //do work SubmitUrl “http://AddNickName…” SubmitUrl “http://AddNickName…” SubmitUrl “http://UpLoadHighScore…” //check db TestNumberOfRecordsInDatabase 3 TestCustomerRecordEquals “1,8,…” TestCustomerRecordEquals “2,8,…” TestCustomerRecordEquals “3,8,…”

32. BuildInTestability • Problem • The system in question is difficulut to test because there are no testing hook’s available. • Solution • When desiging/building the system add testability. Easiest done when driving the actual development from acceptance tests. Examples include: • Recording API’s • Query API’s • Determinism • Fakeability • Undoability • State Management

33. Legacy: Characterize It • Problem • Before adding to or refactoring we need a set of acceptance tests that provide enough feedback to ensure correctness. However, the effort to bring the systen under unit test / acceptance test is prohibitive. • Solution • Create a coarse grained set of characterization tests that create an invariant that immdeitely gives feedback as the system as it is changed e.g. • Log File Comparison • Database Comparison • Record and Playback

34. Legacy: AddTracerBullets • Problem • While adding acceptance tests to Legacy Code you find that there are no testable post conditions. • Solution • Add safe tracer bullets, i.e. not behaviour changing, to the code base. As an example add log statements to the major entry and exit points to your system.

35. Other mini patterns • TestsAreSource • BringDataUnderControl • IntegrateWithBuild • It’s all right to fake it • Trust your code • Script in the implementation environment • Add determinism • Allow for some verbosity e.g. FIT • Add baseline capability

36. Process Patterns

37. Process Patterns • TestsAreTheRequirements • TestsAreConversationPieces • AddSpanners • EdgeCaseObsession • HappyPath

38. Framework Round Up

39. Frameworks • Data • FIT • CSV / Excel Driven • Script • GUI based tools • ACT • Record Playback • nuff said! • …

40. ACT Demo

41. allow for the creation of a new customer account (open balance should be zero) allow a customer to deposit money into an account allow a customer to withdraw money from an account Bank System(stories to date)

42. integrate deposit functionality with a fraud checking system Bank System(new story)