1 / 35

Helping Customers Write Acceptance Tests - Making It Easy

Helping Customers Write Acceptance Tests - Making It Easy. Jeremy Stell-Smith jeremy@thoughtworks.com. Agenda. Why Test? Types of Tests Evolving a Framework On a Web Service Project On a J2EE Project with a Swing Client Lessons Learned Other Frameworks FIT Marathon Let’s Try It Out….

ernst
Download Presentation

Helping Customers Write Acceptance Tests - Making It Easy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Helping Customers Write Acceptance Tests - Making It Easy Jeremy Stell-Smith jeremy@thoughtworks.com

  2. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

  3. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

  4. Why Test? • What’s the point? • To make change SAFE • To lead to a flat cost of change curve • As such, testing is the basis of XP

  5. eXtreme Programming The Cost of Change Traditional Development Cost Time

  6. “Test-Time” checking • Why do we like type safety? • As code changes, the compiler catches errors, and these errors are easy to fix • But, compile-time checking can’t check everything • There are runtime errors, and worse, bugs • Tests CAN guard against these • If it compiles, there are no compile-time errors • If the tests pass, there are no errors period

  7. Types of Tests • Fine-grained vs. coarse-grained tests • Developer vs. customer tests • Fine-grained, developer tests (unit tests) are the ideal • Fast, come the closest to 100% coverage • Low duplication -> Low cost of change, low coupling • Use coarse-grained tests to protect yourself against holes in your bricks • Use customer tests to protect yourself from building the wrong thing right

  8. An Acceptance Test is a Customer Test • Does not have to be coarse • Get your customers to write tests and take ownership • Must be in the language of the customer • Your “customer” may be an analyst or QA • Acceptance tests are not specific to XP

  9. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

  10. Web Service Project • XML in / XML out • Stories are in terms of XML API • …so tests should be in terms of XML

  11. Example Story • Gate In Container Story • I would like record a container as Gated In to a facility by submitting this message. • The syntax should be: <gateInContainer time='2001/04/04 17:00’ container=‘123’> <facility code='us.oak.cfs12.testgap.com'/> <transportVehicle companyCode='bvl.com' code='A1234'/> </gateInContainer>

  12. Our Framework • Simplest thing that could possibly work… • Built on top of Junit • Got UI for free • In the “language” of our analysts & customers • Setup / Teardown of Data

  13. Example Test <testCase> <name>#3: Container Number was not provided</name> <input> <gateInContainer time='2001/04/04 17:00'> <facility code='us.oak.cfs12.testgap.com'/> <transportVehicle companyCode='bvl.com' code='A1234'/> </gateInContainer> </input> <expectedOutput> <failure> <exception>Required element container was not found</exception> </failure> </expectedOutput> </testCase>

  14. Things We Added To It • Includes, to remove some duplication • dbValidation section to check against the db to be a bit more thorough • Let’s take a look

  15. Things We Did Right • Analysts wrote the stories & the tests • At the beginning of every iteration, we got both • We kept it simple • Error handling • Analyst owned / developer maintained

  16. Things We Did Wrong • Too much duplication – lot of copy / paste • Occasionally EVERY test broke – not good • Partly because XML isn’t type-safe

  17. J2EE Project with Swing Client • Stories were at the client level • Client was untested • Client had lots of functionality • Big Project – QA Department

  18. First Pass • One weekend • Simple, simple, simple • Used XML to script tests against value objects. • Not general, very specific to our app. • Gave us coarse-grained developer tests • Wrong language • Didn’t solicit input

  19. Example Test <test name=‘mytest1’> <createNewDeal type=‘OTC’ subtype=‘Asian’/> <setValue name=‘dealHeader.name’ value=‘MyDeal 1’/> <setValue name=‘dealHeader.date’ value=‘jan 1, 2001’/> … <save/> <assert name=‘dealHeader.name’ value=‘MyDeal 1’/> <assert name=‘dealHeader.date’ value=‘2001/01/01’/> </test>

  20. Second Pass • Now testing against swing components • Single threaded • Better language, but still difficult • Generic, but didn’t handle everything

  21. Third Pass • Significant effort • JFCUnit • Added a recorder • Added an editor • modify, run, modify, run, modify, run • Had been too hard to debug • Added data population • Scaling became impossible because it wasn’t tested… • Language – Good • Ease of use

  22. Example Test <test name=“foobar”> <window name=“Calculator Example”> <click name=“5”/> <click name=“1”/> <click name=“+”/> <click name=“3”/> <click name=“=“/> <click name=“JTextField”/> <assert name=“JTextField” text=“54”/> <click name=“+”/> <click name=“5”/> <click name=“=“/> <assert name=“JTextField” text=“59”/> </window> </test>

  23. Forth Pass • Marathon • Uses Jython instead of XML • Access to all java code • Functions, classes, imports, external libraries, etc. • Interpreted • Test-First • Open source • JEdit

  24. Example Test from defaultFixture import * def test(): window('Calculator Example') click('5') click('1') click('+') click('3') click('=') click('JTextField') assertText('JTextField', '54') click('+') click('5') click('=') assertText('JTextField', '59') close()

  25. Testing Strategies • Tests that ran with the build • Unit tests • Smoke tests : coarse-grained tests that provided safety net for our unit tests - 1/3 of our acceptance tests • Cruise Control • Nightly build of other tests • Developer maintained

  26. Things We Did Right • Once and Only Once • Encouraged feedback • Bought where we could

  27. Things We Did Wrong • Analysts didn’t write the tests • We didn’t integrate close enough w/ QA • Used them as a substitute for unit tests • Disconnect with stories

  28. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

  29. Lessons Learned • In terms of stories • Acceptance tests are NOT a substitute for unit tests • Buy don’t build • Junit, Jython, etc • Iterative • Make it modular – once and only once • Language of Analysts • Focus on readability == maintainability • Don’t EVER want to re-record tests

  30. Lessons Learned (cont’d) • Yes you have to test your testing framework… • Use a scripting language to write scripts • User friendly • Pair with your customers to write tests • Developers maintained • Error handling • Needs to be Customizable -> Open Source?

  31. Things To Watch Out For • How to deploy • When to run • What to do when they break • Are tests independent? • Data? • Acceptance tests are not all passing? • Long running tests

  32. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

  33. FIT • http://fit.c2.com/ • Open source general purpose functional testing tool • Totally generic, and customizable • Allows fine-grained testing • Remember, customer tests are not necessarily end to end tests • If you want to GUI level tests, I wouldn’t use FIT • Ideal for test first stuff

  34. Marathon – GUI Test Runner • http://marathonman.sourceforge.net/ • More suited for testing GUIs • Recorder & editor • Can’t use a recorder for Test First tests • But you can use it help maintain them • Modular without leaving script

  35. Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J2EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out…

More Related