Helping customers write acceptance tests making it easy
Download
1 / 35

Helping Customers Write Acceptance Tests - Making It Easy - PowerPoint PPT Presentation


  • 142 Views
  • Uploaded on

Helping Customers Write Acceptance Tests - Making It Easy. Jeremy Stell-Smith jeremy@thoughtworks.com. Agenda. Why Test? Types of Tests Evolving a Framework On a Web Service Project On a J2EE Project with a Swing Client Lessons Learned Other Frameworks FIT Marathon Let’s Try It Out….

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Helping Customers Write Acceptance Tests - Making It Easy' - ernst


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Helping customers write acceptance tests making it easy l.jpg

Helping Customers Write Acceptance Tests - Making It Easy

Jeremy Stell-Smith

jeremy@thoughtworks.com


Agenda l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…


Agenda3 l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…


Why test l.jpg
Why Test?

  • What’s the point?

  • To make change SAFE

  • To lead to a flat cost of change curve

  • As such, testing is the basis of XP


The cost of change l.jpg

eXtreme

Programming

The Cost of Change

Traditional

Development

Cost

Time


Test time checking l.jpg
“Test-Time” checking

  • Why do we like type safety?

  • As code changes, the compiler catches errors, and these errors are easy to fix

  • But, compile-time checking can’t check everything

    • There are runtime errors, and worse, bugs

  • Tests CAN guard against these

  • If it compiles, there are no compile-time errors

  • If the tests pass, there are no errors period


Types of tests l.jpg
Types of Tests

  • Fine-grained vs. coarse-grained tests

  • Developer vs. customer tests

  • Fine-grained, developer tests (unit tests) are the ideal

    • Fast, come the closest to 100% coverage

    • Low duplication -> Low cost of change, low coupling

  • Use coarse-grained tests to protect yourself against holes in your bricks

  • Use customer tests to protect yourself from building the wrong thing right


An acceptance test is a customer test l.jpg
An Acceptance Test is a Customer Test

  • Does not have to be coarse

  • Get your customers to write tests and take ownership

  • Must be in the language of the customer

  • Your “customer” may be an analyst or QA

  • Acceptance tests are not specific to XP


Agenda9 l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…


Web service project l.jpg
Web Service Project

  • XML in / XML out

  • Stories are in terms of XML API

  • …so tests should be in terms of XML


Example story l.jpg
Example Story

  • Gate In Container Story

    • I would like record a container as Gated In to a facility by submitting this message.

    • The syntax should be:

      <gateInContainer time='2001/04/04 17:00’ container=‘123’>

      <facility code='us.oak.cfs12.testgap.com'/>

      <transportVehicle companyCode='bvl.com'

      code='A1234'/>

      </gateInContainer>


Our framework l.jpg
Our Framework

  • Simplest thing that could possibly work…

  • Built on top of Junit

    • Got UI for free

  • In the “language” of our analysts & customers

  • Setup / Teardown of Data


Example test l.jpg
Example Test

<testCase>

<name>#3: Container Number was not provided</name>

<input>

<gateInContainer time='2001/04/04 17:00'>

<facility code='us.oak.cfs12.testgap.com'/>

<transportVehicle companyCode='bvl.com'

code='A1234'/>

</gateInContainer>

</input>

<expectedOutput>

<failure>

<exception>Required element container was not found</exception>

</failure>

</expectedOutput>

</testCase>


Things we added to it l.jpg
Things We Added To It

  • Includes, to remove some duplication

  • dbValidation section to check against the db to be a bit more thorough

  • Let’s take a look


Things we did right l.jpg
Things We Did Right

  • Analysts wrote the stories & the tests

    • At the beginning of every iteration, we got both

  • We kept it simple

  • Error handling

  • Analyst owned / developer maintained


Things we did wrong l.jpg
Things We Did Wrong

  • Too much duplication – lot of copy / paste

  • Occasionally EVERY test broke – not good

    • Partly because XML isn’t type-safe


J2ee project with swing client l.jpg
J2EE Project with Swing Client

  • Stories were at the client level

  • Client was untested

  • Client had lots of functionality

  • Big Project – QA Department


First pass l.jpg
First Pass

  • One weekend

  • Simple, simple, simple

  • Used XML to script tests against value objects.

  • Not general, very specific to our app.

  • Gave us coarse-grained developer tests

    • Wrong language

    • Didn’t solicit input


Example test19 l.jpg
Example Test

<test name=‘mytest1’>

<createNewDeal type=‘OTC’ subtype=‘Asian’/>

<setValue name=‘dealHeader.name’ value=‘MyDeal 1’/>

<setValue name=‘dealHeader.date’ value=‘jan 1, 2001’/>

<save/>

<assert name=‘dealHeader.name’ value=‘MyDeal 1’/>

<assert name=‘dealHeader.date’ value=‘2001/01/01’/>

</test>


Second pass l.jpg
Second Pass

  • Now testing against swing components

  • Single threaded

    • Better language, but still difficult

    • Generic, but didn’t handle everything


Third pass l.jpg
Third Pass

  • Significant effort

  • JFCUnit

  • Added a recorder

  • Added an editor

    • modify, run, modify, run, modify, run

    • Had been too hard to debug

  • Added data population

  • Scaling became impossible because it wasn’t tested…

  • Language – Good

  • Ease of use


Example test22 l.jpg
Example Test

<test name=“foobar”>

<window name=“Calculator Example”>

<click name=“5”/>

<click name=“1”/>

<click name=“+”/>

<click name=“3”/>

<click name=“=“/>

<click name=“JTextField”/>

<assert name=“JTextField” text=“54”/>

<click name=“+”/>

<click name=“5”/>

<click name=“=“/>

<assert name=“JTextField” text=“59”/>

</window>

</test>


Forth pass l.jpg
Forth Pass

  • Marathon

  • Uses Jython instead of XML

    • Access to all java code

    • Functions, classes, imports, external libraries, etc.

    • Interpreted

  • Test-First

  • Open source

  • JEdit


Example test24 l.jpg
Example Test

from defaultFixture import *

def test():

window('Calculator Example')

click('5')

click('1')

click('+')

click('3')

click('=')

click('JTextField')

assertText('JTextField', '54')

click('+')

click('5')

click('=')

assertText('JTextField', '59')

close()


Testing strategies l.jpg
Testing Strategies

  • Tests that ran with the build

    • Unit tests

    • Smoke tests : coarse-grained tests that provided safety net for our unit tests - 1/3 of our acceptance tests

  • Cruise Control

  • Nightly build of other tests

  • Developer maintained


Things we did right26 l.jpg
Things We Did Right

  • Once and Only Once

  • Encouraged feedback

  • Bought where we could


Things we did wrong27 l.jpg
Things We Did Wrong

  • Analysts didn’t write the tests

  • We didn’t integrate close enough w/ QA

  • Used them as a substitute for unit tests

  • Disconnect with stories


Agenda28 l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…


Lessons learned l.jpg
Lessons Learned

  • In terms of stories

  • Acceptance tests are NOT a substitute for unit tests

  • Buy don’t build

    • Junit, Jython, etc

  • Iterative

  • Make it modular – once and only once

  • Language of Analysts

    • Focus on readability == maintainability

    • Don’t EVER want to re-record tests


Lessons learned cont d l.jpg
Lessons Learned (cont’d)

  • Yes you have to test your testing framework…

  • Use a scripting language to write scripts

  • User friendly

  • Pair with your customers to write tests

  • Developers maintained

  • Error handling

  • Needs to be Customizable -> Open Source?


Things to watch out for l.jpg
Things To Watch Out For

  • How to deploy

  • When to run

  • What to do when they break

  • Are tests independent?

  • Data?

  • Acceptance tests are not all passing?

  • Long running tests


Agenda32 l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…


Slide33 l.jpg
FIT

  • http://fit.c2.com/

  • Open source general purpose functional testing tool

  • Totally generic, and customizable

  • Allows fine-grained testing

  • Remember, customer tests are not necessarily end to end tests

  • If you want to GUI level tests, I wouldn’t use FIT

  • Ideal for test first stuff


Marathon gui test runner l.jpg
Marathon – GUI Test Runner

  • http://marathonman.sourceforge.net/

  • More suited for testing GUIs

    • Recorder & editor

  • Can’t use a recorder for Test First tests

    • But you can use it help maintain them

  • Modular without leaving script


Agenda35 l.jpg
Agenda

  • Why Test?

    • Types of Tests

  • Evolving a Framework

    • On a Web Service Project

    • On a J2EE Project with a Swing Client

  • Lessons Learned

  • Other Frameworks

    • FIT

    • Marathon

  • Let’s Try It Out…