helping customers write acceptance tests making it easy
Download
Skip this Video
Download Presentation
Helping Customers Write Acceptance Tests - Making It Easy

Loading in 2 Seconds...

play fullscreen
1 / 35

Helping Customers Write Acceptance Tests - Making It Easy - PowerPoint PPT Presentation


  • 140 Views
  • Uploaded on

Helping Customers Write Acceptance Tests - Making It Easy. Jeremy Stell-Smith [email protected] Agenda. Why Test? Types of Tests Evolving a Framework On a Web Service Project On a J2EE Project with a Swing Client Lessons Learned Other Frameworks FIT Marathon Let’s Try It Out….

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Helping Customers Write Acceptance Tests - Making It Easy' - ernst


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
agenda
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
agenda3
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
why test
Why Test?
  • What’s the point?
  • To make change SAFE
  • To lead to a flat cost of change curve
  • As such, testing is the basis of XP
the cost of change

eXtreme

Programming

The Cost of Change

Traditional

Development

Cost

Time

test time checking
“Test-Time” checking
  • Why do we like type safety?
  • As code changes, the compiler catches errors, and these errors are easy to fix
  • But, compile-time checking can’t check everything
    • There are runtime errors, and worse, bugs
  • Tests CAN guard against these
  • If it compiles, there are no compile-time errors
  • If the tests pass, there are no errors period
types of tests
Types of Tests
  • Fine-grained vs. coarse-grained tests
  • Developer vs. customer tests
  • Fine-grained, developer tests (unit tests) are the ideal
    • Fast, come the closest to 100% coverage
    • Low duplication -> Low cost of change, low coupling
  • Use coarse-grained tests to protect yourself against holes in your bricks
  • Use customer tests to protect yourself from building the wrong thing right
an acceptance test is a customer test
An Acceptance Test is a Customer Test
  • Does not have to be coarse
  • Get your customers to write tests and take ownership
  • Must be in the language of the customer
  • Your “customer” may be an analyst or QA
  • Acceptance tests are not specific to XP
agenda9
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
web service project
Web Service Project
  • XML in / XML out
  • Stories are in terms of XML API
  • …so tests should be in terms of XML
example story
Example Story
  • Gate In Container Story
    • I would like record a container as Gated In to a facility by submitting this message.
    • The syntax should be:

<gateInContainer time=\'2001/04/04 17:00’ container=‘123’>

<facility code=\'us.oak.cfs12.testgap.com\'/>

<transportVehicle companyCode=\'bvl.com\'

code=\'A1234\'/>

</gateInContainer>

our framework
Our Framework
  • Simplest thing that could possibly work…
  • Built on top of Junit
    • Got UI for free
  • In the “language” of our analysts & customers
  • Setup / Teardown of Data
example test
Example Test

<testCase>

<name>#3: Container Number was not provided</name>

<input>

<gateInContainer time=\'2001/04/04 17:00\'>

<facility code=\'us.oak.cfs12.testgap.com\'/>

<transportVehicle companyCode=\'bvl.com\'

code=\'A1234\'/>

</gateInContainer>

</input>

<expectedOutput>

<failure>

<exception>Required element container was not found</exception>

</failure>

</expectedOutput>

</testCase>

things we added to it
Things We Added To It
  • Includes, to remove some duplication
  • dbValidation section to check against the db to be a bit more thorough
  • Let’s take a look
things we did right
Things We Did Right
  • Analysts wrote the stories & the tests
    • At the beginning of every iteration, we got both
  • We kept it simple
  • Error handling
  • Analyst owned / developer maintained
things we did wrong
Things We Did Wrong
  • Too much duplication – lot of copy / paste
  • Occasionally EVERY test broke – not good
    • Partly because XML isn’t type-safe
j2ee project with swing client
J2EE Project with Swing Client
  • Stories were at the client level
  • Client was untested
  • Client had lots of functionality
  • Big Project – QA Department
first pass
First Pass
  • One weekend
  • Simple, simple, simple
  • Used XML to script tests against value objects.
  • Not general, very specific to our app.
  • Gave us coarse-grained developer tests
    • Wrong language
    • Didn’t solicit input
example test19
Example Test

<test name=‘mytest1’>

<createNewDeal type=‘OTC’ subtype=‘Asian’/>

<setValue name=‘dealHeader.name’ value=‘MyDeal 1’/>

<setValue name=‘dealHeader.date’ value=‘jan 1, 2001’/>

<save/>

<assert name=‘dealHeader.name’ value=‘MyDeal 1’/>

<assert name=‘dealHeader.date’ value=‘2001/01/01’/>

</test>

second pass
Second Pass
  • Now testing against swing components
  • Single threaded
    • Better language, but still difficult
    • Generic, but didn’t handle everything
third pass
Third Pass
  • Significant effort
  • JFCUnit
  • Added a recorder
  • Added an editor
    • modify, run, modify, run, modify, run
    • Had been too hard to debug
  • Added data population
  • Scaling became impossible because it wasn’t tested…
  • Language – Good
  • Ease of use
example test22
Example Test

<test name=“foobar”>

<window name=“Calculator Example”>

<click name=“5”/>

<click name=“1”/>

<click name=“+”/>

<click name=“3”/>

<click name=“=“/>

<click name=“JTextField”/>

<assert name=“JTextField” text=“54”/>

<click name=“+”/>

<click name=“5”/>

<click name=“=“/>

<assert name=“JTextField” text=“59”/>

</window>

</test>

forth pass
Forth Pass
  • Marathon
  • Uses Jython instead of XML
    • Access to all java code
    • Functions, classes, imports, external libraries, etc.
    • Interpreted
  • Test-First
  • Open source
  • JEdit
example test24
Example Test

from defaultFixture import *

def test():

window(\'Calculator Example\')

click(\'5\')

click(\'1\')

click(\'+\')

click(\'3\')

click(\'=\')

click(\'JTextField\')

assertText(\'JTextField\', \'54\')

click(\'+\')

click(\'5\')

click(\'=\')

assertText(\'JTextField\', \'59\')

close()

testing strategies
Testing Strategies
  • Tests that ran with the build
    • Unit tests
    • Smoke tests : coarse-grained tests that provided safety net for our unit tests - 1/3 of our acceptance tests
  • Cruise Control
  • Nightly build of other tests
  • Developer maintained
things we did right26
Things We Did Right
  • Once and Only Once
  • Encouraged feedback
  • Bought where we could
things we did wrong27
Things We Did Wrong
  • Analysts didn’t write the tests
  • We didn’t integrate close enough w/ QA
  • Used them as a substitute for unit tests
  • Disconnect with stories
agenda28
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
lessons learned
Lessons Learned
  • In terms of stories
  • Acceptance tests are NOT a substitute for unit tests
  • Buy don’t build
    • Junit, Jython, etc
  • Iterative
  • Make it modular – once and only once
  • Language of Analysts
    • Focus on readability == maintainability
    • Don’t EVER want to re-record tests
lessons learned cont d
Lessons Learned (cont’d)
  • Yes you have to test your testing framework…
  • Use a scripting language to write scripts
  • User friendly
  • Pair with your customers to write tests
  • Developers maintained
  • Error handling
  • Needs to be Customizable -> Open Source?
things to watch out for
Things To Watch Out For
  • How to deploy
  • When to run
  • What to do when they break
  • Are tests independent?
  • Data?
  • Acceptance tests are not all passing?
  • Long running tests
agenda32
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
slide33
FIT
  • http://fit.c2.com/
  • Open source general purpose functional testing tool
  • Totally generic, and customizable
  • Allows fine-grained testing
  • Remember, customer tests are not necessarily end to end tests
  • If you want to GUI level tests, I wouldn’t use FIT
  • Ideal for test first stuff
marathon gui test runner
Marathon – GUI Test Runner
  • http://marathonman.sourceforge.net/
  • More suited for testing GUIs
    • Recorder & editor
  • Can’t use a recorder for Test First tests
    • But you can use it help maintain them
  • Modular without leaving script
agenda35
Agenda
  • Why Test?
    • Types of Tests
  • Evolving a Framework
    • On a Web Service Project
    • On a J2EE Project with a Swing Client
  • Lessons Learned
  • Other Frameworks
    • FIT
    • Marathon
  • Let’s Try It Out…
ad