Testing on Replay. During the replay of events, there are various output checking possibilities:Manual: user has to watch the system for anomaliesComplete: all outputs were recorded during capture, and system must reproduce them exactly".Exactly": to the level of detail of the recording.Che
1. Capture and Replay Often used for regression test development
Tool used to capture interactions with the system under test.
Inputs must be captured; outputs may also be recorded and (possibly) checked.
Telelogic Tau: save command history in simulator
GUI testing tools
Capture requires a working system to be available already!
2. Testing on Replay During the replay of events, there are various output checking possibilities:
Manual: user has to watch the system for anomalies
Complete: all outputs were recorded during capture, and system must reproduce them “exactly”.
“Exactly”: to the level of detail of the recording.
Checkpoints: system output is only checked at certain points for specified values.
These could be inserted manually or automatically.
3. The capture record Inputs, outputs, and other information needed to reproduce a session with the system under test need to be recorded during the capture process.
General information: date/time of recording, etc.
System start-up information
Events from test tool to system
Point of control, event
Events from system to test tool
Checkpoints / expected outputs
4. Event-driven architecture Basis of GUI frameworks
Various controls create events when they are created, activated, modified, deactivated, or disposed.
Input devices create events as per their functions: key pressed, key released, mouse moved, …
Events are sent to an event dispatcher
“Listeners” register with the event dispatcher to receive events
Listeners specify the type of events in which they are interested, and which method should be called when the event occurs
When an event is received by the event dispatcher, notifications are sent to each registered listener for the type of event.
Each listener has their event notification method called in turn.
5. Event-driven architecture
6. Integrating a Capture and Replay tool During the capture process, the tool will register as an event listener
Event notification method for the tool will record the details of all events that occurred.
During the replay process, the tool will register as an event source (possibly also as a listener)
For mouse and keyboard events, the tool has to substitute for the actual devices as the event source.
Replay events should be initiated at the same relative time as during the capture.
Other controls issue events as usual (e.g. GUI button deactivated)
9. Integration Alternatives Use the UI event framework associated with an environment:
Example: Swing, SWT, etc.
Create native system events
Example: The Robot class in Java will actually move the mouse cursor on the underlying platform, instead of just recreating the Java framework event that would be generated.
Analyze screen images
A screen capture of a tool’s GUI can be stored while the tool is running, analyzed for widgets, and then used for replay.
10. The Java Robot Class Class to send requests to native platform to generate system events.
Keyboard: keyPress, keyRelease events
Mouse: mouseMove, mousePress, mouseRelease, mouseWheel events
Screen: request screen capture
Timing: delay specified duration
11. Tests Alternatives:
During the capture process, the tool may record events, and then compare the results during playback.
At test checkpoints, the tool can ask various controls for their state and record it.
Contents of text fields
GUI radio button properties:
enabled / disabled
selected / not selected
12. Integration options Compile tool classes and application together
Similar to JUnit approach
User has to determine object instances that should be monitored, and then register instances of tool classes as event source and listeners
Connections are also set up to query controls as to their states.
Provides better integration, more stable replays
Tester has to learn how to use tool classes
Use virtual machine registration (when there is a VM)
Tool has to “discover” objects of interest
13. Example: Jacareto Java-based open source GUI capture + replay tool
Authors: C. Spannagel et. al.
Web site: http://jacareto.sourceforge.net
Jacareto: event record and playback
Picorder: command line interface
CleverPHL: graphical user interface (for Jacareto)
14. Jacareto Uses virtual machine registration
Batch file and XML script used to start Java virtual machine, start the test tool, and load the application’s classes.
Main method of application called by test tool as required.
Jacareto has to discover all instances of subclasses of Component (including JComponent for Swing)
15. Event records Stored as XML
Type of event: mouse move, key pressed, etc.
Includes event-specific information: X-Y location of mouse cursor, which key was pressed, …
Relative time from previous event
Needed to keep events synchronized
<KeyEvent procTime="0" duration="240" source="SampleDialog_(1).JRootPane_(1).JLayeredPane_(1).JPanel_(1).JTextField_(1)" class="javax.swing.JTextField"uuid="89a95465-8cae-467b-bc48-6422a63bcfc4" ID="401" component="null" root="SampleDialog_(1)" xPos="0" yPos="0" width="0" height="0" when="1175219983088" isConsumed="false">
<KeyInfo keyCode="65" keyChar="A" modifiers="1" />
16. XML record displayed for user
17. Tests At various user-defined checkpoints, the state of a control can be tested.
Example: Test if a text field contains the specified text
Also specify that we want to stop if there is an error (i.e. ignore error is false), and that we do not want to correct the state of the control if it does not conform to the record.
<JTextComponentTest component="MainFrame_(1).JRootPane_(1).JLayeredPane_(1).JPanel_(2).JTextField_(1)" isRegExp="false" isCorrecting= "false isEnabled="true" isIgnoring="false" hasFocus="false">Hello, Alan</JTextComponentTest>
18. UI for Test creation
19. Test reports
20. Example: Marathon Capture and playback tool for testing Java/Swing graphical user interfaces.
Uses the “Jython” (Python for Java) scripting language to store test cases.
Test cases can be captured interactively, or written as Jython scripts.
Assertions can be added to check the state of various components.
Uses test runner that is a non-standard version of JUnit.
21. Sample Jython test script useFixture(default)
java_recorded_version = '1.5.0_14'
select('Click Me', 'true')
if window('Enter name'):
assert_p('Label', 'Text', 'Hello, Alan!')
22. Running a Marathon test
23. Test report
24. Abbot Open source project hosted on SourceForge
Does the capture and replay, using an XML file.
Runs test scripts.
Provides for JUnit tests
Interactive test script editor
Uses the Java Robot class to create the replay events
25. Costello script editor
26. An Abbot script capture <?xml version="1.0" encoding="UTF-8"?>
<component class="javax.swing.JToggleButton" id="Click Me" index="1" label="" parent="JPanel Instance" text="Click Me" window="Greetings" />
<component class="guidemo.EnterNameDialog" id="Enter name" parent="Greetings" title="Enter name" />
<component class="guidemo.MainFrame" id="Greetings" root="true" title="Greetings" />
<component class="javax.swing.JLabel" id="Hello, Alan!" index="0" parent="JPanel Instance" text="Hello, Alan!" window="Greetings" />
<component class="javax.swing.JLayeredPane" id="JLayeredPane Instance" index="1" parent="JRootPane Instance" window="Greetings" />
<component class="javax.swing.JLayeredPane" id="JLayeredPane Instance 2" index="1" parent="JRootPane Instance 2" window="Enter name" />
<component class="javax.swing.JPanel" id="JPanel Instance" index="0" parent="JLayeredPane Instance" window="Greetings" />
<component class="javax.swing.JPanel" id="JPanel Instance 2" index="0" parent="JLayeredPane Instance 2" window="Enter name" />
<component class="javax.swing.JRootPane" id="JRootPane Instance" index="0" parent="Greetings" />
<component class="javax.swing.JRootPane" id="JRootPane Instance 2" index="0" parent="Enter name" />
<component class="javax.swing.JTextField" id="JTextField Instance" index="3" parent="JPanel Instance 2" window="Enter name" />
<component class="javax.swing.JButton" id="OK" index="0" parent="JPanel Instance 2" text="OK" window="Enter name" />
<launch args="" class="guidemo.Main" classpath=".;C:\Documents and Settings\Alan Williams\workspace\GUI Demo\bin;C:\Documents and Settings\Alan Williams\workspace\GUI Demo\bin\swing-layout-1.0.3.jar" method="main" />
<wait args="Greetings" class="abbot.tester.ComponentTester" method="assertComponentShowing" />
<action args="Click Me" class="javax.swing.AbstractButton" method="actionClick" />
<wait args="Enter name" class="abbot.tester.ComponentTester" method="assertComponentShowing" />
<action args="JTextField Instance,0" class="javax.swing.text.JTextComponent" method="actionClick" />
<event component="JTextField Instance" kind="MOUSE_RELEASED" type="MouseEvent" x="62" y="9" />
<action args="JTextField Instance,Alan" method="actionKeyString" />
<action args="OK" class="javax.swing.AbstractButton" method="actionClick" />
<wait args="Enter name" class="abbot.tester.ComponentTester" invert="true" method="assertComponentShowing" />
<assert component="Hello, Alan!" method="getText" value="Hello, Alan!" />
27. Replay options The script created by Abbot/Costello can be replayed in two different manners:
From the Costello window, interactively.
The script can be converted on-the-fly to a JUnit test, and run from JUnit.
Appears to be compatible with JUnit 3 only.
28. Run an Abbot script with JUnit public class AbbotScriptTest extends ScriptFixture
public AbbotScriptTest( String filename )
super( filename );
public static Test suite( )
Test theSuite = new ScriptTestSuite( AbbotScriptTest.class,
// In folder described above, this selects scripts to include
public boolean accept( File file )
String name = file.getName( );
return name.endsWith( ".xml" );
29. Use Abbot to create JUnit GUI tests Abbot also provides a class library that provides the following functions for creating GUI tests.
Component tester: provides capability of
Sending GUI events: click, drag, drop, show window, keyboard input, menu item selection, menu popup
Assertions: window or frame is showing
Timing: wait for window or frame to show
There are specialized testers for specific components.
Finder: Obtain references to GUI widgets, based on class, properties, etc.
30. Finders The finders allow the capability of obtaining references to components only from having seen the visual layout of a GUI.
Example: find a button that has the label “Click Me”, that is known to be on a frame created by a class MainFrame.
The finder can return a reference to the button…
…without knowing the actual variable name of the button within MainFrame…
…even if the variable is private…
…as long as the button can be identified from properties.
31. Example: Create a test case that:
Locates a button that says “Click Me”.
Clicks the button.
Obtains the dialog that appears when a button is clicked.
Locates a text entry field in the dialog.
Enters some text into the field.
Locates a button labelled “OK”.
Clicks the OK button.
Locates a label on the original frame.
Checks that the label was updated as a result of entering the text.
32. Sample of a component finder: public class MainFrameTest extends ComponentTestFixture
JButton okButton = ( JButton ) this.getFinder( )
.find( enterNameDialog, // component containing button
new Matcher( )
public boolean matches( Component c )
return c instanceof JButton &&
((JButton)c).getText( ).equals( "OK" );
assertNotNull( "okButton is null", okButton );
33. Performing GUI actions public class MainFrameTest extends ComponentTestFixture
tester = new ComponentTester( );
final MainFrame mainFrame = new MainFrame( );
this.showWindow( mainFrame );
JLabel testLabel = // use finder on mainFrame
JButton clickMeButton = // use finder on mainFrame
tester.actionClick( clickMeButton );
EnterNameDialog dialog = // use finder on mainFrame
JTestField nameEntryField = // use finder on dialog
JButton okMeButton = // use finder on dialog
tester.actionKeyString( nameEntryField, "Alan" );
tester.actionClick( okButton );
String expected = "Hello, Alan!";
String actual = testLabel.getText( );
assertEquals( expected, actual );
34. Running the test The test will be run as usual for a JUnit test.
During the test, the actual GUI will appear, and the Abbot robot class will perform actions on the GUI.
Assertions will be checked from the GUI component properties.
During the test, Abbot will take control of the mouse, and it will be unusable.
Make sure that windows created will be visible, or Abbot won’t be able to find them.
35. Eclipse TPTP Auto GUI recorder Capture and replay of GUI events and actions within the Eclipse IDE.
Test suite is set up as an Eclipse plug-in project that integrates itself with the IDE.
Any application that is also an Eclipse plug-in can be tested with the auto GUI recorder.
The plug-in mechanism exposes how Eclipse can interact with the plug-in, and the auto GUI recorder uses this information to send and receive events to/from the plug-in.
36. Pros and cons Advantages:
Requiring the test application to be an Eclipse plug-in allows for better knowledge of the environment for the record / replay tool.
Less difficulty and more reliability for locating user interface widgets.
Tool can call methods directly, instead of simulating mouse operations and keystrokes.
Cannot be used for stand-alone applications
Requires the use of Eclipse’s Standard Widget Toolkit (SWT), instead of Java’s Abstract Widget Toolkit (AWT) or the Swing toolkit.
37. Additional features Verification hooks
Points at which tests can be inserted to check the state of various widgets.
Allows script to substitute variable values for variable names during run-time.
Example: specification of temporary directory, relative file names
38. TPTP Auto GUI recorder
39. Action script <macro version="1.0">
<shell descriptive="Test - AutomatedGUISuite - Eclipse SDK"
<command descriptive="Project..." type="select" referenceId="1"/>
<shell descriptive="New Project" referenceId="2" return-code="0">
<command descriptive="Java Project“
<command type="focus" referenceId="3"/>
<command descriptive="Next >"
<command type="modify" referenceId="6">
<![CDATA[A Java Project]]>
<command descriptive="Next >"
<command descriptive="Finish" type="select" referenceId="7"/>
40. TPTP Auto GUI recorder –test setup
41. TPTP Auto GUI recorder –execution log
42. TPTP Auto GUI recorder – execution log details
43. TPTP API recorder Another tool being developed for Eclipse is an API recorder.
Essentially, this performs capture and replay without having a user interface.
Uses the TPTP profiler.
As methods are called, the parameter values are stored on entry, and the values that are returned are stored as well.
After execution, the saved parameter values are used to create JUnit test cases.
Still in the “technology preview” phase.