software testing for non traditional software
Download
Skip this Video
Download Presentation
Software Testing for Non-traditional Software

Loading in 2 Seconds...

play fullscreen
1 / 62

Software Testing for Non-traditional Software - PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on

Software Testing for Non-traditional Software . Mary Lou Soffa University of Pittsburgh soffa@cs.pitt.edu Currently on sabbatical at IBM T.J. Watson Research. Shanghai, China December 4, 2003. Research Interests . Software Engineering Testing Regression testing - maintenance

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Software Testing for Non-traditional Software' - mckile


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
software testing for non traditional software

Software Testing for Non-traditional Software

Mary Lou Soffa

University of Pittsburgh

soffa@cs.pitt.edu

Currently on sabbatical at

IBM T.J. Watson Research

Shanghai, China

December 4, 2003

research interests
Research Interests
  • Software Engineering
    • Testing
    • Regression testing - maintenance
    • Debugging
  • Optimizing compilers
    • Particular code optimizations
    • Properties of optimizations
      • Specification/profitability
  • Connection: program analysis
non traditional software
Non Traditional Software
  • Traditional software
    • Input
    • Program executes without user interaction and is self contained
    • Program produces results
  • Non Traditional
    • Event driven – user interacts with code as it executes – Graphical User Interfaces
    • Application operates in an environment
      • Database, physical environment
outline
Outline
  • Testing Graphical User Interfaces
    • Challenges – how currently done
    • Test case generation
    • Oracles – expected results
    • Regression testing
testing of graphical unit interfaces gui
Testing of Graphical Unit Interfaces (GUI)

50% of code

GUI

Interactionsbetween the

GUI and the underlying code

Underlying

Code

challenges of gui testing
Challenges of GUI Testing
  • How much testing is enough?
    • (Test coverage)
  • Is the GUI executing correctly during testing?
    • (Test oracles)
  • What test results can be salvaged from the previous test runs to test a new version?
    • (Regression testing)
  • How to represent the GUI to handle all the above?
    • (Representation)
  • How to do the testing automatically without user’s involvement?
gui framework
GUI Framework
  • New representation for GUIs – event driven software
  • Test case generation based on AI planning
  • Unique test oracles
  • New event-based coverage criteria
  • New automated regression testing algorithms
  • Automated test executor
  • A comprehensive framework that integrates all the above components
overview of the framework
Overview of the Framework

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Representation

GUI

Implementer

Executing

GUI

Regression

Tester

Test

Oracle

Test Case

Generator

Test

Coverage

Evaluator

Test

Executor

gui representation
GUI Representation

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Regression

Tester

Test

Oracle

Test Case

Generator

Test

Coverage

Evaluator

Test

Executor

creating the gui model
Creating the GUI Model
  • Modeling
    • GUI’s state in terms of objects & their properties
    • Events as state transducers
    • GUI’s hierarchical structure
modeling the gui s state

Optional Objects

Caption(Button, “Cancel”)

Modeling the GUI’s State
  • A GUI at time T is modeled using:
    • Objects O = {o1, o2, o3, …, om}
    • Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni>= 1) Boolean relation of the form:

Property(oj, oa, ob, …,ox, value)

True/False

Property Name

Object

example modeling the gui s state
Example: Modeling the GUI’s State

Form1

WState(Form1,

Width(Form1,

Scroll(Form1,

wsNormal)

1088)

TRUE)

Button1

Label1

Caption(Button1,

Enabled(Button1,

Visible(Button1,

Height(Button1,

Cancel)

TRUE)

TRUE)

65)

Align(Label1,

Caption(Label1,

Color(Label1,

Font(Label1,

alNone)

“Files of type:”)

clBtnFace)

(tfont))

gui events
GUI Events
  • GUI’s state is not static
  • Events change the GUI’s state
  • Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj
  • Notation: Sj = e1(Si)
example an event

This is the text.

This is the text.

This

Example: An Event

State: Si

SelectText

(“This”)

Event: e

State: Sj

Sj = e(Si)

slide15

Operator Example

Operator :: CUT

Precondition:

isCurrent(Menu2).

Effects:

FORALL Obj in Objects

Selected(Obj) 

ADD inClipboard(Obj)

DEL onScreen(Obj)

DEL Selected(Obj)

ADD isCurrent(Menu1)

DEL isCurrent(Menu2).

Menu1

Cut

Menu2

Primitive Operators

Now we know how to represent the events of the GUI

gui modeling steps
GUI Modeling Steps
  • From the GUI’s specifications (formal/informal),
    • Identify objects and properties
    • Create operators for GUI events
    • Using the event classification, create hierarchical and system-interaction operators
test case generator
Test Case Generator

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Regression

Tester

Test

Oracle

Tasks

Test-Case

Generator

Test

Coverage

Evaluator

Test

Executor

Test Case

gui test cases
GUI Test Cases
  • Individual user events
    • Not enough !
    • Sequences of user events lead to different states
  • Test case: sequence of user events
a test case for wordpad

SelectText

(“text”)

SelectText

(“This”)

OK

Font

18

Format

This is the text.

This is the text.

Underline

OK

Format

Font

A Test Case for WordPad

S0

selecting test sequences
Selecting Test Sequences
  • Infinitely many
  • Randomly choose sequences
  • Expert chooses sequences
  • Automatically generate events for COMMONLY USED TASKS

Multiple

Event

Sequences

This is the text.

This is the text.

Initial State

Goal State

a plan for a gui task

SelectText(“This”)

This

is the

text.

SetFontSize(18)

SelectText(“text”)

MouseClick(U)

This

is the

text.

A Plan for a GUI Task

Initial State

Goal State

This is the text.

This is the text.

using primitive operators
Using Primitive Operators
  • One operator for each event

Operator :: CUT

Precondition:

isCurrent(Menu2).

Effects:

FORALL Obj in Objects

Selected(Obj) 

ADD inClipboard(Obj)

DEL onScreen(Obj)

DEL Selected(Obj)

ADD isCurrent(Menu1)

DEL isCurrent(Menu2).

File Edit View Ins

Menu1

Cut

Menu2

create system interaction operators

File

Mail Recipient

Send To

Create System-Interaction Operators

Sys-Interaction Operator:

File_SendTo_MailRecipient

= <File + SendTo + MailRecipient >

effects of exploiting the gui s structure
Effects of Exploiting the GUI’s Structure
  • Reduction in planning operators
    • 362 operators  32 operators
    • Ratio 10:1 for MS Wordpad
    • 20:1 for MS Word
  • Efficiency
slide26

Initial

State

This is the text.

Goal

State

This is the text.

test case

Planner

Planner

FormatFont

Underline

OK

OK

FormatFont

18

Mapping

Mapping

Font

Format

Format

Font

SelectText

(“text”)

SelectText

(“This”)

OK

Font

18

Format

Underline

OK

Format

Font

Test Case

Primitive

Operator

Hierarchical

Operator

Hierarchical

Operator

Primitive

Operator

INITIAL

GOAL

FormatFont

(“text”, Underline)

FormatFont

(“This”, 18pt)

SelectText

(“text”)

SelectText

(“This”)

test oracles
Test Oracles

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Oracle

GUI State

(run-time)

Regression

Tester

Expected-state

Generator

Tasks

Expected-state

sequence

Test-Case

Generator

Execution

Monitor

Test

Coverage

Evaluator

Verifier

Test Case

Actual State

Information

Verdict

Test Executor

what is correct behavior

?!

SelectText

(“This”)

Font

OK

Format

18

This is the text.

This is the text.

This is the text.

SelectText

(“text”)

Underline

OK

Format

Font

What Is Correct Behavior

Check State, not only Output !!

determine correct behavior

Actual State

Expected State

Automated

Verifier

Verdict

Determine Correct Behavior
  • To check the GUI’s state after each event
  • Approaches
    • Manual
    • Automated
  • Challenges
    • Generating expected state
    • Extracting actual state
    • Comparing expected & actual states
overview of gui oracle
Overview of GUI Oracle

Test Case

Run-time

information from

executing GUI

Expected-state

Generator

GUI

Model

Expected State

Execution

Monitor

Actual

State

Verifier

Oracle

Verdict

expected state

e2

e3

en

e1

Expected State
  • Obtaining Next State
    • Given a test case T with S0, the initial state, and
    • A sequence of events
    • The next state Sy is obtained from the current state Sx and operator Op
      • Delete property P from Sx if Effects(Op) contains the command “DEL P”
      • Add property P in Sy if Effects(Op) contains the command “ADD P”
deriving expected state

S1

S2

S3

Sn

Deriving Expected State
  • Obtain S1 = e1(S0)
  • And Si = ei(Si-1)

en

e2

e3

e1

S0

  • Expected state is a conjunction of properties of the form
    • Property(Objects…, Value)
experiments
Experiments
  • Purpose: determine
    • Time to derive expected state
    • Time to execute monitor and verifier
  • Experimental design
    • GUI: our version of MS Wordpad (36 modal windows, 362 events)
    • Test cases: generated 290 test cases (6-56 events) using an AI planner
    • Hardware platform: 350 mhz pentium based machine, 256 MB RAM
    • Properties: reduced set
    • Level of checking: GUI relevant property set
deriving expected state1
Deriving Expected State

Total CPU time (all 290 test cases and expected states)

75.84 sec.

automated execution1
Automated Execution

GUI Relevant-properties checking

Total running time < 10 minutes

regression testing
Regression Testing

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Oracle

GUI State

(run-time)

Regression

Testing

Algorithms

Expected-state

Generator

Tasks

Expected-state

sequence

Test-Case

Generator

Coverage

Criteria

Coverage

Evaluation

Algorithms

Execution

Monitor

Verifier

Test Case

Coverage

Report

Actual State

Information

Verdict

Test Executor

gui regression testing problem

A test case of length 3

Adobe Acrobat 5.0

File

Document Security

OK

GUI Regression Testing Problem
gui regression testing problem1

A test case of length 3

File

Document Security

OK

Adobe Acrobat 6.0

GUI Regression Testing Problem

Event “Document Security”

no longer in menu

Test case cannot be executed – unusable!!!

a new test case
A New Test Case

File

Security

Document Properties

Repair the old test case

to obtain this new one

OK

overview of our regression testing
Overview of our Regression testing
  • Traditional regression testing
  • Event-flow/component model
  • Repairing test cases
  • Case studies
traditional regression testing
Traditional Regression Testing

original test suite

usable test cases

unusable (obsolete) test cases

discard

select

Selected usable

test cases

new

test cases

regression test suite

  • Regression test-all
  • Selective regression testing process
regression testing for guis
Regression Testing for GUIs
  • Test case generation is manual – capture/replay tools
    • Small number of test cases
    • Each test case is valuable
  • Changes result in a large number of obsolete test cases
    • Structural and layout changes
  • Frequent modifications
    • Driven by constant user feedback – prototyping
    • Need frequent testing
our approach gui regression testing
Our approach GUI Regression Testing

original test suite

usable test cases

unusable test cases

select

repair

not repairable

Selected usable

test cases

new

test cases

discard

repaired

regression test suite

gui regression model
GUI regression model
  • Model represent all legal sequences of events and event interactions
  • Two levels
    • Component-call-graph – built using the concept of modal window
      • Forms GUI component
    • GUI event flow graph – event structure of a component
    • Helps to isolate changes and repairs
representing a component

follows

Cut

Copy

Paste

About …

Contents …

Open …

Save

Open …

Save

Definition: Event ex follows ey iff ex can be performed immediately after ey.

Representing a Component

File Edit Help

Forms event flow graph

component call graph
Component-Call Graph

Main

FileNew

FileSave

FileOpen

PageSetup

Print

ViewOptions

FormatFont

Properties

  • Definition: Component-Call-Graph
  • Nodes are components in the GUI
  • Main is distinguished component
  • Edges represent the invokes relation between components, i.e., (Cx, Cy) є{edges} iff Cx invokes Cy.
event flow graph
Event Flow Graph

File

Help

Edit

Open

Save

About

Contents

Cut

Copy

Paste

To File, Edit

and Help

To File, Edit

and Help

  • Definition: Event Flow FG
  • Vertices represent events
  • Edges show the follows relationship
  • Events first available (shown in red)
  • Events that invoke other components (dotted lines).
to repair test cases
To repair test cases

Two phases

  • Checker
    • Determine modifications made to original GUI
    • categorizes test case usable or unusable
  • Repairer
    • Determines if unusable cases repairable
      • If yes, repairs test case

Note: can be done in same step in implementation

checker
Checker

Automatically determines modifications –

  • Models for original and modified GUIs
    • Component call graph
      • Components & calls added and deleted
    • Event flow graph (per component)
      • Events & interactions added and deleted
  • Additions do not cause a test case to become unusuable but deletions do
  • One GUI modification – multiple changes in test case
repairer
Repairer

Takes unusable test case sequence: searches from the deleted node/edge

    • Events and edges that can be deleted and lead to a legal sequence
    • An event that can be inserted that leads to a legal sequence with deletion
  • More than one way to repair – do all to produce more test cases
repairing process

ek

...

e1

ei+l

ei-1

ei

ei+m

ei+1

follows

en

Repairing Process
  • A deleted event – delete to repair

...

...

repairing process1

ek

e1

ei+1

ei-1

ei

ex

follows

en

Repairing Process
  • A deleted event – insert to repair

...

ei+m

...

...

repairing process2

e1

e1

e1

delete

en

...

ek

ei

ei

ei

ei+1

en

ex

...

Repairing process

Delete edge

en

ei+1

...

...

ei-1

...

follows

...

ei-1

add

...

ei-1

case studies
Case Studies
  • Questions
    • How many test cases are made unusable by GUI modifications across versions?
    • How many unusable test cases are repairable?
    • How much time does the checker and repairing processes take?
two applications
Two Applications
  • Each with two versions
      • Adobe’s Acrobat Reader
        • version 4.0 (for Linux)
          • 15 components with 176 events (not counting short-cuts)
        • version 5.0 (for MS Windows 2000)
          • 25 components with 351 events
      • MS WordPad – our own version
        • 36 modal windows, and 362 events (not counting short-cuts)

Menu added to version 2

adobe application
Adobe application
  • 400 test cases generated manually using a capture/replay tool
    • Generate test cases: 8 hours
  • Old and new versions
    • Checker time: 7 sec
    • Unusable: 296 (74%) test cases
    • Repaired: 211 (71%)
    • Repairer: 18 sec.
    • Regression test suite: 315 (79%)
wordpad application
Wordpad application

Comparable results for Wordpad

400 test cases

    • Generate test cases: 5 hours
  • Old and new version
    • Checker time: 6 sec
    • Unusable: 210 (52%)
    • Repaired: 210 (100%)
    • Repairer: 18 sec
    • Regression test suite: 100%
putting it all together
Putting it All Together

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Oracle

GUI State

(run-time)

Regression

Testing

Algorithms

Expected-state

Generator

Tasks

Expected-state

sequence

Test-Case

Generator

Coverage

Criteria

Coverage

Evaluation

Algorithms

Execution

Monitor

Verifier

Test Case

Coverage

Report

Actual State

Information

Verdict

Test Executor

conclusions future work
Conclusions & Future Work
  • Broaden the testing of GUIs to the testing of event driven systems
  • Debugging of event driven software
ad