Software testing for non traditional software
Download
1 / 62

Software Testing for Non-traditional Software - PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on

Software Testing for Non-traditional Software . Mary Lou Soffa University of Pittsburgh [email protected] Currently on sabbatical at IBM T.J. Watson Research. Shanghai, China December 4, 2003. Research Interests . Software Engineering Testing Regression testing - maintenance

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Software Testing for Non-traditional Software' - mckile


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Software testing for non traditional software

Software Testing for Non-traditional Software

Mary Lou Soffa

University of Pittsburgh

[email protected]

Currently on sabbatical at

IBM T.J. Watson Research

Shanghai, China

December 4, 2003


Research interests
Research Interests

  • Software Engineering

    • Testing

    • Regression testing - maintenance

    • Debugging

  • Optimizing compilers

    • Particular code optimizations

    • Properties of optimizations

      • Specification/profitability

  • Connection: program analysis


Non traditional software
Non Traditional Software

  • Traditional software

    • Input

    • Program executes without user interaction and is self contained

    • Program produces results

  • Non Traditional

    • Event driven – user interacts with code as it executes – Graphical User Interfaces

    • Application operates in an environment

      • Database, physical environment


Outline
Outline

  • Testing Graphical User Interfaces

    • Challenges – how currently done

    • Test case generation

    • Oracles – expected results

    • Regression testing


Testing of graphical unit interfaces gui
Testing of Graphical Unit Interfaces (GUI)

50% of code

GUI

Interactionsbetween the

GUI and the underlying code

Underlying

Code


Challenges of gui testing
Challenges of GUI Testing

  • How much testing is enough?

    • (Test coverage)

  • Is the GUI executing correctly during testing?

    • (Test oracles)

  • What test results can be salvaged from the previous test runs to test a new version?

    • (Regression testing)

  • How to represent the GUI to handle all the above?

    • (Representation)

  • How to do the testing automatically without user’s involvement?


Gui framework
GUI Framework

  • New representation for GUIs – event driven software

  • Test case generation based on AI planning

  • Unique test oracles

  • New event-based coverage criteria

  • New automated regression testing algorithms

  • Automated test executor

  • A comprehensive framework that integrates all the above components


Overview of the framework
Overview of the Framework

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Representation

GUI

Implementer

Executing

GUI

Regression

Tester

Test

Oracle

Test Case

Generator

Test

Coverage

Evaluator

Test

Executor


Gui representation
GUI Representation

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Regression

Tester

Test

Oracle

Test Case

Generator

Test

Coverage

Evaluator

Test

Executor


Creating the gui model
Creating the GUI Model

  • Modeling

    • GUI’s state in terms of objects & their properties

    • Events as state transducers

    • GUI’s hierarchical structure


Modeling the gui s state

Optional Objects

Caption(Button, “Cancel”)

Modeling the GUI’s State

  • A GUI at time T is modeled using:

    • Objects O = {o1, o2, o3, …, om}

    • Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni>= 1) Boolean relation of the form:

Property(oj, oa, ob, …,ox, value)

True/False

Property Name

Object


Example modeling the gui s state
Example: Modeling the GUI’s State

Form1

WState(Form1,

Width(Form1,

Scroll(Form1,

wsNormal)

1088)

TRUE)

Button1

Label1

Caption(Button1,

Enabled(Button1,

Visible(Button1,

Height(Button1,

Cancel)

TRUE)

TRUE)

65)

Align(Label1,

Caption(Label1,

Color(Label1,

Font(Label1,

alNone)

“Files of type:”)

clBtnFace)

(tfont))


Gui events
GUI Events

  • GUI’s state is not static

  • Events change the GUI’s state

  • Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj

  • Notation: Sj = e1(Si)


Example an event

This is the text.

This is the text.

This

Example: An Event

State: Si

SelectText

(“This”)

Event: e

State: Sj

Sj = e(Si)


Operator Example

Operator :: CUT

Precondition:

isCurrent(Menu2).

Effects:

FORALL Obj in Objects

Selected(Obj) 

ADD inClipboard(Obj)

DEL onScreen(Obj)

DEL Selected(Obj)

ADD isCurrent(Menu1)

DEL isCurrent(Menu2).

Menu1

Cut

Menu2

Primitive Operators

Now we know how to represent the events of the GUI


Gui modeling steps
GUI Modeling Steps

  • From the GUI’s specifications (formal/informal),

    • Identify objects and properties

    • Create operators for GUI events

    • Using the event classification, create hierarchical and system-interaction operators


Test case generator
Test Case Generator

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Regression

Tester

Test

Oracle

Tasks

Test-Case

Generator

Test

Coverage

Evaluator

Test

Executor

Test Case


Gui test cases
GUI Test Cases

  • Individual user events

    • Not enough !

    • Sequences of user events lead to different states

  • Test case: sequence of user events


A test case for wordpad

SelectText

(“text”)

SelectText

(“This”)

OK

Font

18

Format

This is the text.

This is the text.

Underline

OK

Format

Font

A Test Case for WordPad

S0


Selecting test sequences
Selecting Test Sequences

  • Infinitely many

  • Randomly choose sequences

  • Expert chooses sequences

  • Automatically generate events for COMMONLY USED TASKS

Multiple

Event

Sequences

This is the text.

This is the text.

Initial State

Goal State


A plan for a gui task

SelectText(“This”)

This

is the

text.

SetFontSize(18)

SelectText(“text”)

MouseClick(U)

This

is the

text.

A Plan for a GUI Task

Initial State

Goal State

This is the text.

This is the text.



Using primitive operators
Using Primitive Operators

  • One operator for each event

Operator :: CUT

Precondition:

isCurrent(Menu2).

Effects:

FORALL Obj in Objects

Selected(Obj) 

ADD inClipboard(Obj)

DEL onScreen(Obj)

DEL Selected(Obj)

ADD isCurrent(Menu1)

DEL isCurrent(Menu2).

File Edit View Ins

Menu1

Cut

Menu2


Create system interaction operators

File

Mail Recipient

Send To

Create System-Interaction Operators

Sys-Interaction Operator:

File_SendTo_MailRecipient

= <File + SendTo + MailRecipient >


Effects of exploiting the gui s structure
Effects of Exploiting the GUI’s Structure

  • Reduction in planning operators

    • 362 operators  32 operators

    • Ratio 10:1 for MS Wordpad

    • 20:1 for MS Word

  • Efficiency


Initial

State

This is the text.

Goal

State

This is the text.


Test case

Planner

Planner

FormatFont

Underline

OK

OK

FormatFont

18

Mapping

Mapping

Font

Format

Format

Font

SelectText

(“text”)

SelectText

(“This”)

OK

Font

18

Format

Underline

OK

Format

Font

Test Case

Primitive

Operator

Hierarchical

Operator

Hierarchical

Operator

Primitive

Operator

INITIAL

GOAL

FormatFont

(“text”, Underline)

FormatFont

(“This”, 18pt)

SelectText

(“text”)

SelectText

(“This”)




Test oracles
Test Oracles

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Oracle

GUI State

(run-time)

Regression

Tester

Expected-state

Generator

Tasks

Expected-state

sequence

Test-Case

Generator

Execution

Monitor

Test

Coverage

Evaluator

Verifier

Test Case

Actual State

Information

Verdict

Test Executor


What is correct behavior

?!

SelectText

(“This”)

Font

OK

Format

18

This is the text.

This is the text.

This is the text.

SelectText

(“text”)

Underline

OK

Format

Font

What Is Correct Behavior

Check State, not only Output !!


Determine correct behavior

Actual State

Expected State

Automated

Verifier

Verdict

Determine Correct Behavior

  • To check the GUI’s state after each event

  • Approaches

    • Manual

    • Automated

  • Challenges

    • Generating expected state

    • Extracting actual state

    • Comparing expected & actual states


Overview of gui oracle
Overview of GUI Oracle

Test Case

Run-time

information from

executing GUI

Expected-state

Generator

GUI

Model

Expected State

Execution

Monitor

Actual

State

Verifier

Oracle

Verdict


Expected state

e2

e3

en

e1

Expected State

  • Obtaining Next State

    • Given a test case T with S0, the initial state, and

    • A sequence of events

    • The next state Sy is obtained from the current state Sx and operator Op

      • Delete property P from Sx if Effects(Op) contains the command “DEL P”

      • Add property P in Sy if Effects(Op) contains the command “ADD P”


Deriving expected state

S1

S2

S3

Sn

Deriving Expected State

  • Obtain S1 = e1(S0)

  • And Si = ei(Si-1)

en

e2

e3

e1

S0

  • Expected state is a conjunction of properties of the form

    • Property(Objects…, Value)


Experiments
Experiments

  • Purpose: determine

    • Time to derive expected state

    • Time to execute monitor and verifier

  • Experimental design

    • GUI: our version of MS Wordpad (36 modal windows, 362 events)

    • Test cases: generated 290 test cases (6-56 events) using an AI planner

    • Hardware platform: 350 mhz pentium based machine, 256 MB RAM

    • Properties: reduced set

    • Level of checking: GUI relevant property set


Deriving expected state1
Deriving Expected State

Total CPU time (all 290 test cases and expected states)

75.84 sec.


Automated execution1
Automated Execution

GUI Relevant-properties checking

Total running time < 10 minutes


Regression testing
Regression Testing

GUI Implementation:

Tools (Languages/Toolkits)

GUI Specifications

GUI Specifications

GUI

Implementer

GUI Model

Test Designer

Executing

GUI

Oracle

GUI State

(run-time)

Regression

Testing

Algorithms

Expected-state

Generator

Tasks

Expected-state

sequence

Test-Case

Generator

Coverage

Criteria

Coverage

Evaluation

Algorithms

Execution

Monitor

Verifier

Test Case

Coverage

Report

Actual State

Information

Verdict

Test Executor


Gui regression testing problem

A test case of length 3

Adobe Acrobat 5.0

File

Document Security

OK

GUI Regression Testing Problem


Gui regression testing problem1

A test case of length 3

File

Document Security

OK

Adobe Acrobat 6.0

GUI Regression Testing Problem

Event “Document Security”

no longer in menu

Test case cannot be executed – unusable!!!


A new test case
A New Test Case

File

Security

Document Properties

Repair the old test case

to obtain this new one

OK


Overview of our regression testing
Overview of our Regression testing

  • Traditional regression testing

  • Event-flow/component model

  • Repairing test cases

  • Case studies


Traditional regression testing
Traditional Regression Testing

original test suite

usable test cases

unusable (obsolete) test cases

discard

select

Selected usable

test cases

new

test cases

regression test suite

  • Regression test-all

  • Selective regression testing process


Regression testing for guis
Regression Testing for GUIs

  • Test case generation is manual – capture/replay tools

    • Small number of test cases

    • Each test case is valuable

  • Changes result in a large number of obsolete test cases

    • Structural and layout changes

  • Frequent modifications

    • Driven by constant user feedback – prototyping

    • Need frequent testing


Our approach gui regression testing
Our approach GUI Regression Testing

original test suite

usable test cases

unusable test cases

select

repair

not repairable

Selected usable

test cases

new

test cases

discard

repaired

regression test suite


Gui regression model
GUI regression model

  • Model represent all legal sequences of events and event interactions

  • Two levels

    • Component-call-graph – built using the concept of modal window

      • Forms GUI component

    • GUI event flow graph – event structure of a component

    • Helps to isolate changes and repairs


Representing a component

follows

Cut

Copy

Paste

About …

Contents …

Open …

Save

Open …

Save

Definition: Event ex follows ey iff ex can be performed immediately after ey.

Representing a Component

File Edit Help

Forms event flow graph


Component call graph
Component-Call Graph

Main

FileNew

FileSave

FileOpen

PageSetup

Print

ViewOptions

FormatFont

Properties

  • Definition: Component-Call-Graph

  • Nodes are components in the GUI

  • Main is distinguished component

  • Edges represent the invokes relation between components, i.e., (Cx, Cy) є{edges} iff Cx invokes Cy.


Event flow graph
Event Flow Graph

File

Help

Edit

Open

Save

About

Contents

Cut

Copy

Paste

To File, Edit

and Help

To File, Edit

and Help

  • Definition: Event Flow FG

  • Vertices represent events

  • Edges show the follows relationship

  • Events first available (shown in red)

  • Events that invoke other components (dotted lines).


To repair test cases
To repair test cases

Two phases

  • Checker

    • Determine modifications made to original GUI

    • categorizes test case usable or unusable

  • Repairer

    • Determines if unusable cases repairable

      • If yes, repairs test case

        Note: can be done in same step in implementation


Checker
Checker

Automatically determines modifications –

  • Models for original and modified GUIs

    • Component call graph

      • Components & calls added and deleted

    • Event flow graph (per component)

      • Events & interactions added and deleted

  • Additions do not cause a test case to become unusuable but deletions do

  • One GUI modification – multiple changes in test case


Repairer
Repairer

Takes unusable test case sequence: searches from the deleted node/edge

  • Events and edges that can be deleted and lead to a legal sequence

  • An event that can be inserted that leads to a legal sequence with deletion

  • More than one way to repair – do all to produce more test cases


  • Repairing process

    ek

    ...

    e1

    ei+l

    ei-1

    ei

    ei+m

    ei+1

    follows

    en

    Repairing Process

    • A deleted event – delete to repair

    ...

    ...


    Repairing process1

    ek

    e1

    ei+1

    ei-1

    ei

    ex

    follows

    en

    Repairing Process

    • A deleted event – insert to repair

    ...

    ei+m

    ...

    ...


    Repairing process2

    e1

    e1

    e1

    delete

    en

    ...

    ek

    ei

    ei

    ei

    ei+1

    en

    ex

    ...

    Repairing process

    Delete edge

    en

    ei+1

    ...

    ...

    ei-1

    ...

    follows

    ...

    ei-1

    add

    ...

    ei-1


    Case studies
    Case Studies

    • Questions

      • How many test cases are made unusable by GUI modifications across versions?

      • How many unusable test cases are repairable?

      • How much time does the checker and repairing processes take?


    Two applications
    Two Applications

    • Each with two versions

      • Adobe’s Acrobat Reader

        • version 4.0 (for Linux)

          • 15 components with 176 events (not counting short-cuts)

        • version 5.0 (for MS Windows 2000)

          • 25 components with 351 events

      • MS WordPad – our own version

        • 36 modal windows, and 362 events (not counting short-cuts)

    Menu added to version 2


    Adobe application
    Adobe application

    • 400 test cases generated manually using a capture/replay tool

      • Generate test cases: 8 hours

    • Old and new versions

      • Checker time: 7 sec

      • Unusable: 296 (74%) test cases

      • Repaired: 211 (71%)

      • Repairer: 18 sec.

      • Regression test suite: 315 (79%)


    Wordpad application
    Wordpad application

    Comparable results for Wordpad

    400 test cases

    • Generate test cases: 5 hours

  • Old and new version

    • Checker time: 6 sec

    • Unusable: 210 (52%)

    • Repaired: 210 (100%)

    • Repairer: 18 sec

    • Regression test suite: 100%


  • Putting it all together
    Putting it All Together

    GUI Implementation:

    Tools (Languages/Toolkits)

    GUI Specifications

    GUI Specifications

    GUI

    Implementer

    GUI Model

    Test Designer

    Executing

    GUI

    Oracle

    GUI State

    (run-time)

    Regression

    Testing

    Algorithms

    Expected-state

    Generator

    Tasks

    Expected-state

    sequence

    Test-Case

    Generator

    Coverage

    Criteria

    Coverage

    Evaluation

    Algorithms

    Execution

    Monitor

    Verifier

    Test Case

    Coverage

    Report

    Actual State

    Information

    Verdict

    Test Executor


    Conclusions future work
    Conclusions & Future Work

    • Broaden the testing of GUIs to the testing of event driven systems

    • Debugging of event driven software


    ad