Automated Developer Testing:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 55

Automated Developer Testing: Achievements and Challenges PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

Automated Developer Testing: Achievements and Challenges. Tao Xie North Carolina State University contact: [email protected] Automation in Developer Testing. Background on developer testing http://www.developertesting.com/

Download Presentation

Automated Developer Testing: Achievements and Challenges

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Automated developer testing achievements and challenges

Automated Developer Testing: Achievements and Challenges

Tao Xie

North Carolina State University

contact: [email protected]


Automation in developer testing

Automation in Developer Testing

  • Background on developer testing

    • http://www.developertesting.com/

    • Kent Beck’s 2004 talk on “Future of Developer Testing”http://www.itconversations.com/shows/detail301.html

  • This talk focuses on developer testing

    • Not system testing etc. conducted by testers

  • Unit Test Automation commonly referred to writing unit test cases manually, executed automatically

  • Automation here is broad, including automatic test generation


Software testing setup

?

=

Software Testing Setup

+

Expected Outputs

Test inputs

Program

Outputs

Test Oracles


Software testing problems

?

=

Software Testing Problems

+

Expected Outputs

Test inputs

Program

Outputs

Test Oracles

  • Faster: How can tools help developers create and run tests faster?


Software testing problems1

?

=

Software Testing Problems

+

Expected Outputs

Test inputs

Program

Outputs

Test Oracles

  • Faster: How can tools help developers create and run tests faster?

  • Better Test Inputs: How can tools help generate new better test inputs?


Software testing problems2

?

=

Software Testing Problems

+

Expected Outputs

Test inputs

Program

Outputs

Test Oracles

  • Faster: How can tools help developers create and run tests faster?

  • Better Test Inputs: How can tools help generate new better test inputs?

  • Better Test Oracles: How can tools help generate better test oracles?


Example unit test case

?

=

Example Unit Test Case

+

Expected Outputs

Test inputs

Program

Outputs

Test Oracles

void addTest() {

ArrayList a = new ArrayList(1);

Object o = new Object();

a.add(o);

AssertTrue(a.get(0) == o);

}

Test Case = Test Input + Test Oracle

  • Appropriate method sequence

  • Appropriate primitive argument values

  • Appropriate assertions


Levels of test oracles

Levels of Test Oracles

  • Expected output for an individual test input

    • In the form of assertions in test code

  • Properties applicable for multiple test inputs

    • Crash (uncaught exceptions) or not, related to robustness issues, supported by most tools

    • Properties in production code: Design by Contract (precondition, postcondition, class invariants) supported by ParasoftJtest, Google CodeProAnalytiX

    • Properties in test code: Parameterized unit tests supported by MSR Pex, AgitarOne

X. Xiao, S. Thummalapenta, and T. Xie. Advances on Improving Automation in Developer Testing. In Advances in Computers, 2012 http://people.engr.ncsu.edu/txie/publications.htm#ac12-devtest


Economics of test oracles

Economics of Test Oracles

  • Expected output for an individual test input

    • Easy to manually verify for one test input

    • Expensive/infeasible to verify for many test inputs

    • Limited benefits: only for one test input

  • Properties applicable for multiple test inputs

    • Not easy to write (need abstraction skills)

    • But once written, broad benefits for multiple test inputs


Assert behavior of multiple test inputs design by contract

Assert behavior of multiple test inputs Design by Contract

  • Example tools: ParasoftJtest, Google CodeProAnalytiX, MSR Code Contracts, MSR Pex

  • Class invariant: properties being satisfied by an object (in a consistent state) [AgitarOne allows a class invariant helper method used as test oracles]

  • Precondition: conditions to be satisfied (on receiver object and arguments) before a method can be invoked

  • Postcondition: properties being satisfied (on receiver object and return) after the method has returned

  • Other types of specs also exist

http://research.microsoft.com/en-us/projects/contracts/


Microsoft research code contracts

Microsoft Research Code Contracts

publicvirtualint Add(object value){ Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if(count == items.Length) EnsureCapacity(count + 1); items[count] = value; returncount++;}

  • Features

  • Language expression syntax

    • Type checking / IDE

  • Declarative

  • Special Encodings

    • Result and Old

[ContractInvariantMethod]voidObjectInvariant() {Contract.Invariant( items != null );}

- Slide adapted from MSR RiSE

http://research.microsoft.com/en-us/projects/contracts/


Parameterized unit testing

Parameterized Unit Testing

[Tillmann&Schulte ESEC/FSE 05]

  • Parameterized Unit Test =

    Unit Test with Parameters

  • Separation of concerns

    • Data is generated by a tool

    • Developer can focus on functional specification

void TestAdd(List list, int item) {

Assume.IsTrue(list != null);

var count = list.Count;

list.Add(item);

Assert.AreEqual(count + 1, list.Count);

}

http://research.microsoft.com/apps/pubs/default.aspx?id=77419


P arameterized unit tests are formal specifications a lgebraic s pecifications

Parameterized Unit Testsare Formal SpecificationsAlgebraic Specifications

  • A Parameterized Unit Test can be read as a universally quantified, conditional axiom.

void TestReadWrite(Res r, string name, string data) {Assume.IsTrue(r!=null & name!=null && data!=null);

r.WriteResource(name, data);Assert.AreEqual(r.ReadResource(name), data);

}

 string name, string data, Res r:

r ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒

equals(

ReadResource(WriteResource(r, name, data).state, name),

data)


Parameterized unit tests in pex

Parameterized Unit Tests in Pex

http://research.microsoft.com/pex/


Parameterized unit testing getting popular

Parameterized Unit TestingGetting Popular

Parameterized Unit Tests (PUTs) commonly supported by various test frameworks

  • .NET: Supported by .NET test frameworks

    • http://www.mbunit.com/

    • http://www.nunit.org/

  • Java: Supported by JUnit 4.X

    • http://www.junit.org/

      Generating test inputs for PUTs supported by tools

  • .NET: Supported by Microsoft Research Pex

    • http://research.microsoft.com/Pex/

  • Java: Supported by AgitarAgitarOne

    • http://www.agitar.com/


Parameterized test driven development

ParameterizedTest-Driven Development

Bug in PUT

Write/refine Contract as PUT

Bug in Code

Write/refine Code of Implementation

Fix-it (with Pex),

Debug with generated tests

failures

Run Pex

no failures

Use Generated Tests for Regression


Assert behavior of multiple test inputs software agitation in agitarone

SoftwareAgitation

Observationson code behavior,

plus

Test Coverage data

Code

If an Observationreveals a bug, fix it

Compile

If it describes desired behavior, click to create a Test Assertion

Code

Agitate

Review

Assert behavior of multiple test inputs Software Agitation in AgitarOne

http://www.agitar.com/

- Slide adapted from Agitar Software Inc.


Software agitation in agitarone

Software Agitation in AgitarOne

Image from http://www.agitar.com/


Automated test generation

Automated Test Generation

  • Recent advanced technique: Dynamic Symbolic Execution/Concolic Testing

    • Instrument code to explore feasible paths

  • Example tool: Pex from Microsoft Research (for .NET programs)

P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In Proc. PLDI 2005

K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In Proc. ESEC/FSE 2005

N. Tillmann and J. de Halleux. Pex - White Box Test Generation for .NET. In Proc. TAP 2008


Automated developer testing achievements and challenges

Dynamic Symbolic Execution in Pex

Choose next path

Solve

Execute&Monitor

void CoverMe(int[] a)

{

if (a == null) return;

if (a.Length > 0)

if (a[0] == 1234567890)

throw new Exception("bug");

}

Constraints to solve

a!=null

a!=null &&

a.Length>0

a!=null &&

a.Length>0 &&

a[0]==123456890

Observed constraints

a==null

a!=null &&

!(a.Length>0)

a==null &&

a.Length>0 &&

a[0]!=1234567890

a==null &&

a.Length>0 &&

a[0]==1234567890

Input

null

{}

{0}

{123…}

a==null

T

F

a.Length>0

T

Done: There is no path left.

F

a[0]==123…

F

T

http://pex4fun.com/HowDoesPexWork


Automating test generation

Automating Test Generation

@NCSU ASE

  • Method sequences

    • MSeqGen/Seeker [Thummalapenta et al. OOSPLA 11, ESEC/FSE 09], Covana[Xiao et al. ICSE 2011], OCAT [Jaygarl et al. ISSTA 10], Evacon[Inkumsah et al. ASE 08], Symclat[d'Amorim et al. ASE 06]

  • Environments e.g., db, file systems, network, …

    • DBApp Testing [Taneja et al. ESEC/FSE 11], [Pan et al. ASE 11]

    • CloudApp Testing [Zhang et al. IEEE Soft 12]

  • Loops

    • Fitnex[Xie et al. DSN 09]

http://people.engr.ncsu.edu/txie/publications.htm


Pex on msdn devlabs incubation project for visual studio

Pex on MSDN DevLabsIncubation Project for Visual Studio

  • Download counts (20 months)(Feb. 2008 - Oct. 2009 )

    • Academic: 17,366

    • Devlabs: 13,022

    • Total: 30,388

http://research.microsoft.com/projects/pex/


Open source pex extensions

Open Source Pex extensions

http://pexase.codeplex.com/

Publications:http://research.microsoft.com/en-us/projects/pex/community.aspx#publications


Writing test oracles learning formal methods

Writing Test Oracles Learning Formal Methods!?

  • Parameterized Unit Test =

    Unit Test with Parameters

  • Separation of concerns

    • Data is generated by a tool

    • Developer can focus on functional specification

void TestAdd(List list, int item) {

Assume.IsTrue(list != null);

var count = list.Count;

list.Add(item);

Assert.AreEqual(count + 1, list.Count);

}


Automatic test generation human assistance to test generation

Automatic Test Generation  Human Assistance to Test Generation?!

Running Symbolic PathFinder ...

====================================================== results

no errors detected

====================================================== statistics

elapsed time: 0:00:02

states: new=4, visited=0, backtracked=4, end=2

search: maxDepth=3, constraints=0

choice generators: thread=1, data=2

heap: gc=3, new=271, free=22

instructions: 2875

max memory: 81MB

loaded code: classes=71, methods=884


Challenges faced by test generation tools

Challenges Faced by Test Generation Tools

  • Example: Dynamic Symbolic Execution/Concolic Testing

    • Instrument code to explore feasible paths

    • Challenge: path explosion

Total block coverage achieved is 50%, lowest coverage 16%.

  • object-creation problems (OCP) - 65%

  • external-method call problems (EMCP) – 27%


Automated developer testing achievements and challenges

Example Object-Creation Problem

00: classGraph : IVEListGraph { …

03: public void AddVertex (IVertex v) {

04: vertices.Add(v); // B1 }

06: public Edge AddEdge (IVertex v1, IVertex v2) {

07: if (!vertices.Contains(v1))

08: throw new VNotFoundException("");

09: // B2

10: if (!vertices.Contains(v2))

11: throw new VNotFoundException("");

12: // B3

14: Edge e = new Edge(v1, v2);

15: edges.Add(e); } }

//DFS:DepthFirstSearch

18: classDFSAlgorithm{ …

23: public void Compute (IVertex s) { ...

24: if (graph.GetEdges().Size() > 0) { // B4

25: isComputed = true;

26: foreach (Edge e ingraph.GetEdges()) {

27: ... // B5

28: }

29: } } }

  • A graph example from QuickGraph library

  • Includes two classes

    Graph

    DFSAlgorithm

  • Graph

    AddVertex

    AddEdge: requires both vertices to be in graph

[Thummalapenta et al. OOPSLA 11]


Automated developer testing achievements and challenges

Example Object-Creation Problem

  • Test target: Cover true branch (B4) of Line 24

  • Desired object state: graph should include at least one edge

  • Target sequence:

    • Graph ag = new Graph();

    • Vertex v1 = new Vertex(0);

    • Vertex v2 = new Vertex(1);

    • ag.AddVertex(v1);

    • ag.AddVertex(v2);

    • ag.AddEdge(v1, v2);

    • DFSAlgorithmalgo = new DFSAlgorithm(ag);

    • algo.Compute(v1);

00: classGraph : IVEListGraph { …

03: public void AddVertex (IVertex v) {

04: vertices.Add(v); // B1 }

06: public Edge AddEdge (IVertex v1, IVertex v2) {

07: if (!vertices.Contains(v1))

08: throw new VNotFoundException("");

09: // B2

10: if (!vertices.Contains(v2))

11: throw new VNotFoundException("");

12: // B3

14: Edge e = new Edge(v1, v2);

15: edges.Add(e); } }

//DFS:DepthFirstSearch

18: classDFSAlgorithm{ …

23: public void Compute (IVertex s) { ...

24: if (graph.GetEdges().Size() > 0) { // B4

25: isComputed = true;

26: foreach (Edge e ingraph.GetEdges()) {

27: ... // B5

28: }

29: } } }

28

[Thummalapenta et al. OOPSLA 11]


Challenges faced by test generation tools1

Challenges Faced by Test Generation Tools

  • Example: Dynamic Symbolic Execution/Concolic (Pex)

    • Instrument code to explore feasible paths

    • Challenge: path explosion

Total block coverage achieved is 50%, lowest coverage 16%.

  • object-creation problems (OCP) - 65%

  • external-method call problems (EMCP) – 27%


Example external method call problems emcp

Example External-Method Call Problems (EMCP)

  • Example 1:

    • File.Existshas data dependencies on program input

    • Subsequent branch at Line 1 using the return value of File.Exists.

1

  • Example 2:

    • Path.GetFullPathhas data dependencies on program input

    • Path.GetFullPaththrows exceptions.

2

  • Example3: String.Formatdo not cause any problem

3


Human can help object creation problems ocp

Human Can Help! Object Creation Problems (OCP)

Tackle object-creation problems with Factory Methods


Human can help external method call problems emcp

Human Can Help!External-Method Call Problems (EMCP)

Tackle external-method call problems with Mock Methods or Method Instrumentation

Mocking System.IO.File.ReadAllText


State of the art practice testing tools

Tools typically don’t communicate challenges faced by them to enable cooperation between tools and users.

We typically don’t teach people how to cooperate with tools.

State-of-the-Art/Practice Testing Tools

Running Symbolic PathFinder ...

====================================================== results

no errors detected

====================================================== statistics

elapsed time: 0:00:02

states: new=4, visited=0, backtracked=4, end=2

search: maxDepth=3, constraints=0

choice generators: thread=1, data=2

heap: gc=3, new=271, free=22

instructions: 2875

max memory: 81MB

loaded code: classes=71, methods=884

X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise Identification of Problems for Structural Test Generation. In Proc. ICSE 2011

http://people.engr.ncsu.edu/txie/publications/icse11-covana.pdf


Coding duels

Coding Duels

1,206,095 clicked 'Ask Pex!'


Coding duels1

Coding Duels

Pex computes “semantic diff” in cloud

code written in browser vs.

secret reference implementation

You win when Pex finds no differences

secret


Behind the scene of pex for fun

Behind the Scene of Pex for Fun

behavior

Secret Impl== Player Impl

Player Implementation

class Player {

public static int Puzzle(int x) {

return x;

}

}

Secret Implementation

class Secret {

public static int Puzzle(int x) {

if (x <= 0) return 1;

return x * Puzzle(x-1);

}

}

class Test {

public static void Driver(int x) {

if (Secret.Puzzle(x) != Player.Puzzle(x))

throw new Exception(“Mismatch”);

}

}


Coding duels fun and engaging

Coding DuelsFun and Engaging

Iterative gameplay

Adaptive

Personalized

No cheating

Clear winning criterion


Example user feedback

Example User Feedback

Released since 2010

“I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.”

X

“I’m afraid I’ll have to constrain myselfto spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.”

“It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”


Coding duel competition @icse 2011

Coding Duel [email protected] 2011

http://pexforfun.com/icse2011


Teaching and learning

Teaching and Learning


Coding duels for automatic grading @grad software engineering course

Coding Duels for Automatic [email protected] Software Engineering Course

http://pexforfun.com/gradsofteng


Coding duels for training testing

Coding Duels for Training Testing

public static string Puzzle(int[] elems, int capacity, intelem) { if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))

return "Assumption Violation!"; Stack s= new Stack(capacity); for (inti = 0; i < elems.Length; i++)s.Push(elems[i]);intorigSize = s.GetNumOfElements(); //Please fill in below test scenario on the s stack //The lines below include assertions to assert the program behaviorPexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());PexAssert.IsTrue(s.IsMember(elem)); return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “

+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();

}

Set up a stack with some elements

Cache values used in assertions


Usage scenarios of pex 4 fun

Usage Scenarios of Pex4Fun

  • Massive Open Online Courses (MOOC): Challenges

    • Grading, addressed by Pex4Fun

    • Cheating [Open Challenge]

  • Course assignments (students/professionals)

    • E.g., intro programming, software engineering

  • Student/professional competitions

    • E.g., coding-duel competition at ICSE 2011

  • Assessment of testing/programming/problem solving skills for job applicants

    • Not just final results of problem solving but also process!


More reading

More Reading

Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop

Teaching and Learning Programming and Software Engineering via Interactive Gaming

In Proceedings of the 35th International Conference on Software Engineering (ICSE 2013), Software Engineering Education (SEE), San Francisco, CA, May 2013.

http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf


Conclusion

Conclusion

  • Software testing is important and yet costly; needs automation

  • Better Test Inputs: help generate new better test inputs

    • Generate method arguments

    • Generate method sequences

  • Better Test Oracles: help generate better test oracles

    • Assert behavior of individual test inputs

    • Assert behavior of multiple test inputs

  • Software Testing  Educational Gaming

    • http://www.pexforfun.com/


Example industrial developer testing tools

Example Industrial Developer Testing Tools

  • AgitarAgitatorOnehttp://www.agitar.com/

  • ParasoftJtesthttp://www.parasoft.com/

  • Google CodeProAnalytiXhttps://developers.google.com/java-dev-tools/codepro/doc/

  • SilverMark Test Mentor http://www.silvermark.com/

  • Microsoft Research Pex (for .NET) http://research.microsoft.com/Pex/

  • Microsoft Research Spec Explorer (for .NET)http://research.microsoft.com/specexplorer/


Trends in practice

Trends in Practice

  • Regression Test Selection/Prioritization

  • Cloud Computing for Test Execution, e.g., http://www.skytap.com/

  • Crowdsourcing for Testing, e.g., http://www.utest.com/

  • Mocking Environments

    • Google: EasyMock

    • Microsoft VS: Fake/Moleshttp://research.microsoft.com/en-us/projects/pex/

  • Automatic Test Generation

    • Microsoft: Pex, SAGE http://research.microsoft.com/en-us/um/people/pg/


Automated developer testing achievements and challenges

Q & A

Thank you!

contact: [email protected]

Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,

a Microsoft Research SEIF Award, and a Microsoft Research Award.


Automated combinatorial testing

Automated Combinatorial Testing

http://csrc.nist.gov/groups/SNS/acts/index.html

Goals – reduce testing cost, improve cost-benefit ratio

Accomplishments – huge increase in performance, scalability, 200+ users, most major IT firms and others

Also non-testing applications – modelling and simulation, genome


Failure triggering interactions

Failure-triggering Interactions

  • Additional studies consistent

  • > 4,000 failure reports analyzed

  • Conclusion: failures triggered by few variables


Nist acts tool

NIST ACTS Tool

Covering array generator

Coverage analysis - what is the combinatorial coverage of existing test set?

.NET configuration file generator

Fault characterization - ongoing

http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html

Current users

approximately 200 users as of July 2009, in IT, defense, finance, telecom, and many other industries


Defining a new system

Defining a New System


Variable interaction strength

Variable Interaction Strength


Constraints

Constraints


Covering array output

Covering Array Output


  • Login