Contract Checking and Automated Test Generation With Pex

1. TL51 Contract Checking and Automated Test Generation With Pex  Mike BarnettRSDE Microsoft Research  Nikolai TillmannRSDE Microsoft Research

2. Two Tools. One Talk. Code ContractsFor .NET  Mike Barnett RSDE Microsoft Research Automated TestGeneration WithPex  Nikolai Tillmann RSDE Microsoft Research

3. Contract Library in .NET 4.0 announcing Code Contracts for .NET Coming soon: Tools for contracts at

4. How Do You Know What It Does? string s = o.GetDescription(i); … s.Length … stringGetDescription(int x); /// <param name=“x”> /// should be greater than zero /// </param> /// <returns> /// non-null string /// </returns> stringGetDescription(int x);

5. Let’s Fix It For Once And All! Design-by-Contract meets .NET! stringGetDescription(int x) { Contract.Requires( 0 < x ); Contract.Ensures(Contract.Result<string>() != null ); … }

6. Contracts • Contracts document design decisions • Preconditions, postconditions, invariants • Facilitate code evolution • Contracts are executable • Checked documentation • Test oracles • Contracts help static verification • Early error detection • Clarify responsibility at API boundaries • Assertions help, but everyone has their own flavor • Difficult to tool against • Not seen as a contract between 2 parties

7. demo Introduction  Mike Barnett RSDE Research in Software Engineering

8. What Can I Do With Contracts? classRational {public Rational(int n, int d) {Contract.Requires( 0 < d );this.N = n;this.D = d; }} Static Checking Runtime Checking Test Generation (Pex) Documentation

9. What Contracts Can I Write? • Requires • What must be true at method entry • Ensures • What must be true at method exit • Invariants • What must be true at all method exits • Assertions • What must be true at a particular point • Assumptions • What should be true at a particular point

10. What Can I Put In A Contract? • Any boolean expression • In your favorite programming language! • Including method calls (but must be marked Pure) • Quantifiers • Contract.ForAll(0,A.Length,i => A[i] > 0); • Contract.Exists(0,A.Length,i => A[i] > 0); • Contract.Result • refer to the return value of the method • Contract.OldValue • refer to values at method entry

11. How Do I Write A Contract? publicvirtualint Add(object value){ Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if(count == items.Length) EnsureCapacity(count + 1); items[count] = value; returncount++;} • Features • Declarative • Language expression syntax • Type checking / IDE • Special Encodings • Result and OldValue voidObjectInvariant() {Contract.Invariant( items != null );}

12. demo Runtime Checking  Mike Barnett RSDE Research in Software Engineering

13. How Does It Work? publicvirtualint Add(object value){ Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++; } Executable Runtime Contract Checking .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { .locals init (int32 'Contract.Old(Count)', int32 'Contract.Result<int>()') ldarg.0 call instance int32 TabDemo.BaseList::get_Count() stloc.3 ldarg.1 ldnull ceq ldc.i4.0 ceq ldstr "value != null" call void __RewriterMethods::RewriterRequires$PST06000009(bool, string) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue IL_004d nop ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) nop nop ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br IL_0072 ldloc.0 stloc.s 'Contract.Result<int>()' br IL_007a ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldloc.3 ldc.i4.1 add ceq ldstr "Count == Contract.Old(Count) + 1" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ldloc.s V_4 ceq ldstr "Contract.Result<int>() == Contract.Old(Count)" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ret } csc/vbc/… .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret } // end of method BaseList::Add ReleaseCompile /d:CONTRACTS_FULL csc/vbc/… ccrewrite .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0029 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_004b ldloc.0 ret } • Inheritance • Style checks

14. Static Contract Checking • No silver bullet • But helps catch errors earliest • Best used in a focused manner • Guides development • Discovers implicit assumptions • Propagates assumptions • Not only explicit contracts • Dereferencing null • Indexing arrays • Arithmetic exceptions

15. What Do You Ship? src src src src ReleaseAssemblies Contract ReferenceAssemblies PowerLib.dll PowerLib.Contracts.dll + (minimal runtime checks) All contracts, no code

16. Contract Reference Assemblies publicvirtualint Add(object value){ Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++; } Contract ReferenceAssembly .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret } // end of method BaseList::Add .method public hidebysignewslot virtual instance int32 Add(object 'value') cil managed { ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) } // end of method BaseList::Add ccRefGen Compile /d:CONTRACTS_FULL

17. Interface Contracts [ContractClass(typeof(CloneableContract))]publicinterfaceICloneable { object Clone(); } [ContractClassFor(typeof(ICloneable))]publicclassCloneableContract : ICloneable { publicobject Clone() {Contract.Ensures( Contract.Result<object>() != null ); … } } All classes implementing the interface inherit the contract

18. Code Contracts Summary • Contract library class enables contracts in all .NET languages • No restrictions on what can be expressed • Contracts are being used in the BCL today • Contract library is a core component of .NET 4.0 • Same contracts used for • Runtime checking • Static checking • Documentation generation

19. Get Pex for Visual Studio 2010 at announcing Do you care about quality?Shake your code with Pex. Pex

20. Testing Before Pex • Testing is tedious • Too easy to miss cases • Old tests get stale • Too much legacy code − what does it do? Don’t give up, Pex has arrived.

21. demo Tests for Freewith Pex Pex  Nikolai Tillmann RSDE Microsoft Research

22. What The Demo Showed • It’s painless: Just right-click your code • It’s effective: Test cases for free • It’s fun: • Inspect table, • Save tests, • file work items, or • debug right away

23. demo Code Coverage for Free with Pex Pex  Nikolai Tillmann RSDE Microsoft Research

24. What The Demo Showed • Pex automatically generatedcomprehensive test suitewith high code coverage • Pex finds contract violations • The generated test suite • integrates withVisual Studio Team Test • can be used asBuild Verification Test (BVT)

25. There Is More… • A traditional Unit Test is amethod without parameters. Handwritten [TestMethod] voidAddZeroTest() { var r = new Rational(1,2); var q = r + 0; Assert(r == p); }

26. There Is More…Parameterized unit testing with pex • A Parameterized Unit Test is simply atest method that takes parameters • Pex generates traditional Unit Tests (in C#) which fix particular values Handwritten Automatically generated Pex [PexMethod] voidAddZeroTest( Rational r) { var q = r + 0; Assert(r == p); } [TestMethod] void AddZeroTest01() { var r = new Rational(1,2); this.AddZeroTest(r); } calls

27. A Tester's Nightmare: ResourceReader • How to test this code?(Actual code from .NET base class libraries)

28. A Tester's Nightmare: ResourceReader

29. Parameterized Unit Tests • [PexAllowedException(typeof(ArgumentException))] • [TestClass] • public partial classResourceReaderTest{ • [PexMethod] • public void ReadEntries(byte[] data) { • if (data == null) return; • fixed (byte* p = data) • using (varstream = • new UnmanagedMemoryStream(p, data.Length)) { • varreader = new ResourceReader(stream); • foreach(varentry in reader) • { /* reading entries */ } • } • } • }

30. demo Parameterized Unit Testing Pex  Nikolai Tillmann RSDE Microsoft Research

31. PexUnderstands The Code • Pex does not guess. • Pex does not generate random inputs, • enumerateall possible values, or • make you write test input generators • Instead, Pex analyzes your .NET code. • Pexpartitions inputs into equivalence classes • One equivalence class per branching behavior • Test inputs computed by Z3, the world classconstraint solver for program analysisfrom Microsoft Research • Precise inter-procedural, path-sensitiveanalysis • As a result, you geta small test suite with high code coverage

32. Pex Summary • Pex generates small test suites with high code coverageand bug reports for free • Use previously generated test suites asBuild Verification Test (BVT) • Reduce test maintenance costsby parameterized unit testing • Pex has been used in Microsoftto test core .NET components • Download Pex for Visual Studio 2010 CTP onhttp://msdn.microsoft.com/DevLabs

33. Summary • Code Contracts for .NET:preconditions, postconditions, invariants http://research.microsoft.com/Contracts/ • Pex: test generation for .NET http://research.microsoft.com/Pex/ Research in Software Engineering

34. Evals & Recordings Please fill out your evaluation for this session at: This session will be available as a recording at: www.microsoftpdc.com

35. Q&A Please use the microphones provided

36. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.