Exercise: Using The Pairwise Tool PICT

Exercise: Using The Pairwise Tool PICT Philippe CHARMAN charman@fr.ibm.com http://users.polytech.unice.fr/~charman/ Last update: 10 April 2015

Preliminary • Suppose a function f takes five Boolean variables as arguments: x1, x2, x3, x4 and x5. • Boolean values are expressed as 0 and 1 for simplicity reason • Design the smallest test suite that meets the pairwise property

Overview PICT • PICT is a pairwise testing tool developed by Microsoft • PICT is the acronym of Pairwise Independent Combinatorial Testing • Used to design test cases and test configurations for software systems • PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.

Installing PICT • Install PICT from pict33.msi • Update the environment variable PATH • Help in PICTHelp.htm

Model files • A model is composed of 3 sections: parameter definitions [sub-model definitions] [constraint definitions] • Model sections should always be specified in the order shown above and cannot overlap • The sub-model and the constraint sections are optional • Sections do not require any special separators between them • Empty lines can appear anywhere • Comments can be included by prefixing lines with the “#”character.

Simple Model • Syntax of the parameter definitions <ParamName> : <Value1>, <Value2>, <Value3>, ... • Sample model for testing volume create/delete functions Type: Primary, Logical, Single, Span Size: 10, 100, 500, 1000, 5000, 10000 Format method: quick, slow File system: FAT, FAT32, NTFS Cluster size: 512, 1024, 2048, 4096, 8192, 16384 Compression: on, off

Generating Test Cases • Given this model C:\>more mymodel.txt X: 1, 2 Y: A, B, C Z: 1, 2, by default pairwise test cases are generated: C:\>pict mymodel.txt X Y Z 2 B 2 1 A 1 2 C 1 2 A 2 1 B 1 1 C 2

Configuration test cases • Suppose we want to test on different platforms: C:>more conf.txt OS: x86, x64 Win: Win7, Win8 C++: msvc10, msvc11 Java: JDK6, JDK7 by default: C:\>pict conf.txt OS Win C++ Java x64 Win7 msvc10 JDK6 x64 Win8 msvc11 JDK7 x86 Win8 msvc10 JDK7 x86 Win7 msvc11 JDK7 x86 Win8 msvc11 JDK6

Conditional constraints • A term [parameter] relation value is an atomic part of a predicate. The following relations can be used: =, <>, >, >=, <, <=, and LIKE. LIKE is a wildcard-matching operator (* - any character, ? – one character). [Size] < 10000 [Compression] = "OFF" [File system] like "FAT*" • Operator IN allows specifying a set of values that satisfy the relation explicitly: IF [Cluster size] in {512, 1024, 2048} THEN [Compression] = "Off"; IF [File system] in {"FAT", "FAT32"} THEN [Compression] = "Off";

Conditional constraints • Parameters can be compared to other parameters, as in this example: # Machine 1 OS_1: Win2000, WinXP SKU_1: Professional, Server, Datacenter, WinPowered LANG_1: EN, DE # Machine 2 OS_2: Win2000, WinXP SKU_2: Professional, Server, Datecenter LANG_2: EN, DE IF [LANG_1] = [LANG_2] THEN [OS_1] <> [OS_2] AND [SKU_1] <> [SKU_2];

Unconditional constraints (Invariants) • An invariant declares an always-valid limitation of a domain: # At least one parameter must be different to be a meaningful test case (use the OR operator) [OS_1] <> [OS_2] or [SKU_1] <> [SKU_2] or [LANG_1] <> [LANG_2]; # All parameters must different (use the AND operator) [OS_1] <> [OS_2] and [SKU_1] <> [SKU_2] and [LANG_1] <> [LANG_2];

Parameter Types • PICT uses the concept of parameter types. There are two types of parameters: string and numeric. A parameter is considered numeric only when all its values are numeric. If a value has multiple names, only the first one counts. Types are only important when evaluating constraints. You can only compare a numeric parameter to a number, and a string parameter to another string. For example: Size: 1, 2, 3, 4, 5 Value: a, b, c, d IF [Size] > 3 THEN [Value] > "b"; • String comparison is lexicographical and case-insensitive by default. You can make comparisons case-sensitive if you specify the /c option. Numeric values are compared as numbers.

Exercise • An application is available in 32-bit and 64-bit and can be used on the following platforms: • Windows 7 – x86 and x64 • Windows 8 – x86 and x64 • AIX 6.1 - Power7 32-bit and 64-bit • AIX 7.1 – Power7 32-bit and 64-bit • The following locales are supported: EN, CN, JP, DE, FR • The following databases are supported: Oracle 9i, Oracle 10g, Oracle 11g, DB2 9.1, DB2 9.7

Writing the model • Write a model to generate pairwise test cases • Trivial constraints: • The 32-bit application should be tested on the 32-bit and 64-bit architectures. • The 64-bit application cannot be tested on a 32-bit architecture • About 27 test cases should be generated

Configuration test cases • In the solution proposed: OS Win C++ Java x64 Win7 msvc10 JDK6 x64 Win8 msvc11 JDK7 x86 Win8 msvc10 JDK7 x86 Win7 msvc11 JDK7 x86 Win8 msvc11 JDK6 • No tests are done on x64/Win7 to test msvc11 or x64/Win8 to test msvc10 • Testing the pair (OS, Win) is more important than testing the pair (C++,Java)

Sub Models • Syntax of the sub-model definitions: {<ParamName1>, <ParamName2>, ... } @ <Order> • Example: PLATFORM: x86, ia64, amd64 CPUS: Single, Dual, Quad RAM: 128MB, 1GB, 4GB, 64GB HDD: SCSI, IDE OS: NT4, Win2K, WinXP, Win2K3 IE: 4.0, 5.0, 5.5, 6.0 APP: SQLServer, Exchange, Office { PLATFORM, CPUS, RAM, HDD } @ 3 { OS, IE } @ 2

How the generation would look for the previous model $ | | order = 2 (defined by /o) | +------------------------------+-----------------------------+ | | | | order = 3 | order = 2 | | | | { PLATFORM, CPUS, RAM, HDD } { OS, IE } APP

Sub Models • You can define as many sub-models as you want; any parameter can belong to any number of sub-models. Model hierarchy can be just one level deep • The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the first sub-model can be any value between one and four • If you do not specify the order for a sub-model, PICT uses the default order or the order specified by /o option

Sub Models C:\> more conf.txt OS: x86, x64 Win: Win7, Win8 C++: msvc10, msvc11 Java: JDK6, JDK7 {OS,Win} @ 2 C:>pict conf.txt OS Win C++ Java x86 Win8 msvc10 JDK7 x86 Win7 msvc11 JDK7 x86 Win7 msvc10 JDK6 x64 Win7 msvc11 JDK7 x86 Win8 msvc11 JDK6 x64 Win7 msvc10 JDK6 x64 Win8 msvc10 JDK6 x64 Win8 msvc11 JDK7

Sub Models C:\> more conf2.txt x:0,1 y:2,3 z:4,5 t:6,7 {x,y} @ 2 {z,t} @ 2 x,y: S1 = {(0,2), (0,3), (1,2), (1,3)} z,t: S2 = {(4,6), (4,7), (5,6), (5,7)} The solution for this model is the Cartesian product S1xS2

Sub Models C:>pict conf2.txt x y z t 1 2 5 7 0 2 4 6 0 2 5 7 0 2 4 7 0 3 5 6 1 2 4 7 0 3 4 6 0 3 5 7 0 2 5 6 1 3 5 7 1 3 4 7 1 2 5 6 1 3 5 6 1 2 4 6 1 3 4 6 0 3 4 7

Defining a sub-model • In the model, express the pair (Application, Architecture, OS) as a sub-model • How many test cases are now generated ?

Trying to reduce the test case • Still too many test cases • Among the locales, we need to test EN above all. Other locales should be tested only once on only platform • Among the databases, we need to test DB2 9.7 above all. Other databases should be tested only once on any platform • Is it possible to express these constraints in PICT ?

Randomized generation • C:\more try.txt x:0,1 y:0,1 z:a,b t:a,b {z,t} @1

Randomized generation C:\>pict try1.txt /r:1 Used seed: 1 x y z t 1 0 a b 0 1 b a 1 1 b a 0 0 b a 0 1 a b C:\>pict try1.txt /r:1000 Used seed: 1000 x y z t 0 1 b b 1 0 b b 0 0 a a 1 1 a a

Generating Test Cases • N-wise test cases can be generated with /o:N C:\>pict mymodel.txt /o:3 X Y Z 1 B 1 2 B 2 1 C 2 1 B 2 2 C 2 2 B 1 2 A 1 1 A 1 1 A 2 1 C 1 2 A 2 2 C 1

Further reading • More information on PICT: http://msdn.microsoft.com/en-us/library/cc150619.aspx • Many resources on pairwise testing: http://www.pairwise.org/ The site is maintained by Jacek Czerwonka who developed PICT

Exercise: Using The Pairwise Tool PICT