Course Notes Set 9: Functional Testing

1. Course Notes Set 9:Functional Testing Computer Science and Software Engineering Auburn University

2. Functional Testing • Dynamic (running) black-box (blindfolded) testing • Also known as behavioral testing • Based on specification - if one not available, the software is the spec • Exam inputs/outputs: needs test cases

3. Purpose of Testing • Test-to-pass - make sure the software minimally works - don’t push it to the limit - apply simplest and/or straightforward cases - not to find bugs, initially - do this test FIRST

4. Purpose of Testing • Test-to-fail - after test-to-pass - design and run test cases with the purpose to break the software - probe the known and unknown weaknesses - errors forcing

5. Functional (Black Box) Testing • Knowing the specified function (requirements), design test cases to ensure that those requirements are met. • Example : Sort (list); • Structural Testing - How well is the code exercised? • Functional Testing - How well does Sort perform its intended function? • In general, complete functional testing is not feasible • Attempting to test every possible input to the function • A randomly selected set of test cases is statistically insignificant • “Not all test cases are created equally” • Test case selection • Based on characteristics of input and output sets relative to specified functionality.

6. Functional Testing • Types of errors looked for during functional testing • Incorrect function or missing function • Interface errors • External database errors • Performance errors (including stress testing) • Initialization/termination errors • Tests are designed to answer the following questions • How is functional validity tested? • What classes of input will make good test cases? • Is the system particularly sensitive to certain input values? • How are the boundaries of a data class isolated? • What data rates and data volume can the system tolerate? • What effect will specific combinations of data have on system operations? [Adapted from Software Engineering 4th Ed, by Pressman, McGraw-Hill, 1997]

7. Goals and Methods of Functional Testing • Goals • Produce test cases that reduce the overall number of test cases • Generate test cases that will tell us something about the presence or absence of errors for an entire class of input • Methods/Approaches • Equivalence partitioning • Boundary value analysis • Matrix of functional possibilities • Cause-effect graphing • Decision Tables

8. Equivalence Partitioning • It is impossible to test all cases • Equivalence partitioning provides a systematic means for selecting cases that matter and ignoring those that don’t • An equivalence class or equivalence partition is a set of test cases that tests the same aspect or reveals the same bugs e.g., If X >= 15 then do-this else do-that (- 15) 15 (15 )

9. Equivalence Partitioning • Equivalence partitions – groups for similar inputs, outputs, and/or operation of the software e.g., file-name, 1 .. 255 characters - valid characters - invalid characters - valid length - invalid length

10. Equivalence Partitioning e.g., copy operation - copy menu - c or C - Ctrl-c or Ctrl-Shift-c • Fully tested in the first effort, equivalence partitioning (1 case each) test for new versions • Goal: to reduce the set of possible test cases • Too few partitions => may not reveal all catchable bugs

11. Equivalence Partitioning • Equivalence partitioning divides the input domain of a program into classes of data from which test cases can be derived • Ideally, each test case could uncover classes of errors, thereby reducing the total number of test cases that must be developed • Input condition - some kind of condition placed on the input • Typically a specific value, a range of values, a set of related values, or a Boolean condition • Equivalence Class - a set of valid or invalid states for input conditions • Range - 1 valid and 2 invalid equivalence classes are defined • Specific Value - 1 valid and 2 invalid equivalence classes are defined • Set - 1 valid and 1 invalid equivalence class are defined • Boolean - 1 valid and 1 invalid equivalence class are defined [Adapted from Software Engineering 4th Ed, by Pressman, McGraw-Hill, 1997]

12. Example Equivalence Classes: (1) Inputs where input1 is in the list (2) Inputs where input1 is not in the list Specific Input Partitions: Listinput1 Empty ? One element In the list One element Not in the list >One element First element >One element Last element >One element Middle element >One element Not in list Test Cases Listinput1Output <nil> ? false bird bird true bird fish false bird, cat, owl bird true dog, pig, chicken chicken true ... Ϭ¹¹¹¹¹¹¹¹¹ Þßàfunction in_list (input1:name_type; ϧÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏÏinput_list : list_names) ϧÏreturn boolean is Ϫ˹¹¹¹¹¹¹¹ ÏϧÏíÏp : list_names; Ïϧbegin ÏϨ¹¹Ïp := input_list; ÏϨ¹¹±while not(p = null) loop ÏϧÏÏ·¹³´if Ada.Strings.Fixed.Index ÏϧÏϯϵ§(Source => String(p.name), ÏϧÏϯϵ§Pattern => String(input1)) /= 0 then ÏÂÄÏϯϵ¾¹¹Ïreturn true; ÏϧÏϯ϶´else ÏϧÏϯϸ¾¹¹Ïp := p.next_name; ÏϧÏϯÏÈÏend if; ÏϧÏÏ°end loop; ÏÂĹ¹Ïreturn false;

13. Boundary Value Analysis • Range : a..b • Example : 100..200 • Test cases : 99, 100, 101, 199, 200, 201 • Number of values • Test cases that exercise minimums and maximums • Apply the above to the output conditions • Try to drive output to invalid range • Internal data structures with boundaries • Example : A(1..100) with test cases A(0), A(1), A(2), A(99), A(100), A(101) • A(0) and A(101) should generate exceptions ( ( ) ( ) a b

14. Boundary Condition Test Cases • If software can operate on the edge of its capabilities, it will almost certainly operate well under normal conditions • For I = 1 to 10 data (I) = -1; end; 10 elements, data(0), data(1), data(2) data(9), data(10), data(11)

15. Boundary Condition Test Cases • Boundary conditions for a legitimate triangle • Boundary conditions for side classification • Boundary conditions for angle classification • Valid input/extremes

16. Boundary Condition Test Cases • Types of Boundary conditions numeric character position quantity speed location size Also, extremes first/last min/max start/finish over/under empty/full Shortest/longest slowest/fastest largest/smallest …

17. Boundary Condition Test Cases • Partitions - boundary - one or two valid points inside the boundary - one or two invalid points outside the boundary e.g., First – 1 / Last + 1 Smallest –1 / Largest + 1

18. Sub-boundary Conditions • Also known as Internal boundaries • Bit, nibble, byte, word, K, M, G, T • Why? E.g., 256 commands, 15 are frequently used. Needs only a nibble.  0XXXX nibble, 1XXXXXXXX byte

19. Sub-boundary Conditions • ASCCI table – boundaries not obvious • Default, empty, blank, null, zero, none (may be of a separate equivalence partition and treated individually) • Invalid, wrong, incorrect, garbage data (test to fail)

20. Matrix of Functional Possibilities • Input/Output Conditions • If the number of combinations of input/output is manageable, then consider using a matrix of functional possibilities • Especially useful if the input/output combinations are enumerated in the requirements specification • Example : Input (or output) will be a combination of {A,B} and {x,y,z}

21. Example : The Triangle Problem • Input • 3 floating point numbers • Processing • Determine if the 3 numbers form a triangle • If not, print message “Not a Triangle” • If it is a triangle • Classify according to side : equilateral, isosceles, scalene • Classify according to largest angle : acute, right, obtuse • Output • List the 3 numbers • List the classification or “Not a triangle”

22. MFP for the Triangle Problem Additional Functional Test Cases (if any):

23. Cause Effect Graphing • Causes (input conditions) and effects (actions) are listed for a module, and an identifier is assigned to each • A cause-effect graph is developed • Looking for causes without effects • Looking for effects without causes • The graph is converted to a decision table (if a decision table has been used as a design tool, developing the graph and table is not necessary) • Decision table rules are converted to test cases

24. Cause-Effect Graph Symbology Symbology Constraints Identity e1 c1 a a a E b O I “Not” c1 e1 b b c Exclusive Only One c1 Inclusive “And” e1 c2 a a R M c1 b b “Or” e1 Requires Masks c2

25. Cause-Effect Graphing Example • The CHANGE subcommand - used to modify a character string in the “current line” of the file being edited • Inputs • Syntax : C /string1/string2 • String1 represents the character string you wish to replace • 1-30 characters • Any character except ‘/’ • String2 represents the character string that is to replace string1 • 0-30 characters • Any character except ‘/’ • At least one blank must follow the command name “C”

26. Cause-Effect Graphing Example • Outputs • Changed line is printed to the terminal if the command is successful • “NOT FOUND” is printed if string1 cannot be found • “INVALID SYNTAX” is printed if the command syntax is incorrect • System Transformations • If the syntax is valid and string1 can be found in the current line, then string1 is removed and string2 replaces it • If the syntax is invalid or string1 cannot be found, the line is not changed

27. Cause-Effect Graphing Example • Cause 1: The first nonblank character following the “C” and one or more blanks is a ‘/’ • Cause 2: The command contains exactly two ‘/’ characters • Cause 3: String1 has length 1 • Cause 4: String1 has length 30 • Cause 5: String1 has length 2-29 • Cause 6: String2 has length 0 • Cause 7: String2 has length 30 • Cause 8: String2 has length 1-29 • Cause 9: The current line contains an occurrence of string1 • Effect 1: The changed line is typed • Effect 2: The first occurrence of string1 in the current line is replaced by string2 • Effect 3: NOT FOUND is printed • Effect 4: INVALID SYNTAX is printed

28. Complete Cause-Effect Graph c1 e1 c2 c3 i1 c4 e2 i3 c5 c6 e3 i2 c7 c8 c9 e4

29. Converting to a Decision Table