Introduction to Software Testing

Introduction to Software Testing

Outline • Introduction to Software Testing • Types of Software Testing • Unit Testing Strategies • Equivalence Class Testing • Boundary Value Testing • Output Testing • System Testing • Usability Testing

What is Software Testing?

What is Software Testing? • software testing: Evaluating software by observing its execution • Primary goal of testing is to find bugs! • During testing, should execute tests that find bugs. • “Testing can only prove the presence of bugs, not their absence.” - Dijkstra • When do you stop testing?

Test Terminology • test case: A set of inputs, execution conditions, and a pass/fail criterion. • Example: height input is 5, width input is “Hello”, the test passes if the program produces an error message. • test or test execution: The activity of executing test cases and evaluating their results. • test suite: A set of test cases.

More Test Terminology • defect: A mistake in the source code (static) • Synonyms: fault, bug • failure: An observable incorrect program behavior (output) • debugging: The process of finding and correcting a defect in a program given a failure

Software Testing is Hard! Why?

Testing Inputs // Solve ax2 + bx + c = 0 struct RootAnswer { float root_one; float root_two; } RootAnswer roots(float a, float b, float c); • Which values for a, b, c should we test? • Need to select test cases intelligently!

Test Case Selection • Due to time constraints, only a small subset of actual inputs can be used to test. • Need to choose a test suite such that… • Each “functionality” provided by a program is tested by at least one test case. • Tests target common programming mistakes. • Tests exercise areas of code that are likely to be “buggy”.

Other Techniques for Finding Bugs • Code Inspection: Looking at the source code directly for bugs. • Tools • compiler • static analyzer • memory leak detector • profiler / performance monitors

Functional vs. Non-Functional Testing • Functional testing: Testing to determine if the software is working properly based on the specification (has the proper function). • Non-functional testing: Testing other desirable properties of the software. • Examples: performance, usability • Testing is vastly different.

Unit Testing • Unit Testing: Testing of a “unit” in the program. • Unit: smallest part of the program – only assigned to one developer. • Often a single function or class. • Focuses on the functionality of that unit. • Can be automated using unit testing tools • Visual Studio • Junit • Google Testing Framework

Integration Testing • Integration Testing: Testing the interactions of the different units. • Assumes unit testing is complete or will catch errors within the unit. • Focuses on the interactions between units. • Can be tested manually or can be automated.

System Testing • System Testing: Testing the entire system. • Requires the entire system to be present. • Used to test common usage scenarios. • Used to test non-functional requirements. • Best tested manually. • Testing approaches are vastly different for non-functional requirements.

Unit Testing Strategies • Derive tests from the specification of the unit. • Need to choose tests… • Intelligently • Systematically • Thoroughly • Consider the unit under a test as a “function” with inputs and outputs • All unique and interesting inputs should be tested • All unique and interesting outputs should be produced • Focus on common errors

Equivalence Class Testing • Equivalence Class Testing: Partition the input space into equivalence classes. • The key is to divide the input space into equivalence classes such that all inputs within each equivalence class have the same functionality. • Very likely that each input within a equivalence class will exercise the same code within the function. • Need to have at least one test in each equivalence class. • More tests are better.

Equivalence Class Testing Failures are sparse in the space of possible inputs ... ... but dense in some parts of the space If we systematically test some cases from each part, we will include the dense parts Failure (valuable test case) No failure The space of possible input values Functional testing is one way of drawing orange lines to isolate regions with likely failures

Equivalence Class Testing • The division into equivalence classes is based on the desired / specified functionality of the subject under test. • Equivalence classes may consist of: • one particular value • a range of values • a subset (possibly discontinuous) set of values • The set of equivalence classes should include both valid and invalid inputs.

Equivalence Class Testing Examples • Volume discount if you buy more than 1000 units. • Function that returns the number of days in a month (1-12). • If the area code is 800 or 888, there is no charge for the call. • Shipping rates differ depending on what country you are in.

Class Problem: Equivalence Class Testing Identify equivalence classes to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from 0-100.

Boundary Value Testing • Boundary Value Testing: Test the values near and at the boundary of an ordered input. • For ordered inputs, history suggests that values near the boundary are most error prone. • Programmers may forget to check for a boundary condition • Programmers may code the boundary incorrectly (use < instead of <=) • Special case or different code may be used at the boundary Possible test case

Boundary Value Testing • For each boundary n, want to have three cases: n – 1, n, n +1 • For a range of values from x to y, have at least six cases: x – 1, x, x + 1, y – 1, y , y + 1 • Boundary value testing can be combined with equivalence class testing: • Boundaries mark the endpoints of equivalence classes. • Need to test the boundaries of each equivalence class. • Still desirable to test one or more values in the middle of the range.

Boundary Value Testing • Some values are outside of the range: • May also be part of another equivalence class (worry about redundancy later) • May be invalid – should check to make sure it’s invalid • Boundary analysis can be applied to more situations than handling numeric data: • string length • size of arrays (or other data structures) • data type limitations (size of integer)

Class Problem: Boundary Value Testing Identify boundary values to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from 0-100.

Output Testing • Equivalence class and boundary value testing can also be applied to the output of a function. • Outputs can be divided into equivalence classes. • At least one test should be executed for each output equivalence class. • In many cases, but not all, tests created for output are redundant to equivalence class tests created for inputs. • Boundaries of an ordered output should also be tested.

System Testing • System Testing focuses on the whole system: • Consists of not only software, but possibly hardware and other components. • Necessary to understand the solution / product as a whole. • Encompasses functional and non-functional testing: • Functional: Does the software work? • Non-functional: Performance, usability, security, etc.

Scenario-Based Testing • Scenario-Based Testing: Develop a scenario or a script that is important to execute. • Scenarios tend to mimic realistic use of the system. • How would a normal user use the software? • Include invalid input scenarios: • How would a user accidentally or intentionally misuse the system?

Exploratory Testing • Exploratory Testing: Test the program without constraint. • Also called ad-hoc testing. • No script or scenario. • Allows the tester to creatively try tests as they explore the application.

Usability Testing What makes a software product usable?

Usability Testing • Usability Testing: Tests the usability of a software product. • Typically done with representative samples of real users to try out the software. • Objective criteria: • Time and number of operations to perform a task • Frequency of user error • blame user errors on the product!

Basic Usability Features • Ask questions with the user in mind: • Does the user interface make sense? • Are the prompts / messages clear? • Is the response appropriate when an invalid input is entered? • Is the program intuitive? • May want to have someone outside the class try out your programs. • Don’t let them read the specifications.

Introduction to Software Testing