Dynamic Black-box Testing: Key Techniques and Test Cases

1. Chapter 5. 黑盒测试

2. 要点 • 测试用例 • 等价类划分 • 数据测试 • 状态测试 • 其它黑盒测试技术

3. 动态黑盒测试 • Dynamic black-box testing is testing without having an insight into the details of the underlying code. • Dynamic, because the program is running • Black-box, because testing is done without knowledge of how the program is implemented. • Sometimes referred to as behavioral testing(行为测试). • Requires an executable program and a specification (or at least a user manual).

4. Test Cases(测试用例) • Test data: Inputs which have been devised to test the system. • Test cases: • Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification. • are formulated as a set of pairs

5. 通过性测试和失效性测试 • two fundamental approaches • Test-to-pass: • assures that the software minimally works, • does not push the capabilities of the software, • applies simple and straightforward test cases, • does not try to “break” the program. • Test-to-fail: • designing and running test cases with the sole purpose of breaking the software. • strategically chosen test cases to probe for common weaknesses in the software.

6. Discussion … • Why should a tester always start with a test-to-pass approach? • Isn’t this a waste of time? • What assurance does test-to-pass give us? • Shouldn’t the programmers (i.e., not the testers) do test-to-fail?

7. Black-box testing • Characteristics of Black-box testing: • Program is treated as a black box. • Implementation details do not matter. • Requires an end-user perspective. • Criteria are not precise. • Test planning can begin early.

8. Black-box testing

9. 等价类划分 • Equivalence partitioning is the process of methodically reducing the huge (or infinite) set of possible test cases into a small, but equally effective, set of test cases.

10. Search routine specification procedure Search (Key : INTEGER ; T: array 1..N of INTEGER; Found : BOOLEAN; L:1..N) ; Pre-condition -- the array has at least one element 1 <= N Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, 1 >= i >= N, T (i) = Key ))

11. Search routine input partitions • Inputs which conform to the pre-conditions. • Inputs where a pre-condition does not hold. • Inputs where the key element is a member of the array. • Inputs where the key element is not a member of the array.

12. Search routine input partitions

13. Search routine test cases

14. Data Testing(数据测试) • If you think of a program as a function, the input of the program is its domain. • Examples of program data are: • words typed into MS Word • numbers entered into Excel • picture displayed in Photoshop • the number of shots remaining in an arcade game • …

15. Boundary input data(边界测试) • Boundary conditions are situations at the edge of the planned operational limits of the software. • E.g., negative to zero to positive numbers, exceeding the input field length of a form, etc. • Choose input data that lie on the boundary when formulating equivalence partitions. • Test the valid data just inside the boundary • Test the last possible valid data • Test the invalid data just outside the boundary • Security flaws such as buffer overflow attacks exploit boundaries of array buffers.

16. Sub-Boundary Conditions • 次边界测试 • internal to the software aren't necessarily apparent to an end user but still need to be checked • Powers-of-Two

17. Default, Empty, Blank, Null, Zero, and None • may seem obvious is when the software requests an entrysay, in a text boxbut rather than type the correct information

18. Garbage-In Garbage-Out • “Garbage-In equals Garbage-Out” is one of the worst cop-outs ever invented by the computer industry. • GI-GO does not explain anything except our failure to: • install good validation checks • test the system’s tolerance for bad data. • Systems that interface with the public must be especially robust and consequently must have prolific input-validation checks.

19. Million Monkey Phenomenon • A million monkeys sit at a million typewriters for a million years and eventually one of them will type Hamlet! • Input validation is the first line of defense against a hostile world.

20. Input-Tolerance Testing • Good user interface designers design their systems so that it just doesn’t accept garbage. • Good testers subject systems to the most creative “garbage” possible. • Input-tolerance testing is usually done as part of system testing and usually by independent testers.

21. 5.State Testing(状态测试) • verify the program's logic flow through its various states • software state is a condition or mode that the software is currently in

22. Testing Logic Flow • the software's states and logic flow is infinite • is similar to the well-known traveling salesman problem • to apply equivalence partition techniques to the selection of the states and paths, reducing that risk by making intelligent choices

23. State Transition Map(状态转换图) • 绘图技术不重要

24. Reducing the Number of States and Transitions to Test • Visit each state at least once • Test the state-to-state transitions that look like the most common or popular • Test the least common paths between states • Test all the error states and returning from the error states • Test random state transitions

25. 5.2 Testing States to Fail • Race Conditions and Bad Timing(竞争条件和时序错乱) • two or more events line up • Examples • Saving and loading the same document at the same time with two different programs • Sharing the same printer, communications port, or other peripheral • Pressing keys or sending mouse clicks while the software is loading or changing states • Shutting down or starting up two or more instances of the software at the same time • Using different programs to simultaneously access a common database

26. 5.2 Testing States to Fail • Repetition testing（重复测试） • 主要是资源泄露 • Stress（压迫测试） • under less-than-ideal conditionslow memory, low disk space, slow CPUs, slow modems, and so on • Load testing(负载测试，压力测试) • handle thousands of simultaneous connections • Long time • automation

27. 6 Other Black-Box Test Techniques • Behave Like a Dumb User • Look for Bugs Where You've Already Found Them • the more bugs you find, the more bugs there are • Many programmers tend to fix only the specific bug you report • Think like a Hacker • Follow Experience, Intuition, and Hunches

28. You now know … • … test-to-pass test-to-fail testing • … black-box testing • … equivalence partitions • … data testing • … state testing • … syntax testing as a special case of data testing

29. Quiz • True or False: You can perform dynamic black-box testing without a product specification or requirements document. • If you're testing a program's ability to print to a printer, what generic test-to-fail test cases might be appropriate?

30. Quiz • True or False: Visiting all the states that a program has assures that you've also traversed all the transitions among them. • What actions do you perform on software when attempting to expose race condition bugs? • True or False: It's an unfair test to perform stress testing at the same time you perform load testing.

31. 第一次上机实习 • 1.以三角形(Triangle)程序 • 写一个库 • 2.windows的计算器 • 写一个库和一个界面 • 详实的开发文档（包括需求说明书、设计说明书 • 3.Web应用程序 • 可以用上次的课程设计内容 • 要补充需求说明书、设计说明书