Software Testing - a necessary evil

1. Software Testing - a necessary evil SQNZ April 2002 Michael Osborne mike@osborne.gen.nz

2. Does this guy know what he’s talking about? • Testing is about bugs • This guy knows about bugs • In 25+ years of development he’s created thousands of them • Now, I pay other people to do this

4. Simply • Quidquid latine dictum sit, altum sonatur. • Whatever is said in Latin sounds profound. • Sedulously eschew obfuscatory hyperverbosity and prolixity. • Let’s all keep it as simple as possible. • If you don’t understand - interrupt! From - How To Write Unmaintainable Code http://mindprod.com/unmain.html

5. What’s going on here? • Why this presentation? • enquire vs. inform • tomorrow vs. yesterday • quality vs “quality” • continuous improvement • how to get and give good testing bang for buck From - Joel on Software http://www.joelonsoftware.com

6. The Principles Quality Software Quality Quality & Testing The Necessity The Evil Techniques Types of testing Testing 2002 If time permits we’ll do some testing What we will cover

7. Why it’s necessary • Analysts & developers are imperfect • Capers Jones - bug input is fps to power of 1.2 • there is no perfect prevention - yet • you can’t ship (successfully) software that doesn’t work • it’s a process measure • in a highly sophisticated environment • it goes from primarily being a QC tool • to a primarily being process measure

8. Why it’s evil • You can’t test quality in • You can’t test it all • You can’t prove it works • It’s costly and inefficient • The point is to prove it doesn’t work • The more problems found the better

9. Testing Explained • Testing is about finding bugs

10. Which part don’t you understand? • If you’re not going to find any bugs why are you doing it? “A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” • Testing Computer Software by Kaner, Falk, Nguyen

11. It’s an Economics Problem • Cost of Quality = Cost of Conformance + Cost of Non-Conformance • Testing adds to Cost of Conformance • It must directly reduce Cost of Non-Conformance or else it is waste • Testing to do anything other than find bugs is a WOMBAT • a Waste of Money Brains and Time

12. How it works - roughly

13. When CoNC is high... • It’s appropriate to raise the CoC

14. High “quality” software • It is possible to develop very high quality software • An example • NASA space shuttle on-board flight software • Approx. 3 million lines of code • <1 error per 10k LOC after release • Cost: the order of $1000 / LOC

15. The “Q” word • quality vs. “quality” • it’s important we’re clear about this • it will show the necessity for Software Testing • it will expose some evils of Software Testing • So, what is quality? • And, what is “quality”?

16. What is software development? • Putting bugs into software • ie. hiding bugs in software • Finding the bugs • Taking them out again • Putting (hiding) new / different bugs in when you take others out • Finding those bugs • Taking them out again • And so on ...

17. What is better software development? • Putting fewer bugs into software • exposing bugs in software • Finding the important bugs faster • Taking important bugs out again • assessing the associated risk • Putting fewer new / different bugs in when you take others out • And so on … • Today’s focus is on Software Testing

18. What is better software testing? • What is quality software testing? • Quality is … • conformance to requirements • Note: test cases are not test requirements • Finding the important bugs faster? • which leads to the critical question -

19. When is testing done? • Typically - • when the testing schedule ends • when all the test cases have been exercised • when all the bugs have been found and fixed • If you don’t have a set of testing requirements • How do you answer the question?

20. The Zero Defects Approach • Quality = conformance to requirements • Test exhaustively • Find all defects • Fix all defects • Result - quality software • a quick digression into Zero Defects • a personal view

21. Direction vs. Destination • Current state - some level of defects • Desired state • more defects • same level of defects • fewer defects • An asymptote • Note “six sigma” is not zero • “Zero defects” is a pointer and an accelerator • use it that way

22. Uh oh - there’s more than one requirement!! • Quality = conformance to requirements • There are generally three sets of requirements • Functionality • Schedule • Quality • The bad news • they compete

23. The Good-Enough Software Triangle • Conformance to all requirements Schedule Delivery Date Functionality Features Quality Absence of Defects

24. Software Testing - A Wicked Problem • A “mess” is a complex problem with a “simple” solution • A “wicked problem” is a “simple” problem with a complex solution • usually because of stakeholder differences • Each project (testing problem) potentially has a different solution

25. So? • How do we do it?

26. Software Testing 2002 • Context-driven testing • all at www.satisfice.com • Exploratory testing • heuristics • short bursts • focus on results • purposeful wandering • you must think

27. white box - glass box, structural path testing - coverage black box top-down, bottom-up unit function system integration regression performance alpha beta compatability quicky port usability exploratory hallway others ... Testing Types

28. Later Test Phase Middle Phase ? ? Frequent Annoyances - Remove Ignore Frequent Niggles - ? Bug removal strategy Important Find & Fix Immediately Ignore Urgent

29. Usability Testing • Graphical User Interfaces coupled with complex processes create usability issues • what’s clear and obvious to designer/programmer is meaningless to users • Doesn’t require a lot of people • 5 to 7 is adequate • Observe, and listen • understand how they see things • why they do what they do • Is your interface clear and consistent?

30. Why usability is critical • It doesn’t matter if it processes everything right • if they don’t like it (can’t/won’t use it, it failed) • Prototype, prototype, prototype • Consistent vs. clever • at all times • Users will be forgiving with a highly usable system • ask Microsoft

31. Hallway Testing • A simple form of usability testing • Open up development to interact more with users • Approach a person (in your organisation) at random • invite for 5-10 minute testing session • see what they do with your software from no background • is it understandable, usable

32. The evil necessity • It’s all about finding bugs • A good test is a test that finds a bug • Be clear about what represents quality • situation by situation • Establish the context • and be guided by it • Good software testing is a challenging intellectual process. • Go for it!