Why?

1. Why? When we test we are trying to answer a big question: • Can we release this (system, feature, code)? This question can be broken down into many sub-questions: • Have we tested all the code? • Have we tested all the features? • Have we found all the bugs we expected to find? • Have we fixed all the bugs we need to fix? Definition: the exercise of some system, feature, component, or code such as to exhibit a bug Test Management Summit 2007 2

2. Feature coverage Question: what is a feature? • A1: anything stakeholders say it is • A2: something users can use to do a job • A3 some collection of screens and screen icons which users use to fulfil some unique task Test Management Summit 2007 3

3. Feature definitions Feature definitions vary from the ludicrous (verbal assertions) through user stories, to the highly-structured (UML). Somewhere in between is text. Any definition is good if it lets you: • Measure: • how many there are • how big each is • how important each is to the user • Test them in any combination such as to exhibit bugs • Identify inconsistencies • Know the limits of coverage Test Management Summit 2007 4

4. Problems (you knew we’d get here sometime) The existence and extent of the feature depends on: • Who is defining it (management, users, sales staff, business analysts, tech. writers. Er, testers?) • How they are defining it (text, UML, memoranda, backs of envelopes, user stories) • How it is expected to be used, and by whom Lehman:the existence of the tool changes the nature of the job Test Management Summit 2007 5

5. Test coverage by structure • You need to be sure the developers have exercised some minimum part of the code using several approaches. • Code coverage has been the subject of much academic interest: there are lots of measures: • Lines • Decision points • Definition-Use paths • Linear Code Sequence and Jumps Great hammers for great nails Test Management Summit 2007 7

6. Test coverage by feature • Do we have a test (set) for every feature plus installation, deinstallation, start-up and shut-down? • Can we decompose every feature into testable sub-bits? Has every one got a test? Does every test include at least one negative case? Objection: • No we don’t have any spec. worthy of the name nor any time to write it. Test Management Summit 2007 8

7. Test coverage by GUI icon • The user interface has a number of screens, buttons, pull-downs, tabs, menus etc. Do we have them all listed, and with tests which exercise every one? • Objections: • Q: d’you know how many such icons there are in the application? A: if it’s that big, you need a list to be sure you’ve hit them all • Q: just pulling down icons, hitting buttons and entering text in fields doesn’t exercise the system. A: true but if any of those don’t work you need to know asap. Test Management Summit 2007 9

8. Test coverage by scenario Users have goals they want to achieve. • They achieve them using a number of (parts of) features. This sets up subtle feature interactions which no other coverage approach will mimic. • Problems: • P: The system is new and the users don’t know how they will use it. S: use a model office or a simulator. • P: The release adds new features - have we got to test all the old ones as well? S: What is the risk if they interact in unexpected ways and you haven’t tested all these ways? (aka “yes”) • P: User management doesn’t want us to talk to users. S: Have management define a manager as a model user and put him or her in the model office. Test Management Summit 2007 10

9. Test coverage by transition • Web and conventional applications have “paths” a user may take through the system to achieve a goal. • Identify the paths in the form of a state transition diagram (typically from URL to URL in the case of a web test) such that a minimum number of paths can be identified and traversed. • Objections: • O: Far too many paths to do this. R: What’s the risk of something going wrong? Model your paths at whatever level you find useful to test against • O: The whole point of a web app. is to be agile. R:It doesn’t matter how agile your development is, if users click out because the app.’s unusable, because you haven’t tested all the paths. If the app is well-made there won’t be many paths. Test Management Summit 2007 11

10. Test coverage by web script, web page, application, and component • Web sites are built from bits. • Testing each bit returns us to an older, component-based test model • Having identified the risk level of the web site, decide the level of coverage of each component. It doesn’t matter how stable the web site is if the user experience is (to use a technical term) crap Test Management Summit 2007 13

11. More questions • Question: where does a feature start and end? • Question: how d’you measure the size of a feature? (See the IFPUG counting practices manual) • Question: the number of states the system can be in is astronomic - how do we decide what sets of variables to use? • Question: are there coverage questions in user interface testing? Test Management Summit 2007 15

12. Conclusions Coverage must: • answer a question conclusively • have a baseline, • be user-, and business-relevant at some level Coverage is very definition-dependant: • Invent a new coverage type and every manager will want to know why you haven’t used it. • If the features are ill-defined, the coverage will be. Coverage remains a major question Test Management Summit 2007 16