Test vs. inspection Part 1. Torbjørn Skramstad. What we will cover. Part 1 Introduction Inspection processes Testing processes Part 2 Tests and inspections – some data Inspection as a social process – two experiments and some conclusions. Introduction. Adam’s data - 1.
The main information that you get from the table on the previous slide is that
How can we tell the difference?
As can be seen from the next slide, testing is not an acceptable solution alone. It will
We can generate tests automatically, but would never the less have to use large resources to check the result – the oracle problem
The following relation holds under a rather wide set of conditions:
The initial number of defects – N0 – must be estimated e.g. based on experience from earlier projects as number of defects per KLOC.
The important message here is that testing cannot always be done.
In the first, important phases, we have nothing to execute and will thus always have to do some type of inspection.
This might be considered one of the weaknesses of traditional software engineering over Agile development.
In order to understand the main differences between testing and inspection, we should consider Fits’ list.
Based on this, we will give a short discussion of the relative merits of testing and inspection.
Good when we need the ability to
Not so good when we need the ability to
In order to do the best job possible we need processes where we let each part
Architecture, system, sub-system and component design plus pseudo code. Here we can only use inspections.
Man will use experience and knowledge to identify possible problems
Machine can support by identifying information – e.g. find all occurrences of a string.
For executable code, we can use inspection, testing or a combination of both.
The size and complexity – degree of dynamism – of the code will, to a large degree, decide our choice.
Other important factors are the degree of experience with
The term “inspection” is often used in a rather imprecise manner. We will look at three types of inspection:
The first two types are usually project internal while the last one is used as a final acceptance activity for a document.
For all types of inspections:
Walkthrough is a simple process – mostly used for early decisions for an activity. The document owner:
Rules,checklists,proceduresThe informal inspection process
The formal inspection process described below is – with small variations – the most commonly used. The version shown on the following slides stem from T. Gilb and D. Graham.
We recommend this process as the final acceptance process for all important documents
PlanningFormal inspection process overview
Edit and follow-up
Important planning points are:
Important activities here are:
This is the main activity of the inspection. Each participant read the document to look for
The logging meeting has three purposes:
The author receives the log from the inspection meeting. All items - issues - in the log are categorised as one of the following:
This is the responsibility of the inspection leader. He must assure that all issues raised in the log are disposed of in a satisfactory manner:
We will look at three types of testing:
Unit testing – does the code behave as intended. Usually done by the developer
Function verification testing – also called systems test. Does the component or system provide the required functionality?
System verification testing – also called acceptance test. Does the hardware and software work together to give the user the intended functionality?
Unit testing is done by the developer one or more times during development. It is a rather informal process which mostly run as follows:
Implement (part of) a component.
Define one or more tests to activate the code
Check the results against expectations and current understanding of the component
Simple way to check that the code works.
Can be used together with coding in an iterative manner.
Will only test the developer’s understanding of the spec.
May need stubs or drivers in order to test
A systems test has the following steps:
Based on the requirements, identify
Test for each requirement, including error handling
Initial state, expected result and final state
Identify dependencies between tests
Identify acceptance criteria for test suite
Run tests and check results against
Acceptance criteria for each test
Acceptance criteria for the test suite
Tests system’s behavior against customer requirements.
It is a black box test. If we find an error, the systems test must be followed by extensive debugging
The acceptance test usually has three activities – all involving the customer or his representatives:
Rerun the systems test at the customer’s site.
Use the system to solve a set of real-world tasks.
Try to break the system – by stressing it or by feeding it large amounts of illegal input
Creates confidence that the system will be useful for the customer
Shows the system’s ability to operate in the customer’s environment
Might force the system to handle input that it was not designed for, thus creating an unfavorable impression.
Experience shown that testing and inspections
Thus, we need both testing and inspection.