justifying software testing in the 21 st century l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Justifying Software Testing in the 21 st Century PowerPoint Presentation
Download Presentation
Justifying Software Testing in the 21 st Century

Loading in 2 Seconds...

play fullscreen
1 / 20

Justifying Software Testing in the 21 st Century - PowerPoint PPT Presentation


  • 149 Views
  • Uploaded on

Justifying Software Testing in the 21 st Century. Presented by Ian Gilchrist IPL Software Products Manager. Introduction. The Software industry is about 50 years old. Testing has generally been regarded as the mainstay verification technique But testing is an unpopular job.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Justifying Software Testing in the 21 st Century' - kendra


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
justifying software testing in the 21 st century

Justifying Software Testing in the 21st Century

Presented by

Ian Gilchrist

IPL Software Products Manager

introduction
Introduction
  • The Software industry is about 50 years old.
  • Testing has generally been regarded as the mainstay verification technique
    • But testing is an unpopular job.
  • New software techniques have emerged in the last 5-10 years
    • Can these be used to justify omitting testing?
    • Does testing still have a role to play in the 21st century?
new techniques
‘New’ techniques
  • Use of these following new techniques can all be said to lead to ‘better’ software:
    • Formal verification, via automated analysis of pre-/post- conditions
      • Best-known form is SPARK for Ada code (sub-set of)
    • ‘Advanced’ defect detection, via deep static analysis of potential run-time errors
      • Best-known forms include Polyspace, Klocwork, and Coverity, mainly for C/C++, Ada, and Java
    • Automated code generation, direct from design, supposedly eliminating human coding-errors
      • Examples include Matlab/Simlink, and SCADE, generating C code
new software techniques
New Software Techniques

Formal Ver’f’n or Defect removal

‘Better’ source code

Source code

Hand code

Compiler

Design Spec

Source code

Code Gen.

Object Code

claims
Claims
  • A speaker at a recent Ada conference (June 2005), stated, “It is rarely cost-effective to rely on testing for evidence of testability… The way forward is more to rely on use of…deep static analysis tools.”
  • Polyspace website used to claim that its use removed the need to unit test code. Now it says, “…our solutions enables organizations to reach unmatched levels of software reliability while reducing the time and cost of software testing.”
  • SCADE website says, “Coding errors are eliminated.”
context and definitions
Context and Definitions
  • Context
    • Restricted to unit/module testing, of ‘high-integrity’ software, coding in Ada, C and C++
  • Definitions
    • Verification is a set of techniques directed towards confirming that a software component “performs its intended function, and does not perform any unintended function.” [IEC 61508]

‘Good’ verification involves

      • Generation of evidence
      • Repeatability
      • Automated (within limits)
verification categories
Verification Categories
  • Verification techniques come in three categories:
    • Reviews/inspections
    • Analyses
    • Testing
  • It is conventional to use a (judicious) mixture of techniques from more than one category, to achieve an ‘acceptable’ level of confidence

“Testing cannot show the absence of errors…Verification is typically a combination of reviews, analysis and testing.” [RTCA DO-178B]

aim of this paper
Aim of this Paper
  • Take a set of ‘old’ and ‘new’ verification techniques
    • representing commonly used methods in industry.
  • Subject them to three ‘usefulness’ criteria
  • Come to a conclusion about the continuing relevance of testing in the 21st century.
    • Still just as relevant?
    • Less relevant?
    • Irrelevant?
the selected techniques
The Selected Techniques
  • Reviews/Inspections
    • Enforcement of coding standards
    • Fagan Inspections
  • Analyses
    • ‘Advanced’ defect detection
    • Formal verification
  • Testing
    • Functional testing
    • Structural (i.e. with added coverage)
    • Statistical (use data to mimic operational conditions)
the three criteria
The Three Criteria
  • To what extent does each technique contribute to providing confidence that the software performs its intended function?
  • To what extent does each technique contribute to providing confidence that the software does not perform unintended functions?
  • To what extent is each technique pragmatic:
    • Generates evidence
    • Can be automated
    • Is repeatable
    • Cost-effective
results
Results
  • The ‘results’ presented here are subjective.
  • They represent only the observations, opinions, and prejudices of the author!
big elephant in the corner
BIG ELEPHANT in the corner
  • The best static verification technique (formal verification) can only demonstrate that the source code ‘looks good’
  • The weak link is the compiler
    • For code to work as intended the compiler must correctly convert source to object code
  • IPL’s experience is that all compilers have faults
    • Some are better than others
      • Ada compilers are better than C
      • C compilers are better than C++
  • Source code can be ‘perfect’ but still not work correctly when compiled
wild card
Wild Card
  • What about Code Generation?
  • Can we by-pass the need to verify software at all?
    • Do we 100% trust the code generator?
    • There is still the compiler issue
  • I think even generated code needs verification.
the strengths of testing g
The Strengths of Testingg
  • Testing is nearly the only technique that demonstrates that code ‘performs its intended function’
    • It has a more limited role in demonstrating the absence of faults
    • It can be highly automated, repeatable, and generates evidence
    • It need not cost a lot
      • Be objective about when to stop testing
conclusion
Conclusion
  • Reviews and Analyses (IMO) at best serve to demonstrate that code is READY FOR TESTING
    • The new techniques however can justify the use of less testing than was considered necessary in the past
    • How to define ‘less’ testing…
slide20

Any

questions?