The Testing Profession

1. The Testing Profession Geoffrey BessinMarket Manager, Software QualityIBM Rational Software

2. Obligatory Dilbert

3. Companies want YOU!

4. Agenda • The Value of Testers to the Organization

5. The bad old days….Part One Feature Set Fundamental Project Determinants locked early Cost Feature Set Cost spent early Quality Time to Delivery CHOOSE ONE Time to Market Quality

6. The bad old days….Part Two VS Software engineering is the only engineering discipline for which defects are accepted

7. The bad old days….Part Three Here, we’re serious this time START GOAL Ok, try now Should be ready Uh, you’ve got two days “Iterative Development” “Unit Testing” “Process driven”

8. The bad old days….Part Four • No formal Tester training • Not treated as a profession • Self-fulfilling prophesy – novice testers, underappreciated value • Developers weren’t being trained to test either….

9. So what changed? Gartner estimates that, on average, a failed application costs an enterprise approximately $100,000 per hour. And, with the National Institute of Standards and Technology ("The Economic Impact of Inadequate Infrastructure for Software Testing," May 2002) reporting an estimated $22.2 billion to $59.5 billion lost per year as a result of poor-quality software, catastrophic failures are here to stay. MONEY Approximately 25 percent of all applications deployed on distributed systems are effectively tested using modern tools and techniques. Through the second half of 2004, the number of tested deployed applications will remain close to 25 percent. DIFFERENTIATION

10. Agenda • The Value of Testers to the Organization • Testing in Various Development Processes

11. Legacy Development Processes • Code & fix • Purely an engineering effort • No testers at all • Fully dependent on skills of the programmers

12. Modern Development Processes…. • Waterfall • First proposed in 1970 by W. W. Royce • Development proceeds linearly through phases of requirements analysis, design, implementation, testing (validation), integration and maintenance • Risk is considered mitigated via problem analysis • Testing is typically direct derivative of requirements • Considered discredited for use as a Real World process • However, in practice, the process rarely proceeds in a purely linear fashion

13. Modern Development Processes • V model • Portrays several distinct testing levels / illustrates how each level addresses a different stage of the lifecycle • Testing process also involves identifying what to test (test conditions) and how they'll be tested (designing test cases), building the tests, executing them and evaluating the results, checking completion criteria and reporting progress. • Says nothing about how or how much requirements and designs should be documented • Doesn’t define when tests of any level should be designed or built.

14. Modern Development Processes • Spiral/Iterative • Defined by Barry Boehm • Each cycle considers critical stakeholder objectives and constraints, product and process alternatives, risk identification and resolution, stakeholder review, and commitment to proceed. • Level of effort, degree of detail within each cycle driven by risk considerations • Each phase starts with a design goal and ends with the client reviewing the progress thus far. • Using analysis and measurement as drivers of the enhancement process is one major difference between iterative enhancement and the current Agile Methods.

15. Modern Development Processes • Agile • Distillation of Scrum/Feature Driven/XP/Crystal family/Adaptive Software Development/DSDM • Considered “lightweight” process family • Have short iterative cycles, feature-based planning, constant feedback, change tolerance, team proximity, customer intimacy, and a focus on the overall team ecology • Requires • Uncertain or volatile requirements • Responsible and motivated developers • Customer who understands and will get involved. • Minimizes risk by ensuring that software engineers focus on smaller units of work. • Advocates developer testing, scenario-based testing, user testing http://agilemanifesto.org/

16. Standards, Not Processes • Capability Maturity Model (CMM) - http://www.sei.cmu.edu/cmm/ • model for judging the maturity of the software processes of an organization • No rules for creating such an organization though • Five levels of maturity (Initial, Repeatable, Defined, Managed, Optimizing) • Recently replaced by CMMI (CMM Integrated) • ISO 9000 - http://www.iso.ch/iso/en/iso9000-14000/index.html • Family of standards primarily concerned with "quality management“ • Focused on what the organization does to fulfil: • the customer's quality requirements, and • applicable regulatory requirements, while aiming to • enhance customer satisfaction, and • achieve continual improvement of its performance in pursuit of these objectives • Six Sigma - http://www.isixsigma.com/ • methodology that uses data and statistical analysis to measure and improve a company's operational performance by identifying and eliminating "defects" in manufacturing and service-related processes • commonly defined as 3.4 defects per million opportunities • ITIL - http://www.itil.co.uk/ • best practice for IT Service Management - series of books giving guidance on the provision of quality IT services, and on the accommodation and environmental facilities needed to support IT

17. Clarification….

18. Agenda • The Value of Testers to the Organization • Testing in Various Development Processes • Testing Approaches

19. Various Black Box Testing Approaches • All tests involve choices and tradeoffs in at least • these five areas: • Coverage: What you’re testing • Risks: Problems you are trying to find • Evaluation: How you know a test passed or failed • Activities: How you test • Testers: Who is testing • How you think about these issues dictates what combinations of test techniques you will use. • Function testing • Domain testing • Specification-based testing • Risk-based testing • Stress testing • Regression testing • User testing • Scenario testing • State-model based testing • High volume automated testing • Exploratory testing Caner, Bach, Pettichord “Lessons Learned in Software Testing”

20. Agenda • The Value of Testers to the Organization • Testing in Various Development Processes • Testing Approaches • Being a Good Tester

21. A Good Test Engineer • Has a 'test to break' attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail • Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful • Previous software development experience can be helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers' point of view, and reduce the learning curve in automated test tool programming • Judgment skills are needed to assess high-risk areas of an application on which to focus testing efforts when time is limited. “Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives." — Willa A. Foster http://www.softwareqatest.com/qatfaq2.html

22. Succeeding as a Tester and as a Test Organization • Facilitate communication across the team • Raise team consciousness about quality, best practices • Possible because they are not insulated from the customer context • Clarify requirements through your understanding of the customer • From customer point of view • From user point of view • From programmer’s point of view • Mismatches between these views • Identify assumptions • Understand team characteristics to determine suitability of approaches to testing • Skills • Team size • Management • Schedule • Severity of failure • Testability needs “There is nothing as mysterious as something clearly seen.” Robert Frost

23. Succeeding as a Tester, as a Test Organization • Plan, plan, plan • Refactoring tests as you go along without a goal is just bad refactoring • Where is the finish line? • Test design finds errors (Boris Beizer, Software Testing Techniques (Thomson Computer Press, 1990) • Collaborate to help design for testability • We can learn from hardware…. • Simplifies automation efforts for regression testing • Know the dimensions of test case quality • Information • Reduction of uncertainty. How much do you expect to learn from this test? • Power • If two tests have the potential to expose the same type of error, one test is more powerful if it is more likely to expose the error. • Credibility • Failure of the test should motivate someone to fix it, not to dismiss it. • Feasibility • How hard is it to set up the test, run the test, and evaluate the result? “In preparing for battle I have always found that plans are useless, but planning is indispensable.” Dwight D. Eisenhower

24. Agenda • The Value of Testers to the Organization • Testing in Various Development Processes • Testing Approaches • Being a Good Tester • How IBM Rational Can Help

25. Analyst Architect Developer Tester Operations Test Debug Deploy Load Test Define Design Code Functional Test Monitor Quality Drivers in the IBM SWG Product Line RUP – ClearCase (CM) – ClearQuest (DCT) – RequisitePro (RM) – TestManager (TM) – ProjectConsole TEAM UNIFYING PLATFORM Rational PurifyPlus Rational Test RealTime Rational Rose XDE Visual Trace WSAD tracing Rational RobotRational XDE Tester Rational Performance Tester Tivoli TMTP WebSphere Business Integrator Rational Rose XDE

26. Continuously Ensure QualityUniting innovative solutions and best practices to prevent, detect, diagnose and removedefects all across the iterative software application development and deployment lifecycle Analyst Architect Developer Tester Operations Test Debug Deploy Load Test Define Design Code Functional Test Monitor Data and metrics must be open, accessible, traceable IT Operations must be tied into the lifecycle Quality-enabling activities must start here and involve the entire team Integrated, Optimized Quality Solutions

27. IBM Rational - Defining and Implementing Standards OMG UML2 Testing Profile • Extends UML 2.0 to accommodate static and dynamic test specification • Enables sharing of profiling and testing information between tools, vendors • Establishes same test abstraction benefits as UML provides to developers Consortium lead 1st implementation! Eclipse/Hyades/EMF • Open source, integrated test, trace and monitoring infrastructure • Promotes testing across lifecycle by opening door for innovation • Avoids lock-in forced by vendors • Already shipping in WSAD Founding member and primary contributor Laying groundwork for process and data integration

28. The IBM Software Development Platform Continuously EnsureQuality Develop Iteratively Focus on Architecture Manage Changeand Assets Analyst Architect Developer Deployer Tester DB2Lotus RationalTivoliWebSphere Visually construct, transform, integrateand generate code Model, simulate, assemble,and monitor business processes Pervasiveand Embedded Provision, configure, tune and troubleshoot applications Design, create, and execute tests Model applications and data • Follow a common process • Track project status • Manage requirements • Manage changeand assets • Manage quality IBM and3rd Party Servers Project Manager Partners TEAM UNIFYING PLATFORM Microsoft IndustryStandards Training & Mentoring

29. Agenda • The Value of Testers to the Organization • Testing in Various Development Processes • Testing Approaches • Being a Good Tester • How IBM Rational Can Help • Resources

30. Reading Suggestions • Better Software Magazine - http://www.stickyminds.com/BetterSoftware/magazine.asp • Software Test and Performance - http://www.stpmag.com/ • ASQ Software Quality Professional - http://www.softwarecertifications.com/ • Journal of Software Testing Professionals – http://www.testinginstitute.com/journal.php • Quality Techniques Newsletter - http://www.soft.com/News/QTN-Online/ • Methods and Tools - http://www.martinig.ch/mt/index.html • Software Development Magazine - http://www.sdmagazine.com/

31. Organizations • SEI - http://www.sei.cmu.edu/ • IEEE - http://shop.ieee.org/store/default.asp?tabtype=stand • American Society for Quality - http://www.asq-software.org/ • Certification - http://www.asq.org/cert/types/csqe/index.html • Quality Assurance Institute - http://www.qaiusa.com/ • Certification - http://www.softwarecertifications.com/ • International Institute for Software Testing - http://www.testinginstitute.com/

32. QUESTIONS

33. Thank You