1 / 34

Classic Testing Mistakes: Revisited

Classic Testing Mistakes: Revisited. Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson. Matthew Heusser mheusser@charter.net Presented at the Better Software Conference San Francisco, CA - Sept. 20 th, 2005. Organization.

makani
Download Presentation

Classic Testing Mistakes: Revisited

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Classic Testing Mistakes:Revisited Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson Matthew Heusser mheusser@charter.net Presented at the Better Software Conference San Francisco, CA - Sept. 20th, 2005

  2. Organization • Classic Mistakes: A different approach • The mistakes enumerated • Test Management Mistakes • Test Automation Mistakes • Development Mistakes • Test Strategy Mistakes • Root Causes • What to do tomorrow

  3. On the shoulders of pioneers

  4. Classic Mistake #1:De-humanize the test process AKA Management by Spreadsheet, Management by Email, Management by MS Project … Test Management Mistakes

  5. Classic Mistake #2: Testers Responsible for Quality “It’s strange that QA let that bug slip through” Test Management Mistakes

  6. Classic Mistake #3:IV&V Determines ship date Do they really? Test Management Mistakes

  7. Classic Mistake #4:Task-based status reporting • Examples: • Testing is “on schedule” • Testing “should be done by Tuesday” • Consequences • Loss of credibility • Bad information for decision makers Test Management Mistakes

  8. Classic Mistake #5:Evaluating testers by bugs found … and developers by number of bugs injected • Consequences: • Friction • Focus on easy-to-find yet trivial bugs (usability) • Information hiding Test Management Mistakes

  9. Classic Mistakes #6Inappropriate Models for Test Improvement • NO

  10. Classic Mistake #7:Lack of test training for developers • Testing is a skill. • It won’t appear like magic. Development Testing Mistakes

  11. Classic Mistake #8:Separate devs and testers • To create friction, emphasize division • Anything that increases the length of the feedback loop is bad. • To improve get rid of waste and tighten the feedback loop. Development Testing Mistakes

  12. Mistake #9:When late, add Test Automation • Someone has to learn the tool • Someone has to record the scripts Test Automation Mistakes

  13. Mistake #10:Mine Field Test Automation Test Automation Mistakes

  14. Mistake #11:Hiring for test tool skills • Technology skills can be taught • Talent can’t • The “Hit the Ground Running” Argument Test Automation Mistakes

  15. Classic Mistake #12:Insufficient diversity in test strategy • Examples: • Only requirements based testing • Only coverage testing • Consequence: • Missing entire classifications of defects Test Strategy Mistakes

  16. Classic Mistake #13:Over-reliance on scripted testing All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute. - Joel Spolsky, JoelOnSoftware.com Test Strategy Mistakes

  17. Mistake #14:Untrained exploratory Testing • “Just think creatively” • “Try to break it” • Exploratory Testing is a discipline Test Strategy Mistakes

  18. Classic Mistake #15:Separate test design and execution Test Strategy Mistakes

  19. Classic Mistake #16:Vacuous Documentation • Examples: • The issue resolution document • Physical signoff/check marks • Elaborate test case templates • Consequence: • Time spent documenting is time not spent testing Test Strategy Mistakes

  20. Mistake #17:Trying to fix things beyond your reach The Meta-Mistake

  21. Don’t treat the symptoms Find & Fix the root cause!

  22. Root Cause #1Lack of Systems Thinking in Testing • The law of unintended consequences Root Causes

  23. Root Cause #2:Translation Problems Example: - “You need to completely test this module” Root Causes

  24. Root Cause #3:Process Myopia • Example: • The [in]famous Issue Resolution Document • “We don’t do things that way here” • Elevating process over skills • Solutions: • The ear of the king / History Lessons • If your boss doesn’t care – ignore it Root Causes

  25. Root Cause #4:Technology Myopia • Example • “Use XML on the next project” • “I just bought 5 copies of WinRunner …” • Solution: • If you’re technical, they need you to do it • If you’re a manager, focus on business impact and risk Root Causes

  26. Root Cause #5:Fred Taylor in the organization • Examples: • Factory Mentality • High Specialization • Mixing of skill sets is verboten • Solutions: • Peopleware, or anything by Weinberg • First Break all the rules – Jim Collins • Lead. Insulate your team. Root Causes

  27. A ‘new’ methodology • Root Cause Analysis • Pareto Analysis • Drive out waste/tighten the feedback loop • Then worry about better practices (Image from Rapid Development, (c) 1996 by Steve McConnell. Used with permission from the author) What to do tomorrow

  28. Why • New practices take permission • It is hard (but possible) to get more done by adding work • It is easy to get more done by subtracting work • So start by looking to remove worst practices

  29. What to do tomorrow • Discuss • Q&A

  30. Where to go for more • “Classic Mistakes in Software Testing”, Brian Marick, STQE, 1997 • Rapid Development, Steve McConnell • An Introduction to general systems thinking, Gerold Weinberg, 1975 • Lessons Learned in Software Testing, Kaner, Bach, Pettichord

  31. Bonus Section

  32. Root Cause #6:Pressure for short-term results • Example: • “Ship to make 4th quarter numbers” • Putting off problems instead of addressing them • Solution: • Save your team • Professionalism means something • The Quake Example Root Causes

More Related