1 / 31

Effective Reporting of Test Execution Progress

Effective Reporting of Test Execution Progress. Introduction. In this session we will look in detail at two different methods of reporting test progress Which can be used individually or in combination

vandiver
Download Presentation

Effective Reporting of Test Execution Progress

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Effective Reporting of Test Execution Progress Amadori Course: Delivering Effective Test Management

  2. Introduction • In this session we will look in detail at two different methods of reporting test progress • Which can be used individually or in combination • Unless we report testing progress effectively the value of even our best test execution efforts is considerably reduced Amadori Course: Delivering Effective Test Management

  3. What does EFFECTIVE mean in this context? • Progress MUST be measured in terms of requirements coverage • Recording progress in terms of tests executed tells you nothing • Except how hard the test team are working • And even this can be “gamed” Amadori Course: Delivering Effective Test Management

  4. Question 1. testing have executed 907 tests over the last 5 days. 602 tests have yet to be executed 2. Of the 90 requirements tested 32 have been fully tested and 36 partially tested Which summary adds most value to stakeholders And Why? Amadori Course: Delivering Effective Test Management

  5. Answer 2. Of the 90 requirements tested 32 have been fully tested and 36 partially tested Is the better summary as it measures BREADTH of test coverage 1. testing have executed 907 tests over the last 5 days. Measures only test effort Amadori Course: Delivering Effective Test Management

  6. Word of Warning • If you haven’t been through a structured requirements extraction and mapping process it is very difficult to accurately report requirements coverage • Not something you can add-on at the last minute Amadori Course: Delivering Effective Test Management

  7. Methods of tracking requirements coverage • Mapping against an existing requirements map • Using a Boston Matrix to provide a high level indication of test coverage Two approaches can be combined to provide both summary and detailed metrics to stakeholders Amadori Course: Delivering Effective Test Management

  8. Mapping test progress against requirements • Simple when tests relate to a single requirement • Not so simple when there is a many to many relationship between tests and requirements • Let’s start with a simple example…. Amadori Course: Delivering Effective Test Management

  9. A simple example Requirements Tests 1-10 A 11-15 B 16-95 C 96 D 97-100 Amadori Course: Delivering Effective Test Management

  10. Requirements Tests 1-10 A 11-15 B 16-95 C 96 D • % executed and % passed measure scope of testing • Pass rate measures quality of the code tested to date 97-100 Amadori Course: Delivering Effective Test Management

  11. Something More Complex… 1-10 A Tests Requirements B 11-15 C 16-95 D 96 97-100 Amadori Course: Delivering Effective Test Management

  12. Two Stage Process • Allocate test results against each requirement • Work out test metrics as before Amadori Course: Delivering Effective Test Management

  13. What else can these numbers tell us? • Can help prioritise testing effort • Ensures that requirements aren’t forgotten about during the heat of battle…. Amadori Course: Delivering Effective Test Management

  14. But not all requirements are equal….. • Two ways to take this into account when reporting progress…. • Add a weighting factor so more important requirements count for more • Or produce different versions of the report • 1. all requirements • 2. Only business critical requirements Amadori Course: Delivering Effective Test Management

  15. Issues with this approach • This approach provides excellent detail about how well or badly individual requirements are being delivered BUT… • Requires the reader to have a good understanding of how testing works • Can be difficult to see the bigger picture • This is where a Boston Matrix can help Amadori Course: Delivering Effective Test Management

  16. What is a Boston Matrix ? 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  17. Understanding the 4 quadrants is the Key.. • Quadrant A – High Pass Rate, Low Coverage • Looks good at first sight but be careful….. • Quadrant B – High Pass Rate, High Coverage • This is where you want to be • Quadrant C – Low Pass Rate, Low Coverage • Bad news, unless testing has just started….. • D – Low Pass Rate, High Coverage • Lots of issues but at least we know where we are….. Amadori Course: Delivering Effective Test Management

  18. Boston Matrix Heat Map 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  19. Alternative Views 100% 100% A A B B Pass Rate Pass Rate C C D D 0% 0% 100% 100% Test Coverage Test Coverage Amadori Course: Delivering Effective Test Management

  20. How is it populated? • Divide testing scope into a number of areas • Try where possible to reflect the way the project is seen by stakeholders…. • For each area work out • Test Coverage - % of requirements tested to date • Pass Rate (tests passed/tests failed) Amadori Course: Delivering Effective Test Management

  21. A Worked Example Where on the Matrix does each area go? Amadori Course: Delivering Effective Test Management

  22. Trade Entry: Pass Rate 92%, Coverage 64% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  23. Confirmations: Pass Rate 92%, Coverage12% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  24. Risk: Pass Rate 43%, Coverage 46% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  25. Analytics: Pass Rate 12%, Coverage 80% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  26. Reporting: Pass Rate 67%, Coverage 20% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  27. Compliance: Pass Rate 71%, Coverage 90% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  28. Accounts: Pass Rate 78%, Coverage 40% 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  29. Questions? • What if a requirement is only partially tested? • Establish a rule of thumb and stick to it….. • Either • If a requirement is more than 50% tested it counts as complete • Requirements are only covered when 100% of tests have been executed • Not all areas are equally important…. • Then highlight the important areas in some way to make them stand out…. Amadori Course: Delivering Effective Test Management

  30. Highlight the key areas of delivery 100% A B Pass Rate C D 0% 100% Test Coverage Amadori Course: Delivering Effective Test Management

  31. Exercise • You have a meeting with the project manager coming up • Using the Boston Matrix • What would you tell him about the project’s overall status? • Which areas give you most concern? • What are the key activities in each area moving forward? • Where would you prioritise available test resource? Amadori Course: Delivering Effective Test Management

More Related