Most Important Tests (MITs)

1. Most Important Tests (MITs) • Most Important Tests is just a method which focuses on the risky and important areas of the system to test • Based on statistical analysis of the “risky” areas of the system that needs testing. • Where is the statistic coming from ? • Inspections and reviews? • Personnel history of the developers? • Past projects • The risky areas are then prioritized to identify the “important” areas • How would you prioritize them? • Gain agreement on the prioritized list of test areas • Make sure resources are received for the test effort

2. Example from your Assignment 1 Test 2 Test 1 type Review (formal/inform) Code execution material Requirements Specification doc “Unit tested” code size 3 pages major functions 4 3 x 30minutes = 1.5 pers.-hr effort expended total problems found * 6 Any other items that you may want to track ? 90 p-m/6 bugs= 15p-m/bug found effectiveness * Breakdown of the problems found and their resolution

3. Breakdown of Problems Found - - - - - - Prob. # Severity* Functional Area Status high # 1 • Get statistics by: • - Functional area • Severity • Author • Status • etc. * There exists a severity description

4. MITs Methodology Provides • “tools” for sizing the test effort that allows trade-offs: • Test coverage • Resources • Schedule • “tools” for tracking the testing activities’ progress (using S-curve) • Track problems found • Status reporting • Determining end of testing

5. S- curve Return on testing effort starts to Diminish (time to consider stopping) Total # of Problems found Time or Effort

6. How do We Use S-Curve? • Track a set of tests of a “product” over a period of time. • Run a set of tests (covering different areas) • Fix the errors • Rerun the set of tests • What is your guess of the “shape” of S-curve? • Track a series of different tests (reviews, unit test, functional test, etc.) of different “stages of a product” over time. • Conduct requirements review • Conduct design review • Conduct unit test • Conduct component test • Conduct system test • What is your guess of the “shape” of S-curve?

7. Deciding when to Quit Testing • Seed the product with errors – call it x • Keep account of the number of seeded errors discovered --- call it y • Keep track of real errors found – call it r • Then tabulate or estimate the total number of errors in the product --- call it total, utilizing the following relationship. • y/x = r/total • total = r * x/y • Ask if you can live with an estimated (total – r) errors left in the system.

8. Set of MITs Activities • Publishing test inventory containing all assumptions and requirements of test • Count and enumerate all the identified test areas and test cases (not necessary to be executed) • Prioritize and rank the enumerated test areas and cases • Estimate the effort required to execute each test case • Negotiate with management or person who holds resources on the required resources for each of the test coverage/schedule trade-offs.

9. Set of MITs Activities (cont.) • After picking the most important areas for testing perform the following for designing the actual test cases: • path analysis • Data analysis • If necessary, renegotiate the resources, schedule, etc. once more is known about the test cases • Execute the test cases • Track the test progress utilizing the S-curve and determining when to stop • Perform a test post mortem to evaluate and plan for improvement. Ideally, these steps should all be utilized, but in reality some are skipped.

10. Factors Influencing MITs Activities • Ease of Implementation • Some steps require a lot pre-training • Some steps require very special skills • Cost of Implementation/Return on Investment • Problems found and fixed during pre-release require resources and time. • Problems that are found and fixed during post release require another set of resources and time • (Which is more costly ?) - - - discuss • Development Methodology • Some methodology has extensive built in test steps (from planning to tracking); others don’t.

11. Easier MITs Activities • Bug and problem tracking • Test inventory & test coverage • Planning and performing path and data analysis • Ranking the test areas by risk • Test effort estimation • Test performance metrics Within these activities which ones are easier than others? Why? What do you use as criteria? (resource need, technical skills, etc.)

12. Harder MITs Activities • S-curve analysis • Requires some experience and past data to even have an S-curve • Test re-run automation • Needs to capture the conditions of the previous run • Capturing the test cases for automatically re-run pays off if the test cases are re-ran multiple times, especially for software products with multi-releases. • Automated test plan generation • Developing test areas from requirements may be easier if Use Cases or other better defined requirements language is used

13. How Much MITs ? • The steps in MITs ought to be selected and modified according to the type of project • May use the full range of MITs if the software project is run: • Using traditional “waterfall-like” process • Large and complex software • Risk averse • Highly mission critical • May emphasize certain parts of MITS over others if the software project is run: • Using a more Agile approach such as eXtreme programming • Small to medium size software • Less risk averse • not mission critical

14. How Test Fits in for You? R E L E A S E Requirements Design Code Test & Fix Test Prep Test Planning

15. Word of Caution • Current fashion of Agile development does not equate to no planning. However - - - • Many are mistakenly doing less or no planning • Emphasis on understanding and documenting requirements is casually allowed to erode • User/customer physically close to the developers and are often paired with developers • Lot of the requirements are verbal • Changes to requirements are not closely controlled under the misnomer of “flexibility” What is your guess of the effect of this attitude to: - testing? (more or less testing? More or less effective testing?)) - quality? (more or less satisfied customer? More or less bugs?)