Testing the test metrics
1 / 32

TESTING THE TEST METRICS - PowerPoint PPT Presentation

  • Uploaded on

TESTING THE TEST METRICS. Department of Jaka. Anna Tester. Brian Tester. Anthony Developer. What have they done today ?. Fixed three bugs . Added two new functions to prototype Written unit tests . Updated Release Notes. Raised a bug .

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'TESTING THE TEST METRICS' - ide

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Department of jaka
Department of Jaka







Whathavethey done today?

  • Fixedthreebugs.

  • Addedtwonewfunctionstoprototype

  • Writtenunittests.

  • UpdatedReleaseNotes.

  • Raised a bug.

  • Deployedthereleaseforthisfixandretestedthreetimes but case is stillfailing.

  • HelpedBrianfor his test set upmean time.

  • Deployedtherelease (bygettingsomehelp)

  • Didregressiontestingandexecuted 127 caseswhicharealreadyautomated.

  • Gaveapprovaltoshipment.

Problem 1 showing well what you are doing
Problem 1:Showingwellwhatyouaredoing…

What makes test brillant
Whatmakes test brillant?

  • Finding maximum number of bugs?

  • Finding maximum number of critical bugs?

  • No bugs returned from customer?

  • Long hours of overtime?




Problem 2 distinguish the idea hidden at the back of numbers
Problem 2:Distinguishthe idea hidden at theback of numbers


  • If metric is a number attached to an idea than as the idea attached to these numbers vary from person to person things get more dangerous.

  • Testdepartments need more and more explaining their selves which causes bunches of test metrics.

    Today we are going to analyze and distinguish the productivity of these metrics and try to find the possible fallacies in order to select the most efficient of them.

Today we are going to talk about
Todaywearegoingto talk about

  • whatthetestablewhich is test metricswill be usedfor, whowilluse it and in whatways it will be used?

  • define themetricgroupthatwill be takenunder test

  • examinetheexpectedresultswiththeactualresults

  • comparetheresultswith a controlgroup

Customer s

A kilo of banana weighs a kilo. How muchtestingarewebuying(orselling) in thisproject?

I assureyouthatwearedoing a greatjob here. Seetheresultsattached.

Upper Management

Senior Manager

Itseemsthatweneedreorganizing. Whereto start from?

I am themostexcelenttesteryouhave. Does it bring a pay raise?

Line Manager



I do not wanttobother but areweshippingtoday?

I am at thecustomernow. Do wehaveanyknownbugs I shouldnow?


Project Manager

Pleasefill in theformsforperformanceassessmentsin ordertolet us driveyourperformancefeedbacks.

How shall I knowthesenumbersaremeasuringtheintendedcorrectly?

Human Resources

Me, Metrictester

Choice of metrics
Choice of metrics

Since in a limited time we cannot test all the metrics defined on earth, we will select a set of random methods used. For example we shall use following:

  • Tests Passed vs. Failed Ratio

Defect slippage ratio
Defect slippage ratio

  • Prerequisites:Defect tracking system, a pile of testers, a group of users or supports team whom tracks defects during usage at production, definite time period to finalize until when production defects are accumulated.

  • Execution: Count the number of bugs raised at test execution for a release cycle than until the defined time, count the bugs raised from production. Get the ratio of these two.

  • Expectation: If this ratio is small than we shall say that we have a proficient test group.

Defect slippage ratio1
Defect slippage ratio

Test Group 1

Test Group 2

Test Group 3

Defect slippage ratio2
Defect slippage ratio

Test results show that;

  • Sinceseverity is not taken into consideration this results fools its users. Although our first test group missed three critical bugs they look the best performing among all. Additively unfortunate second group looks really bad performing among all three,may be because of a make up defect.

  • In addition to those since the issues rejected because of no error…etc. are not taken into consideration, which means more false results.

    Among all the issues raised from production depends on the performance of users at production. If there are few or no users than this metric does not tell much about the performance of testers.

Defect rejection ratio
Defect Rejection Ratio

  • Prerequisites:Defect tracking system, one tester, a group of development team whom solves or rejects defects send from testers, a definite release cycle.

  • Execution: Count the number of bugs raised at test execution for a release cycle, than group them due to resolution types after that count the rejected bugs. Get the ratio of these two.

  • Expectation: If this ratio is small than we shall say that we have a proficient test group. If this ratio is big than that means testers are not capable of their product, which causes a big time loss for the development team.

Defect rejection ratio2
Defect Rejection Ratio

Test results show that;

  • This metric causes misbehaving to unfortunate testers who are dealing with difficult development groups. In this case the time consumed should also be investigated with development team if they have provided enough information to testing team at the beginning.

  • In our third case although configuration cases consumed all of development team’s time since development feels the responsibility of successful business results they achieved the process cooperatively which does not give any solid information about testers skill.

    Testers know how generally depends on documentation written by designers or developers and their personal experience with the product. If the product provides detailed logging, configuration items are self-descriptive and do not have duplicates than the tester becomes proficient at his/her work faster than his/her colleagues.

Customer satisfaction index volume of calls
Customer satisfaction index – volume of calls

Customer satisfaction index not only depends on the volume of call but also Number of system enhancement requests per year, Number of maintenance fix requests per year, training time per new user, Number of product recalls or fix releases (software vendors), Number of production re-runs (in-house information systems groups). For simplicity we have chosen volume of calls. Since calls to customer services costs a lot, many people do not use it except really urgent cases. So every unique call deserve our emphasis.

  • Prerequisites:Defect tracking system, a teamof testers, Customerservicessystemthatrecordscustomerscomplaints.

  • Execution: Count the number of bugs raised at test execution for a release cycle, count the number of callscontainingcomplaintsperproduct. Get the ratio of these two.

  • Expectation: If this ratio is bigthan we shall say that we have a proficient test group.

Customer satisfaction index volume of calls1
Customer satisfaction index – volume of calls

Customer satisfaction index volume of calls2
Customer satisfaction index – volume of calls

Test results show that;

  • Since product A is as bad tested at product B and C, the risk of product and the attention of subscriber changes the successive rate direction.

  • Although product C is also critical since it causes company to financially damaged no calls arise from this defect.

    So it is fallible to depend on the customer reaction when judging test activities.

Test cost in
Test cost (in %)

It is a myth that test cost should be a ratio for example one fourth of the development cost. Let us see if this myth is true or not. When we analyze test cost against overall project cost as follows

  • Prerequisites: Project managementsystemthatrecordsworklog time record of peopleworking on a project, testers, developpers

  • Execution: Count the number of bugs raised at test execution for a release cycle than until the defined time, count the bugs raised from production. Get the ratio of these two.

  • Expectation: If this ratio is small than we shall say that we have a proficient test group.

Test cost in2
Test cost (in %)

Test results show that;

  • there is no scientific relation with the job done by testers and developers. Testers deal with configuration, test set up, data preparation issues, as well as testing the whole product against any unpredicted errors occur whereas developer deal with the requirement requested may be effecting a few lines of code.

  • In fact if a developer is good at his work and writes his code with thinking race scenarios, testers workload will be less compared to a tester who does a lot of test set up and retesting. That means in fact there is a reciprocal relationship. If development time is short, it is highly possible that the code is buggy so there needs to be more time for test execution and bug fix period.

    As it is clearly seen from the test results, we can say that this myth has failed.

Tests passed vs failed ratio
Tests Passed vs. Failed ratio

“Test does not show absence of bugs, it shows existence”.

But“How about giving shipment decisions? ”

  • Prerequisites: Test caserepository, toolrecordingcaseresults, test team.

  • Execution: Afteronecycle of test execution, countthenumber of test casespassedandfailed, getthepercentage of thesetwo.

  • Expectation: If this ratio is small than we areclosertoshipment.

Tests passed vs failed ratio2
Tests Passed vs. Failed ratio

Test results show that;

  • Shipmentdecision can be madefor a productwhich has%1 pass rate at reports the previous day.

  • One more month shall be neededto ship the product who had %99 pass rate.

  • The other 50% mightnot be 50% any more.

    Thinkaboutreportingtheseresultsto CTO, how reliablethereportswilllook?

The truth all metrics are fallable what will we do now
Thetruth: Allmetricsarefallable… Whatwillwe do now?

“It’s easy to refuse to provide something that isn’t perfect. But that’s not helpful when the perfect isn’t available.

When it comes to testers or consultants offering a “better alternative”, every executive has both the right and the responsibility to decide which alternative is the better one for her or his situation. ”



Accept all that way

“ Itis important (I thinkveryimportant) to pay attentiontothevalidity of ourmetrics. It is importanttoimprovethem, tofindwaystomitigatetherisks of usingthem, andtoadviseourclientsaboutthecharacteristicsandrisks of the data/statisticswesupplytothem. I thinkit’simportanttousemetrics in waysthatdon’tabusepeople. ”

Cem Kaner

Professor of Software Engineering at Florida Institute of Technology

Control with a control group
Control with a controlgroup

Test progressreport

Stock market report



Comparison metric from finance sector
Comparisonmetricfrom Finance sector

  • One of theCreditAuditMetrics: Cash Rate

    Cash Rate = cashandcashequivalents / short-termdepts

    Expectation: IfCache Rate is morethantwothancompany is foundto be reliableforcreditassesments.

Test results show that;

Financial metricsarealsofallable.

Control with a control group1
Control with a controlgroup….