Comprehensive Software Testing Course

1. Jonathan W. Lartigue

2. CourseObjectives(Goals) • Uponcompletionofthiscourse,thestudentwillbeableto: • DescribehowsoftwaretestingfitsintotheMIL-STD-498process. • DistinguishbetweenSoftwareUnitTestingandSoftware IntegrationTestingtechniques. • DescribethemanyapproachestoSoftwareUnitTesting, theirstrengthsandtheirlimitations. • Describethe several approachestoSoftwareIntegrationTestingandtheirstrengthsandlimitations.

3. CourseOutline(Topics) • Topicswewillcoverinclude: • SoftwareTesting • StagesofaSoftware Project • TestPlanning • SoftwareUnitTesting • BlackBoxTestingMethods • WhiteBoxTestingMethods • SoftwareIntegrationTesting • IntegrationTestMethods • CSCIQualificationTesting

4. SoftwareTesting(Introduction) • Howcanweestablishsomedegreeofconfidencethataprogramdoeswhatitissupposedtodo? • Withthorough,exhaustiveunitandfunctionaltesting. ‘Thereisnosingledevelopment,ineithertechnologyorinmanagementtechnique,thatbyitselfpromisesevenoneorder-of-magnitudeimprovementin productivity,inreliability,insimplicity.’ —FredBrooks

5. SoftwareTestPlanning • Whenshouldwestartthinkingaboutsoftware testing? • Whenthesoftwareisalreadywrittenandduetobeshipped? • Whenwe’rewritingthesoftware? • Whenwe’redesigningthesoftware? • Whenwe’redesigningthesystemthesoftwarerunson?

6. Needfortesting(Whybother?) • Startingattheverybeginningofaproject,theearlieranerrorordefectisdiscovered,themorecost-efficientitsremovalbecomes. Cost of Correcting Errors by Project Stage 250 200 150 100 50 0 Analysis Design Code Test Maintain

7. Needfortesting(Whybother?) • ‘Theprobabilityoftheexistenceofmoreerrorsina sectionofaprogramisproportionaltothenumberoferrorsalreadyfoundinthatsection.’ • —GlenfordMyers • Probability ofundiscoverederrors versuserrorsfound 50 40 30 20 10 0 UndiscoveredErrors 10 20 30 ErrorsFound 40 Source:TheArt of SoftwareTesting, GlenfordMyers, 1979

8. Needfortesting(Whybother?) • We’reonlyhuman… • Example: • Oct.1999lossofNASAMarsClimate Orbiter hasbeenattributedtoametricsystemtoEnglishsystemofmeasurementerror. • Engineersmistakenlyassumedmeasurementswereinadifferentsystem. • CosttoNASA:$125,000,000+ • Couldhavebeen(shouldhavebeen?)caughtduringtesting! Source: SoftwareQAandTestingResourceCenter

9. Definitions(SoftwareTesting) • SoftwareTesting—theprocessofexecutingaprogramwiththeintentoffindingerrors. • Forevenwell-writtensoftware,testingistypicallythelongestphaseofadevelopmentcycle. • TestPlan—anartifactthatencapsulatesassociatedtestcasesandprovidesanoverviewoftherelationshipsofthetestset. • TestCase—anartifactthatdescribesaninput,action,oreventinthesystemsandtheexpectedresult(s). • Source: SoftwareQAandTestingResourceCenter

10. Definitions(SoftwareTestingPhases) • Softwaretestingcanoftenbedividedintotwoseparatephases: • SoftwareUnitTestingistheprocessbywhichanindividualsoftwarecomponentistested,isolatedfromallothersoftwareinthesystem. • SoftwareIntegrationTestingistheprocessbywhich separatesoftwaremodulesaremergedandtheir interfacesaretestedagainstrequirements. • SoftwareUnitTestingandSoftwareIntegrationTestingaredistinctandseparate activities.

11. Definitions(TestPlan) • Toperformsoftwaretesting,wefirstneedasoftwaretestplan. • Atestplanwilldescribeindetail: • Acceptancecriteria. • Anapproachtotestingandintegrating. • Taskstobeperformed. • Aschedule/timeline. • Individualresponsibilitiesofteammembers.

12. Definitions(TestCase) • Testcases: • Asuccessfultestcaseisonethatdiscoversan as-yetundetectederror. • Itisimpossibletohavetestcasesforeverystateinasystem. • Thenumberofpossibleinputsintoasystemincreasesexponentially with thecomplexityofthatsystem.

13. Definitions(TestCase) • Testcases: • Itoftenrequiresonelineoftestcodetovalidateeachlineofdeliverablecodeinasystem • Testcodeis sometimes called‘throwaway’code–itdoesn’tgetdeliveredtothecustomer. • Therefore,itisnotunusualforsoftwareengineerstodiscard50percentormorethecodetheywrite. • Manyengineersrealizethisandthereforeminimizetestingeffortsbecausetheydon’tproduce‘deliverablecode.’ • Thisisapoorphilosophytoadopt.

14. UnitTestingvs.IntegrationTesting • UnitandIntegrationTesting: • S/WUnitTestingallowsforeasierdetectionandlocationoferrorsinthesystem. • Isiteasiertofindanerrorconfinedtooneprocedureorspanningseveral files? • S/WIntegrationallowstestingof interactions betweensoftwaremodules. • Candetectinconsistenciesinparalleldesign(assumptionsthatdifferacrossmultiplemodulesbecauseeachwaswrittenbyadifferentengineer). • UnitandIntegrationtestingareNOTthesame!

15. Definitions(HigherOrderTesting) • CSCIQualificationTestingwillusuallypitthe integratedsystemasawholeagainstitsownsoftwaresystemrequirements. • “Ourcustomerstatedthatweneedtohavethiseventtakeplaceifthisspecifiedconditionoccurs.” • Doesthis happen, andhowdoweverifyit? • Example: • EithertheRedSwitchsoftwaremoduleXproperlycommunicates with moduleYoritdoesn’t.

16. Definitions(HigherOrderTesting) • SystemTestingwilltestthesoftwarewiththerestofthesystemcomponentsandverifiesthesystemoperationmeetsitsspecifications. • “Ourcustomerstatedthatweneedtohavethiseventtakeplaceifthisspecifiedconditionoccurs.” • Doesthis happen, andhowdoweverifyit? • Example: • EithertheRedSwitchsoftwarestillfunctionswhencircuitboard • ‘X’fails,oritdoesn’t.

17. Comparison(Conclusion) • Conclusion • SoftwareUnitTesting,SoftwareIntegration,andSoftwareSystemIntegrationeachserveadistinctandvaluablepurpose. • Theeffectivenessandefficiencyofanyoneofthethreebecomeslessenedifanotherisremovedfromtheoverallprocess. • Totestasoftwaresystemsuccessfully,we needtoimplementallthreeapproaches.

18. SoftwareTesting

19. S/WUnitTestMethods • Breakdownofunittestingapproaches: • UnitTesting StaticTechniques DynamicTechniques BlackBox Testingw/oexecutingcode •EquivalencePartitioning •BoundaryValueAnalysis •RandomTesting •Cause-EffectGraphing Reviews,Inspections, etc. CleanroomTesting White Box •Basis-PathTesting SymbolicExecution •Statement-level coverage •Branch-levelcoverage •Condition-level coverage Source:T. Hilburn, Embry-RiddleAero. Univ.

20. Black-BoxTesting • Black-BoxTesting: • Alsoknownas‘data-driventesting’,itreliesonamoduledescriptiontodevisetestcases. • Usesthearchitecturaldesign,includinginputs,functionality,andoutputs. input output

21. Black-BoxTesting • Black-BoxTesting: • Treatsmodulelikea • “blackbox” • i.e.,youcan’tseetheinnerworkingsofthebox,justwhatgoesin andcomesout. • Blackboxtestcasesshouldbedesigned beforethecoding/implementationphaseends. input output

22. White-Box/Clear-BoxTesting • White-BoxTesting: • Oftencalled‘structuraltesting’,it reliesonanalysisofamodule’ssourcecodetodesignatestcase. • Morecloseanalysisofthedetaileddesignofamodule. input output

23. White-Box/Clear-BoxTesting • White-BoxTesting: • Treatsmodulelikea • “white-box” • Wecanseeandtesttheinnerworkingsofthebox. • Testcasesdesignedasneeded,onacase-by-casebasis. • Whiteboxtesting shouldsupplementblackboxtesting. input output

24. SoftwareTestingDiscussion

25. Black-BoxTestingApproaches • EquivalencePartitioning: • Divideinputdomainofaprogramintoclassesfromwhichtestcasesarederived: • Theunit’sdesignisanalyzedtoidentifyvalidinputvalues. • Whencombined with thespectrumofpossiblevalues,oneormorediscrete ‘partitions’ canbeidentified. • Thesepartitionsrepresentrangesfromwhichtestcasevaluescanbeselected.

26. EquivalencePartitioning,example • Equivalence1: • ◆x<65 • Equivalence2: • ◆65<=x<=90 • Equivalence3: • ◆91<=x<=96 • Equivalence4: • ◆97<=x<=122 • Equivalence5: • ◆122<x • Possible test case inputs: 23, 71, 94, 108,147

27. Black-BoxTestingApproaches • Boundary-ValueAnalysis: • Testcasesaredesignedontheboundariesofacceptableinputdomain. • Studieshaveshownthatmoreerrorsoccurontheboundaryofaninputdomainthaninthecenter. • e.g.The‘infamous’off-by-oneerror. • Complementsequivalencepartitioningandisoftenusedin conjunctionwithit.

28. Boundary-ValueAnalysis,example • Boundary1: • ◆65 • Boundary2: • ◆90 • Boundary3: • ◆97 • Boundary4: • ◆122 • Possible test case inputs: 64, 65, 66 • 89, 90, 91 • 96, 97, 98 • 121,122,123

29. Black-BoxTestingApproaches • RandomTesting: • Sometimes, somepeoplearejust“good”attesting(culminationofintuition,experience,detailedknowledgeofsystem). • Norealprocedurefordesigningtestcases,couldbebasedonguessingwhereerrorsmightoccur. • Alsoknownas“errorguessing”. • Example:Sortingalgorithm • Somerandom,but‘good’testsmightbe: • Emptyinputlist,single entry inlist,allentrieshavesamevalue,entry listisalready sorted, etc.

30. SoftwareTestingDiscussion

31. White-BoxTestingApproaches • White-boxtesting: • isverytimeintensive,asit requiresdetailedanalysisofinternalsourcecode. • isdesignedtobeextremelythoroughandoftentoucheseverysinglelineofcode. input output • Whiteboxtestingisconsideredessentialformission-criticalsoftware(i.e.nosurprises)andistheonlyacceptedwaytocompletelyvalidatesoftware. • Remember,evenwhite-boxtestingcannottesteverypossible input combination.

32. White-BoxTestingApproaches • Basis-PathTesting: • Consistsofsteps: • Convertunitundertestintoa‘flowgraph’ • Computeameasureofthemodule’slogicalcomplexity • » McCabeMetrics‘cyclomaticcomplexity’ • Usethemeasuretoderivea‘basis’setofexecutionpaths,whichareusedtogeneratetestcases.

33. Basis-PathTesting,example start bool isLetter(char value) { if (value >= 65 return true; if (value >= 97 && value <= 90) // A-Z && FALSE 65>= value <= 122) TRUE return true; return false; // a-z FALSE <=90 } TRUE • Wecancreateaflowgraphtodetailthedecisionpathsthroughthecode. • Byidentifyingallpossiblepaths,wecanidentifythemaximumnumberoftestcasesneeded. FALSE 97>= returntrue TRUE FALSE <=122 TRUE returntrue returnfalse end

34. Basis-PathTesting,example • Ifthefirstconditionofthefirstifstatementisfalse,therearethreepossiblepaths. • Remember,C++Booleanconditionalsare‘shortcircuited.’ start start start FALSE FALSE FALSE 65>= 65<>= 65>= TRUE TRUE TRUE FALSE FALSE <=90 FALSE <=90 <=90 TRUE FALSE TRUE TRUE FALSE 97>= FALSE 97>= 97>= return true TRUE return true return true TRUE TRUE FALSE FALSE <=122 FALSE <=122 <=122 TRUE TRUE TRUE return true return false return true return false return true return false end end end

35. Basis-PathTesting,example • Whenthefirstconditionofthefirstifstatementistrue,butthesecondfalse,therearethreemorepathsavailable. start start start FALSE FALSE FALSE 65>= 65<>= 65>= TRUE TRUE TRUE FALSE FALSE <=90 FALSE <=90 <=90 TRUE FALSE TRUE TRUE FALSE 97>= FALSE 97>= 97>= return true TRUE return true return true TRUE TRUE FALSE FALSE <=122 FALSE <=122 <=122 TRUE TRUE TRUE return true return false return true return false return true return false end end end

36. Basis-PathTesting,example • Finally,thereisonlyone pathifbothconditionsonthefirstifstatementaretrue. • Eventhoughwetestedonlythefirst‘if’statement,we produced7uniquepathsthroughthecode. • Luckily,wealsotestedthepossiblepathsthroughtheotherif statement. • Minimum‘basisset’ofpaths: • Withsomerefinement,wecould reducethenumberofpathsneededtotouchevery‘leg’ofthecodegraphtofour. • Thecode’s‘cyclomaticcomplexity,’ thenumberof‘basis set’ paths,is4. • Conclusion: • Basis-pathtestingiscomplex: • Thisresultsinanexponentialnumberofpossibletestcases. • Automatedtoolsareamust. start FALSE 65>= TRUE FALSE <=90 TRUE FALSE 97>= return true TRUE FALSE <=122 TRUE return true return false end

37. White-BoxTestingApproaches • Statement-levelcoverage: • Testcasesinthiscategory,asthenameimplies,havethegoalofreachingeveryexecutablestatementinthecodeatleastonce. • Fewer testcasesarerequiredwiththisapproachthanareneededinbasis-pathtesting. bool isLetter(char value) { if (value >= 65 && value <= 90) { return true; } // A-Z else if (value >= 97 && value <= 122) { return true; } // a-z return false; } RECALL: C++ isshort circuited, so statement-level coverage may not fully test ifstatements.

38. Statement-levelCoverage,example • Thefirstcasetestsapathtoreachthefirst‘true’return condition. • Possibleinputvalue:70 • Thesecondtestcasereachestheother‘true’return condition. • Possibleinputvalue:105

39. Statement-levelCoverage,example • Finally,wemustmakesurethatourtestreachesthe‘false’ returncondition,statement3. • Possibleinputvalue: 40. • Conclusions: • Wemanagedtocoverallstatementsinourcodewith onlythreetestcases. • However,ourtestingisnotas‘bulletproof’asthebasis-path testcases.

40. White-BoxTestingApproaches • Branch-levelcoverage: • Hasthegoaloffollowingeachtrue/falsebranchofaprogram. bool isLetter(char value) { if (value >= 65 && value <= 90) { return true; } // A-Z 3. if (value >= 97 && value <= 122) { return true; } // a-z else { return false; } }

41. Branch-levelCoverage,example • Firstwetestthe‘true’condition(branch)forthefirstifstatement. • Thisallowsustocover • lines1&2. • Nextwetestthe‘false’branchfollowedbya‘true’branchonthesecondif statement. • Thiscoverslines3&4. TRUE FALSE TRUE

42. Branch-levelCoverage,example • Finallywetestthe‘false’branchofthesecondifstatement. • Thiscoverslines5&6. FALSE FALSE • Conclusions: • Eventhoughbranch-levelcoverageisnotconcerned with testingindividuallines,ithasthataddedbenefit. • Branch-levelcoverageisamorerobustalternativetostatement-levelcoverage,butaddstothenumberoftestcases.

43. White-BoxTestingApproaches • Condition-levelcoverage: • Hasthegoaloftestingbothtrueandfalseconditionsfor allbranchconditions. • Thisisespeciallyapplicabletocompound‘if’statements. Example: bool isLetter(char value) { COND #1 COND #2 <= 90) • We can see that there are four distinct conditionals in this code. • Each one must be tested true and false at least once. 1. 2. if (value >= 65 && value { return true; } // A-Z COND #3 COND #4 <= 122) 3. 4. 6. 7. if (value >= 97 && value { return true; } // a-z else { return false; } }

44. Condition-levelCoverage,example bool isLetter(char value) { TRUE TRUE if (value >= 65 && value <= 90) { return true; } // A-Z DON’T CARE DON’T CARE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } • First,wetestbothtrueconditionsonthefirst‘if’ statementanddon’tcareaboutthesecondifstatement. • Next,wetesttrue/falsecombinationsonboth‘if’ statements.Notetheseconditionsarecalculatedtotestas-yet-untestedconditions. bool isLetter(char value) { TRUE FALSE if (value >= 65 && value <= 90) { return true; } // A-Z TRUE FALSE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } =Truecond.onlycovered =Falsecond.onlycovered =Bothconditionscovered

45. Condition-levelCoverage,example bool isLetter(char value) { • Finally,wemustdesigntestcasesthatcovertheremainingconditions. • Wecanaccomplishthis withtwomoretestcases. • Conclusions: • Decision-level coverageisverythoroughandrobust. • Eliminatessurpriseerrorsduetoshort-circuitingthatcanescapemanytestingapproaches. 4. 6. 7. { return else { return true; } // a-z false; } } bool isLetter(char value) { FALSE DON’T CARE if (value >= 65 && value <= 90) { return true; } // A-Z FALSE TRUE 3. if (value >= 97 && value <= 122) 4. { return true; } // a-z else { return false; } } =Truecond.onlycovered =Falsecond.onlycovered =Bothconditionscovered

47. IntegrationTestMethods • SoftwareIntegrationTesting:“…[the]testinginwhichsoftwarecomponentsarecombinedandtestedtoevaluatetheinteractionbetweenthem…” • –IEEEStandardComputerDictionary • Integrationtestingcandetectinconsistenciesinparalleldesign(assumptionsthatdifferacrossmultiplemodulesbecauseeachwaswrittenbyadifferentengineer). • Incontrasttounittesting,wherewemighthaveavoided(or‘stubbed’)callstoothermodules,integration testingwillfocusontheinterfacebetweensoftwaremodules.

48. IntegrationTestMethods • Therearetwomainapproachestosoftwareintegrationtesting: • Incremental: • Onepieceofthepuzzleatatime,followingageneral order. • » Top-Down • » Bottom-Up • Non-Incremental: • Nospecifiedordertointegration,ifatall. • » ‘Big-Bang’

49. IncrementalTesting • Top-DownIntegration: • Startsatthetopoftheprogramhierarchyandtravelsdownitsbranches. • Twoapproachestotop-downintegration: • breadth-first • depth-first

50. IncrementalTesting • Breadth-firstintegration: • Followsacrossthehierarchy‘lefttoright,’completingcoverageofeachlevelbeforemovingdownto • thenextone. • Thisintegrateseachofthecoreunitsbeforeincludingthelessgeneralmodules. • Thiscanrequiretheuseofstubsforlower • modulesnotyetincluded.