Quality vs Testing ?

1. Quality vs Testing ? What is Quality ? What is Testing? How Related? Does finding a Lot of Problems through testing improve Quality?

2. Why Test ? • “Unfortunately” we can’t assume that the software “works” • How do we answer the question: “Does this software work?” • Is it functional ? • Complete • Consistent • Is it reliable ? (system is continuously functioning > 10,000 hrs) • Is it available (can you access the functionality when you want?) • Is it responsive ? (response time < .5 second) • In general, “what is the Quality of our software?” • How do we answer this? How would you answer this about the program that you wrote? How would you answer this about the program your friend wrote?

3. 2 Main Reasons for Testing • Assess the Quality & Acceptabilityof the (software) Artifact 2. Discover Problems in the (software) Artifact If we are already “satisfied” with the software, should we test? How do we know if we are satisfied? ---- based on what?

4. 3 major “Testing” approaches • Traditionally, testing includes executing the code with test cases. • (assuming - code is the main software artifact) • 2. What do we do with the non-executable software artifacts? • Reviews and Inspections • - Expensive in terms of human resources • - A lot to maintain and keep updated • 3. Can we “prove” that the software works or is defect free? • - Theoretically, given an arbitrary program we can not show that it has no bug. • - We can use Formal Proofs to show the behavior of code

5. There are Different Levels of Testing System Testing Functional Testing Component Testing Unit Testing Program unit A Function 1 Program unit B Component 1 Whole System . . . . Function 2 . . Component 3 Function 8 Program unit T Can we really keep the levels this simple?

6. The Effect of Software Process to Testing • Software process which develops a software by increments will naturally need to consider integration and system testing techniques: • Incremental development, where a preliminary high level design of the whole software is developed into components, which must be coded and tested separately (possibly in parallel) and then integrated together and retested. • Evolutionary development, where a small, but well understood, part of the software is first designed, coded and tested with additional parts developed later, must perform continuous integration and testing of the software • Spiral development, where pieces of software evolve through planning, risk assessment, design, coding, and testing, must integrate and retest as the pieces are accumulated. • Prototype specification based and agile process, where the software or parts of the software are prototyped and developed separately “specified”, must integrate and test these separately “specified” and developed software.

7. Review of Major Unit/Functional/ComponentLevel Testing Techniques • Black Box or Functional testing (test cases generated from requirements): • Boundary value • Decision/logic Table • Equivalence class • White Box or Structural testing (test cases generated from design and/or code) • All statements • All Branches (Decision-Decision) • Cyclomatic number • All logical path (not always doable)

8. Black Box:Boundary Value Testing • A type of “Black box” functional testing • The software takes inputs and “maps” some out-puts • The internal of the design/program itself is not considered • A technique to generate test cases via considering (mostly) the inputs to the program • The rationale for this focus is that there are experiences from the past which indicate that errors tend occur at the “extreme” points. • Input data (legal versus illegal) • Loop iteration (beginning and ending of loops) • Output fields (legal versus illegal) Test 5 points (for a lower and an upper boundary): upper boundary, just below upper, middle, lower boundary, just above lower

9. Robustness testing – Boundary Value • This is just an extension of the Boundary Values to include: • Less than minimum • Greater than maximum • There are 7 cases or values to worry about for each independent variable input. • The testing of robustness is really a test of “error” handling. • Do we anticipate the error situations? • Do we issue informative error messages? • Do we allow some kind of recovery from the error? Levels of Problem Severity?

10. Decision table is based on logical relationships just as the truth table. It is a tool that helps us look at the “complete” combination of conditions Black Box: Decision Table Based Testing

11. Components of a Decision Table rules R1 R2 R3 R4 R5 R6 R7 R8 T T T T F F F F C1 C2 C3 values of conditions conditions T T F F T T F F T F T F T F T F a1 a2 a3 a4 a5 x x x x x x actions taken x x actions x x x x x x Read a Decision Table by columns of rules : R1 says when all conditions are T, then actions a1, a2, and a5 occur

12. Black Box: Equivalence Class Testing • Use the mathematical concept of equivalence class to generate test cases for Functional (black-box) testing • The key goals for equivalence class testing are: • “completeness” of test coverage • lessen duplication of test coverage

13. Equivalence Class • Recall a partitioning of a set, A, is to divide the set A into ( a1, a2, - - - -, an ) subsets such that: • a1 U a2 U - - - - U an = A (completeness) • for any i and j, ai∩ aj = Ø (no duplication) • Equivalence relation is a relation defined on the set A such that it is reflexive, transitive, and symmetric • Equivalence class testing is the partitioning of the input variable’s value set into classes (subsets) using some equivalence relation.

14. “Flowchart” Graph from “Triangle-Decision” problem pseudo code White box: - All Statements coverage - 4 paths - Branch (DD-path) coverage - 4 paths - Cyclomatic # = 4+1 = 5 - 5 lin. Ind paths - All logical combinations - 8 paths first 1- 8 9 10 11 Triangle ~Triangle 12 Triangle ~Triangle 13 14 21 16 15 equilateral 17 18 scalene isosceles 19 20 22 Last

15. All Combination paths • Let’s look at the all 8 combination paths • P1: < 8,9,10,12,13,14,15,20,22> (Equilateral) • P2: <8,9,10,12,13,14,16,17,19,20,22> (Scalene) • P3: <8,9,10,12,13,14,16,18,19,20,22> (Isosceles) • P4: <8,9,10,12,13,21,22> (not possible) • P5: <8,9,11,12,13,14,15,20,22> (not possible) • P6: <8,9,11,12,13,14,16,17,19,20,22> (not possible) • P7: < 8,9,11,12,13,14,16,18,19,20,22> (not possible) • P8: <8,9,11,12,13,21,22> (Not a triangle) So, there are 4 decision-decision (dd) paths that make sense. - These are P1, P2, P3, and P8. - We should at least test these four paths.

16. Software Engineering & Configuration Management • In order to controlall the pieces and parts of the software artifacts, we need two basic models • Parts identification model • Parts storage and access model

17. Sample: PartsIdentification model • A software artifact must be uniquely identifiable with a “name” composed of some of the following: • PP : two position product code • CC: two position country code • RR: two position release code • VV: two position version code • TT: two position artifact type code • FF: two position format code A sample artifact identifier: PP.CC.RR.VV.TT.FF where “.” Is used as the delimiter; and then add “xxxx” part number to get PP.CC.RR.VV.TT.FF.xxxx or PP.CC.RR.VV.TT.xxxx.FF

18. Artifact “Parts” Storage and Access model for Configuration Management Parts Database Parts Control System build Individual user Individual user . . . .

19. Parts Storage and Access “model” for Configuration Management • Basic functions: • Create a part • Delete a part • Access functions • View a part • Modify a part • Return a part • Control and service functions • Import part(s) • Export part(s) • List parts • Set release or version numbers • Increment release or version numbers • Change part name, version, release, artifact type, etc. • Gather parts • Merge into a part • Promote parts • Compare parts • Lock / Unlock pars • Where used and Cross Reference the parts

20. Conflict Managementin Parts Control • Whenever a resource is accessed by more than one person, there is a potential for conflict: (for example) • Person A accesses a “part” x and a person B accesses the same “part,” x. • If they both want to just view it - - - no problem • If they both change the same part and return the changes, there is a potential problem because they are both based on the previously unchanged version. Returning and storing back the changes may override each other. • For changes, there must be a control which prevents any potential conflict in the changes by multiple people. • A solution is to use a check-out and check-inmechanism

21. A “key activity” - - handling versioning • A key component to the versioning function of configuration management is the technique of integrating a change: • Complete Replacement of the original with the new • This loses all the information of deletion, change, and add • Keeping the complete original and complete versions of the new changes • This takes a lot of space, even though storage is getting cheaper • History of changes require a “compare” of versions and extracting the information • Keeping the complete original version and keeping only the “delta”changes (delete, update, add) for each version • this saves storage, but pay a performance price for “merges on demand” • If the one original complete version is ever corrupted, one may have an impossible task of recreating any version. (low risk but possible) • Good for showing history of changes

22. System Build with Configuration Manager • Construct a build (dependency) list • Compile • Link • Generating the required executables that are ready to run on the specified platform

23. Where Configuration Manager is Used • Main areas of high configuration managers usage are: • Testing • Software builds • Test case tracking • Problem - fix tracking • Post release support • Problem – fix tracking • Fix release builds • Fix release tracking • Multiple releases • Tracking / building multiple releases within an application for one platform • Tracking and coordinating application with multiple platforms • Tracking and coordinating application releases for multiple country versions • Other areas of usage: • Requirements tracking • Design versioning • User document tracking

24. . . Statement 2’ Statement 3” (del) Statement 5’’ Statement 6’’’ . . . Example: Code Versions . . Statement 3’’ Statement 4’’ Statement 6’’ . Statement 11’ . . . Statement 3’ Statement 4’ Statement 5’ Statement 6’ . . . Statement 1 Statement 2 Statement 3 Statement 4 Statement 5 Statement 6 Statement 7 Statement 8 Statement 9 . . version 3 version 2 version 1 Release 2, containing 3 accumulated changes Release 1 Version 0

25. Example: Requirements Versioning General Requirements (Version 3) General Requirements (Version 2) French ( version 3 ) General Requirements French ( version 2 ) French ( version 1 ) French (version 2.1) Japanese ( version 3 ) Japanese ( version 2 ) Japanese ( version 1 ) Brazilian ( version 3) Brazilian ( version 2 ) Brazilian ( version 1 )