1 / 32

An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection

An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection. Giedre Sabaliauskaite* Fumikazu Matsukawa** Shinji Kusumoto** Katsuro Inoue** *Graduate School of Engineering Science, Osaka University

bachyen
Download Presentation

An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Experimental Comparisonof Checklist-Based Readingand Perspective-Based Readingfor UML Design Document Inspection Giedre Sabaliauskaite* Fumikazu Matsukawa** Shinji Kusumoto** Katsuro Inoue** *Graduate School of Engineering Science, Osaka University **Graduate School of Information Science and Technology, Osaka University

  2. Contents • Background • Research Goals • Experiment description • Reading techniques • Experimental design • Data analysis • Results • Conclusions and Further Work

  3. Key activity – defects are detected Reading techniques are used to guide inspectors Software Inspection • Software inspection is an effective and efficient method to detect defects in early stages of software development life-cycle • Software inspection usually consists of several activities [1] Planning Defect detection Defect collection Defect correction [1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31.

  4. Reading techniques • Ad hoc and Checklist-based Reading are the most popular [1] • Ad hoc: no guidance for inspectors • Checklist-based Reading (CBR):Checklist • Scenario-based Reading is a more recent approach: Scenario • Perspective-based Reading (PBR) [2] is one of Scenario-based approaches • Software product is inspected from different perspectives (designer, tester, etc.) • Provides inspectors with different Scenario for each perspective • We analyzed CBR and PBR inspection techniques during experiment [1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31. [2] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, M.V. Zelkowitz, “The Empirical Investigation of Perspective-Based Reading”, Empirical Software Engineering: An International Journal, vol.1 , no. 2, 1996, pp. 133-164.

  5. Related research • Several works are done in the area of Unified Modelling Language (UML) diagram inspection • One of the works is described in [1] • Experimental comparison of CBR and PBR • 18 subjects (practitioners) and 2 software systems • UML diagrams inspected – Class, State, Sequence, Collaboration • Results - PBR 3-person teams exhibited higher effectiveness and lower cost per defect than CBR teams • Authors of different studies came to the conclusion that UML inspections need to be further investigated [1] O. Laitenberger, C. Atkinson, M. Schlich, K. El Emam. An experimental comparison of reading techniques for defect detection in UML design documents. The Journal of Systems and Software, 53: 183-204, 2000.

  6. Research Goals • Conduct UML diagram inspection experiment in Osaka university • Language used during experiment - Japanese • Compare Reading techniques CBR and PBR with respect to • Defect detection effectiveness • Cost per defect • Time spent on inspection • Compare both individual inspector results and team results • Acquire experience for future inspection

  7. Structure of CBR and PBR techniques, used during experiment • CBR • Diagram-specialized Checklist developed • Included 20 questions • PBR • Three perspectives: User, Designer, Implementer • Three scenarios (one for each perspective) developed • Scenarios consisted of several steps, each of them included • Explanations • Tasks to perform • Questions to answer

  8. Assignment of UML diagrams • All inspectors were given system requirement description and Use-Case diagrams • Defects were injected into Activity, Class, Sequence and Component diagrams

  9. Experiment design • Subjects • 59 third year students of the Software Development course, Osaka University • Objects • Seminar information system ( 24 pages ) • Hospital information system ( 18 pages )

  10. Defects • Three types of defects inserted into UML diagrams • Syntactic – missing or unnecessary information • Semantic – misrepresentation of a concept, or unclear representation • Consistency – inconsistency between UML diagrams • In total, 15 defects inserted into the each software system • 4 into Activity diagrams • 3 into Class diagrams • 5 into Sequence diagrams • 3 into Component diagrams

  11. Experimental Hypotheses • H01: subjects who use PBR technique should spend less time on inspection • H02: subjects who use PBR should have higher cost per defect • H03: individual defect detection effectiveness should be different between CBR and PBR inspectors • H04: team defect detection effectiveness should be different between CBR and PBR inspector teams

  12. Data analysis Two types of analysis • Individual data analysis • Defect detection effectiveness • Time spent on inspection • Cost per defect • Team data analysis • Subjects assigned into 3-person virtual teams • Effectiveness of CBR and PBR teams compared

  13. Both CBR and PBR exhibited similar results Percentage of inspectors, who have detected defects relevant to different perspectives

  14. Syntactic defects: CBR more effective Semantic and Consistency defects: PBR more effective Defect detection effectiveness (according to defect types)

  15. Sequence Diagrams: CBR and PBR exhibited similar effectiveness Activity and Component Diagrams : PBR more effective Class Diagrams: CBR more effective Defect detection effectiveness (according to UML diagram types)

  16. CBR Effectiveness ≈PBR Effectiveness CBR: 39% (4 min / defect) lower cost per defect PBR: 18% (11 min) less time spent on inspection Independent samples t-test Mann-Whitney test • Significant difference in Inspection Time and Cost • No significant difference in Effectiveness Time spent on inspection, Cost per defect and Defect detection effectiveness

  17. Subject assignment into 3-person virtual teams Any three CBR reviewers are included into team One reviewer using each of the three PBR perspectives included into team CBR group C1 C2 C3 C4 C5 C6 C7 C8 C9 • Grouping virtual teams into statistically independent groups • CBR groups: 3 teams included (11 inspectors for Seminar system, 10 subjects for Hospital system) • PBR groups: 6 teams included (7 Users, 6 Designers, 6 Implementers for each systems) PBR group U1 D1 I1 U2 D2 I2 U3 D3 I3 U4 D4 I4 U5 D5 I5 U6 D6 I6 Virtual teams

  18. Number of comparisons: • Seminar system • Hospital system Comparisons between CBR and PBR teams CBR PBR Comparisons CBR group 1 PBR group 1 CBR group 2 … … PBR group 2 …

  19. CBR teams more effective than PBR teams Team data analysis (1/2) • CBR and PBR teams were compared with respect to Team defect detection effectiveness • Number or comparisons, in which either CBR or PBR was more effective, or both techniques showed similareffectiveness

  20. CBR teams more effective than PBR teams Team data analysis (2/2) • T-test with significance level of 2.5% was used to evaluate in which comparisons there was a significant difference between CBR and PBR team effectiveness • Two types of comparisons made • All defects included • Syntactic defects omitted

  21. Experiment results • Individual inspector results • Inspectors who use PBR spend on average 18% less time on inspection • Cost per defect of Inspectors who use PBR is on average 39% higher • There is no statistically significant difference in defect detection effectiveness between individual CBR and PBR inspectors, however on average • PBR is more effective for Semantic and Consistency defects • PBR is more effective for Activity and Component diagrams • CBR is more effective for Syntactic defects • CBR is more effective for Class diagrams • Three-person virtual team results • CBR teams exhibit higher defect detection effectiveness than PBR teams

  22. Threats to validity • Internal validity • Selection: experiment was a mandatory part of the course • Instrumentation: objects which we used could have influence • External validity • Students were used as subjects • Design documents were similar to those which are used in practice, but the size of systems in industry is usually larger • There were threats to internal and external validity, but they were not considered large in this experiment

  23. Conclusions and Further research • UML diagram inspection experiment was conducted • The results of the experiment indicate that • There is no statistically significant difference in effectiveness of CBR and PBR individual inspectors • CBR virtual teams are more effective than PBR virtual teams • Future research will be directed to further investigation of UML diagram inspection • A replication of experiment was conducted in July 2002, during which real team meeting were performed • More detail analysis of both individual and team data • Research of Fault Content Estimation methods

  24. Software Engineering Laboratory http://iip-lab.ics.es.osaka-u.ac.jp/

  25. ADDITIONAL SLIDES: 26-32

  26. Independent variables Reading technique (CBR and PBR) Dependent variables Number of defects found Time spent on inspection Defect detection effectiveness Cost per defect Null Hypotheses For individual inspectors H01: PBR Time > CBR Time H02: PBR Cost < CBR Cost H03: PBR Effectiveness = CBR Effectiveness For 3-person virtual teams H04: PBR Effectiveness = CBR Effectiveness Experiment Variables and Hypotheses

  27. Information systems inspected • Seminar information system • System supports the process of a company which organizes seminars • Includes activities from seminar planning to issuing the qualification certificate • Designs relationships among personnel of seminar company, participants, lecturers, etc. • Hospital information system • System supports the process of a hospital • Includes activities such as patient registration, medical examination, treatment, prescription of the medicines, etc. • Designs the relationships among the personnel of a hospital, patients, pharmacists, etc.

  28. Collected data • Two types of data collected during experiment • Defect data • Time data

  29. CBR Checklist

  30. PBR Scenario

  31. Defect registration form

  32. Software development process • The main steps of software development process • Development of Use-case diagrams • Describing system activities in Activity diagrams • Defining static structure of the system in Class diagrams • Modelling dynamic aspects of the system in Sequence diagrams • Detailed description of object states in Statechart diagrams • Development of the Component diagrams • User’s perspective in our experiment covered the second, and partially the third and the fourth steps • Designer’s perspective covered the third and the fourth steps • Implementer’s perspective covered the sixth step, and partially the third and the fourth steps

More Related