Object-Oriented Software Quality Engineering Fall 2008

1. Object-Oriented Software Quality Engineering Fall 2008

2. Course Overview

3. Logistics • Instructor: Soheila Bashardoust • Office hours: Wednesdays 2:30-4:30PM @ HP5270 • E-mail: sbtajali@scs.carleton.ca • Any relevant material to be put in: www.scs.carleton/~sbtajali/4004 • slides will refer to (but not include) pictures of the textbook: bring it to each lecture! • slides will be posted on the course web-site every week

4. Warnings!! • Prereqs: • you MUST have taken COMP 2004 and COMP 3004 • this course involves MODELING and Java expertise • model creation will be VERY time consuming! • there is fixing more than programming per se • November 7'th: deadline to drop... • Paul Menton Centre: • Students with disabilities requiring academic accommodations in this course are encouraged to contact a coordinator at the Paul Menton Centre for Students with Disabilities to complete the necessary letters of accommodation. After registering with the Paul Menton Centre, make an appointment to meet and discuss your needs with ME at least two weeks prior to the in-class final exam. This is necessary in order to ensure sufficient time to make the necessary arrangements.

5. Books • Compulsory: • R. Binder, Addison-Wesley, 2000 • Testing Object-Oriented Systems • I repeat, I will be referring to BUT NOT POSTING the figures of the book! • Other useful design references: • Hassan Gomaa, Addison-Wesley, 2000 • Designing concurrent, distributed real-time applications with UML • B. Douglass, Addison-Wesley, 1999 • Doing Hard Time

6. Objectives of this Course At the end of this course, you should: • Have a good understanding of how to build testable models. • Roughly put, testable models are models from which tests can be derived • Have a basic understanding of the techniques used to derived tests from testable models. • Understand the issues involved in: • testing classes (esp. in Java)‏ • maintaining traceability between testable models, code and tests.

7. Learning Method • In the assignments, you will: • develop testable models for a specific case study supplied by the instructor • study the Java code corresponding to the supplied models and augment it with assertions (discussed late in the term)‏ • generate tests and execute test cases in order to validate this code and verify it against the models you will have produced • more on tests vs. test cases, validation vs. verification later! • document traceability between models, code, tests and test cases • investigate support for SQE in Eclipse • time permitting: • make necessary corrections and link them to previous versions • In class, we will: • study Binder ’s material • ATTEMPT TO relate his material to the assignments!

8. Course Deliverables • Iteration 1: (test model creation and use) due Oct 20 • study models and code supplied for the J-Unit case study • develop a suite of testable models • this will require augmenting/correcting some of the supplied models as well as developing other testable models • produce a suite of tests for the validation and verification of the system as a whole, as well as for a specific class • select some test cases from these tests • Iteration 2: (model/code-based testing) due Nov 20 • develop a partial contract for specific classes • investigate support for contracts, test cases, test models, test strategies and traceability in Eclipse • use all what you have learnt to test as much as possible the supplied system: document tests and try to find faults!! • Final Exam: open book exam on all material discussed in the course...

9. More Logistics • Marking scheme: • 25% for the first iteration • 25% for the second iteration • 5% for presentation • 45% for the final • About marking: • marking in this course is admittedly subjective: • the responsibility is YOURS to provide all necessary explanations… A posteriori arguments are not accepted! • debates about marks will NOT be allowed • marking will be relative to others and to ‘ planted ’ errors in the material supplied by the instructor! • Assignments may be done alone or preferably in pairs or triples! • No consideration for vanishing members, work distribution, etc.

10. And in the rest of this lecture We shall: • Motivate the importance of Software Quality Engineering • Give you an idea of what the viewpoint of this course will be • Overview what you should already have a basic understanding of

11. Crisis?What Crisis? Quality?What Quality?

12. Hardware vs Software • Hardware costs are decreasing: • hardware is fast and generally reliable • Complexity is now mostly introduced at the software level • And software is constantly maintained and evolved. • Zero-defect programming is still a pious wish! • only 0.3% faults on parachutes does not cut it... • Formal (i.e., math based) approaches (in terms of completeness and proofs) are used for safety-critical applications. • How are we doing with respect to quality in general?

13. Yourdon: The Reality of Software • Software is commonly delivered late, way over budget, and with unsatisfactory quality • Software technologies are seldom evaluated • Software development processes are unstable and uncontrolled • Software quality is poorly measured, monitored, and controlled. • Famous software failures are frequent: • … read on ….

14. Famous Software Bugs • Therac-25 (1985-1987)‏ • Six people were overexposed during radiation treatments for cancer by Canada's Therac-25 radiation therapy machine.  Three of these patients were believed to have died from the overdoses.  The root cause was a lack of quality assurance, which lead to an over-complex, inadequately tested, under-documented system developed, and subsequently to the failure to take adequate corrective action. (Pooley & Stevens, 1999)‏ • London Ambulance System (1992)‏ • A succession of software engineering failures, especially in project management, caused 2 failures of London's (England) Ambulance dispatch system.  The repair cost was estimated at £9m, but it is believed that people died who would not have died if ambulances had reached them as promptly as they would have done without the failures. • Denver baggage handling system • The Denver airport baggage handling system was so complex (involving 300 computers) that the development overrun prevented the airport from opening on time.  Fixing the incredibly buggy system required an additional 50% of the original budget - nearly $200m.

15. Famous Software Bugs • Taurus (1993)‏ • Taurus, the planned automated transaction settlement system for the London Stock Exchange was canceled after 5 years of failed development.  Losses are estimated at £75m for the project and £450m to customers. (Pooley & Stevens, 1999)‏ • Ariane 5 (1996)‏ • The Ariane 5 rocket exploded on its maiden flight in June [4], 1996 because the navigation package was inherited from the Ariane 4 without proper testing.  The new rocket flew faster, resulting in larger values of some variables in the navigation software.   Shortly after launch, an attempt to convert a 64-bit floating-point number into a 16-bit integer generated an overflow.  The error was caught, but the code that caught it elected to shut down the subsystem.  The rocket veered off course and exploded.   It was unfortunate that the code that failed genereated inertial reference information useful only before lift-off; had it been turned off at the moment of launch, there would have been no trouble. (Kernighan, 1999)‏

16. Famous Software Bugs • E-mail buffer overflow (1998)‏ • Several E-mail systems suffer from a "buffer overflow error", when extremely long e-mail addresses are received.  The internal buffers receiving the addresses do not check for length and allow their buffers to overflow causing the applications to crash.  Hostile hackers use this fault to trick the computer into running a malicious program in its place. • USS Yorktown (1998)‏ • A crew member of the guided-missile cruiser USS Yorktown mistakenly entered a zero for a data value, which resulted in a division by zero.  The error cascaded and eventually shut down the ship's propulsion system.  The ship was dead in the water for several hours because a program didn't check for valid input.  (reported in Scientific American, November 1998)‏ • Mars Climate Orbiter (September 23rd, 1999)‏ • The 125 million dollar Mars Climate Orbiter is assumed lost by officials at NASA. The failure responsible for loss of the orbiter is attributed to a failure of NASA’s system engineer process. The process did not specify the system of measurement to be used on the project. As a result, one of the development teams used Imperial measurement while the other used the metric system of measurement. When parameters from one module were passed to another during orbit navigation correct, no conversion was performed, resulting in the loss of the craft. http://mars.jpl.nasa.gov/msp98/orbiter/

17. Books to read by Ed Yourdon : • Death March: The Complete Developer’s Guide to Surviving “Mission Impossible” Projects. Prentice-Hall, 1997 • Rise and resurrection of the American programmer, Prentice-Hall, 1996

18. Yourdon’s Death March Projects • Definition 1: Project parameters exceed the norm by >50% because of: • Schedule compression (most common)‏ • Staff reduction • Budget/resource constraints • Functionality/performance demands • Definition 2: risk assessment (technical, legal, political, etc.) indicates >50% chance of failure... • Observation: this is now the norm, not the exception!!

19. Yourdon’s Explanations • Politics, politics, politics: lack of culture of quality!! • Naive promises made by marketing, senior executives, project managers, etc. • Naive optimism of youth: “we can do it over the weekend” • “Startup” mentality: we’ll go berserk for a few years • “Marine Corps” mentality: “real men don’t need sleep” • Intense competition caused by globalization of markets • Intense competition caused by appearance of new technologies: CORBA and Java as complexity enabling… • Intense pressures caused by unexpected government regulations • Unexpected and/or unplanned crises — e.g., your vendor went bankrupt, or your 3 best programmers just died of Bubonic Plague

20. Why Software Engineering? • We are always looking for recipes and tools for software development. • “It’s better than nothing” is a standard excuse to downplay the flaws of current notations, methods and tools: • there are some serious issues (e.g., the layout of statecharts, R. Castello et al., Software - Practice and Experience, 2001) that should question what the dominant paradigm suggests as a solution... • In theory, we want to: • do the right things • do things right • know that we have done the right things • know that we have done things right • In practice, first-to-market software will often gain acceptance, then will condition its users to expect less, and finally will be eventually improved to a point that is almost the right thing. • In practice, most often, code IS the only reality, with models, if any, quickly falling out of sync. And code and error fixing are usually still the basis for your performance evaluation.

21. About Software Engineering • A huge field: • “Software Engineering”, IEEE Computer Society, 1997 • A possible definition for Software Engineering from Tim Lethbridge (UofO): : • “Solving problems by the development of quality software within cost, time and other constraints.” • B. Douglass (“Doing Hard Time”) has a detailed OO development process for real-time systems but has nothing to directly say with respect to quality. Similarly, H. Gomaa has a wonderful book about the design of OO real-time systems but ignores quality issues altogether... • D’Souza and Wills (“UML,components and framekworks: The Catalysis Approach”) refer to the integrity of software as meeting business requirements. They also value team development and flexibility as key requirements for software. • We are concerned in this course with what quality means and how to achieve it?

22. About Software Development • Types of constraints: • Limited cost, time, supply of people, abilities of people • Inherent complexity of the task we tackle • Limitations of hardware, of compilers, of algorithms, etc. • Legacy code, processes, habits • Human factors in quality emphasize the fact that software development has a engineering and a social side: • specific engineering decisions are often permeated by human biases (wrt notation, use of patterns, use of O-T-Shelf components…, etc.)‏ • Software development as an investment: • There are expected benefits, but also potential risks • Pious wish: We should not compromise on quality even though we do have constraints: • But in fact, this compromise is omnipresent in the software industry… • The questions remain: • What is quality software? • What to do in order to obtain quality s/w?

23. Definitions • S/W Quality Engineering (SQE): The discipline of specifying and controlling the quality of software products • S/W Management: The discipline of managing projects to achieve quality within time constraints and budget • S/W (Quality and Productivity) Improvement: The discipline of identifying weaknesses in current S/W development practices and implement improvements. • Only SQE is the topic of this course! • And in fact, we are even more specifically looking at the use of test(able) models for testing code.

24. About Quality Software • According to Lethbridge: • “Quality software is software that is sufficient in various aspects such that utility over the maximum reasonable life-span is maximized.” • Utility is informally defined in terms of cost and productivity… • Is this a useful definition? Sufficiency? Productivity? • For Bob Probert (UofO): • Quality should be defined using customer-directed terms • Quality attributes must be measurable to some useful degree during design, development and test. • Aspects (or attributes) of quality include: completeness, correctness, understandability, usability, efficiency, reusability, maintainability, etc. • If we had no constraints, we could, at least in theory, achieve infinite quality… • In reality, testing is almost unavoidably incomplete… • see quote at the very beginning of Binder’s book...

25. SQE and Measurement • Usual ‘goals’ of SQE: • Monitor and control quality and productivity • Predict and estimate quality and productivity • Understand and evaluate software development practices • Evaluate the impact of new technologies • Monitoring, estimating, evaluating all depend on the ability to measure...

26. Typical Focus in SQE Most authors concerned with quality emphasize the importance of metrics and measurement: • “You cannot control what you cannot measure”, Tom DeMarco • “Projects without clear goals won’t achieve their goals clearly”, Tom Gilb • “Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way to describe them accordingly to clearly defined rules” Norman Fenton

27. Focus in this Course • We will emphasize product quality engineering, and downplay process issues and measurement. • We will emphasize code testing against testable models. • We will not emphasize: • Software maintenance and evolution • “Software Evolution”, L.J. Arthur, John Wiley & Sons, 1988 • Change management; impact analysis; corrective, adaptive, perfective maintenance; reengineering, maintenance testing, configuration management, managing maintenance • Software release • “Software release methodology”, M. Bays, PH, 1999: • Source code control, builds, defect tracking, system integration, change control, product release classification and numbering, release distribution, service strategies, release management and planning • The mathematics of the art of measurement: • “The art of measurement”, R. Potter, HP, 2000 • Software Management and Software Improvement

28. Pervasive SQE • It is well documented that late errors can be extremely expensive to fix. • We will emphasize pervasive SQE via the use of testable models • conversely, the dominant culture still reduces SQE to testing as a phase at the end of the process… • in practice, a lot of testing occurs after the product has been released… SIGH! • >65% (and up to 80%)of the software lifecycle is spent in maintenance (ie testing, fixing, evolving…)‏ • The more statements need to be modified, the less successful the fix or evolution will be… • Some companies force rewrites when faced with hard-to-find, hard-to-fix bugs… • We will promote SQE activities in parallel with design and code: • design for testability and code for testability are NOT dominant ideas at this point in time… Alas...

29. SQE Texts (1)‏ • Software Verification and Validation • S. Rakitin, Artech House, 1997 • Talks about the software crisis, the economic justification of SQE, why a process for quality. • Promotes an inspection process for requirements, design, code and tests • Addresses software quality metrics, configuration management, testing, validation metrics and reliability very concisely! • Reflects what most companies will do at best… • Customer Oriented Software Quality Assurance • F. Ginac, PH, 1998 • Another short book that addresses SQE from a practical viewpoint • Touches on quality attributes and metrics; test methods, types and tools; CMM and ISO 9000 appraisal of processes

30. SQE Texts (2)‏ • Metrics and Models in Software Quality Engineering • S. H. Kan, Addison-Wesley, 1995 • Excellent discussion of process models and process quality standards • Good introduction to fundamentals of measurement theory, software quality metrics, complexity metrics • Includes more advanced and specific topics, such as defect removal effectiveness, reliability growth models, analyzing customer satisfaction • Please note that Prof. Lionel Briand in Systems and Engineering has a graduate course (94.579) on measurement. He uses: • N. Fenton, S. Pfleeger, Software Metrics: A Rigorous and Practical Approach, PWS, 1996 • B. Kitchenham, Measurement for Process Improvement, 1995, out of print

31. Course Topics • Introduction to the fundamental concepts, terminology and techniques of SQE • Exposure to the models and test derivation techniques used in industry for object-oriented testing: • we are concerned with testable models and how to use them to get tests and test cases • why testable models? Because it is well documented that most ( > 80%) errors are design-related errors… • And code must be tested against models… rather than by itself! • we are concerned with code testing • we are concerned with traceability (eg for regression testing)‏

32. Quality • What do we mean by Quality?

33. The Timeless Way of Building • Christopher Alexander, Oxford University Press, 1979, (Detailed Table of Contents, ix-xv)‏ • The Timeless Way • A building or town will only be alive to the extent that it is governed by the timeless way. • 1. It is a process which brings order out of nothing but ourselves; it cannot be attained, but it will happen of its own accord, if we will only let it.

34. The Quality • To seek the timeless way we must first know the quality with out a name • 2. There is a central quality which is the root criterion of life and spirit in man, a town, a building or a wilderness. This quality is objective and precise, but it cannot be named. • 3. The search which we make for this quality, in our own lives, is the central search of any person, and the crux of any individual person's story. It is the search for those moments and situations when we are most alive. • 4. In order to define this quality in buildings and in towns, we must begin by understanding that every place is given its character by certain patterns of events that keep on happening there. • 5. These patterns of events are always interlocked with certain geometric patterns in space. Indeed, as we shall see, each building and each town is ultimately made out of these patterns in the space, and out of nothing else; they are the atoms and the molecules from which a building or town is made.

35. The Quality • 6. The specific patterns out of which a building or a town is made may be alive or dead. To the extent they are alive, they let out inner forces loose, and set us free; but when they are dead, they keep us locked in inner conflict. • 7. The more living patterns there are in a place - a room, a building, or a town - the more it comes to life as an entirety, the more it glows, the more it has that self-maintaining fire which is the quality without a name. • 8. And when a building has this fire, then it becomes a part of nature. Like ocean waves, or blades of grass, its parts are governed by the endless play of repetition and variety created in the presence of the fact that all things pass. This is the quality itself.

36. The Gate • To reach the quality without a name we must then build a living pattern language as a gate. 9. This quality in buildings and in towns cannot be made, but only generated, indirectly, by the ordinary actions of people, just as a flower cannot be made, but only generated from the seed. • 10. The people can shape buildings for themselves, and have done it for centuries, by using languages which I call pattern languages. A pattern language gives each person who uses it the power to create an infinite variety of new and unique buildings, just as his ordinary languages gives him the power to create an infinite variety of sentences. • 11. These patterns languages are not confined to villages and farm society. All acts of building are governed by a pattern language of some sort, and the patterns in the world are there, entirely because they are created by the pattern languages which people use. • 12. And, beyond that, it is not just the shape of towns and buildings which comes from pattern languages - it is their quality as well. Even the life and beauty of the most awe-inspiring great religious buildings came from the languages their builder used. • 13. But in our time the languages have broken down. Since they are no longer shared, the processes which keep them deep have broken down; and it is therefore virtually impossible for anybody, in our time, to make a building live.

37. The Gate • 14. To work our way towards a shared and living language once again, we must first learn how to discover patterns which are deep, and capable of generating life. • 15. We may then gradually improve these patterns which we share, by testing them against experience; we can determine, very simply, whether these patterns make our surroundings live, or not, by recognizing how they make us feel. • 16. Once we have understood how to discover individual patterns which are alive, we may then make a language for ourselves for any building task we face. The structure of the language is created by the network of connections among individual patterns: and the language lives, or not, as a totality, to the degree these patterns form a whole. • 17. Then finally, from separate languages for different building tasks, we can create a larger structure still, a structure of structures, evolving constantly, which is the common language of a town. This is the gate.

38. The Way • Once we built the gate, we can pass through it to the practice of the timeless way. • 18. Now we shall begin to see in detail how the rich and complex order of a town can grow from thousands of creative acts. For once we have a common pattern language in our town, we shall all have the power to make our streets and buildings live, through our most ordinary acts. The language, like a see, is the genetic system which gives our millions of small acts the power to form a whole. • 19. Within this process, every individual act of building is a process in which space gets differentiated. It is not a process of addition, in which preformed parts are combined to create a whole, but a process of unfolding, like the evolution of an embryo, in which the whole precedes the parts, and actually gives birth to them, by splitting. • 20. The process of unfolding goes step by step, one pattern at a time. Each step brings just one pattern to life; and the intensity of the result depends on the intensity of these individual steps. • 21. From a sequence of these individual patterns, whole buildings with the character of nature will form themselves within our thoughts, as easily as sentences. • 22. In the same way, groups of people can conceive their larger public buildings, on the ground, by following a common pattern language, almost as if they had a single mind.

39. The Way • 23. Once the buildings are conceived like this, they can be built, directly, from a few simple marks made in the ground - again within a common language, but directly, and without using drawings. • 24. Next, several acts of building, each one done to repair and magnify the product of the previous acts, will slowly generated a larger and more complex whole than any single act can generated. • 25. Finally, within the framework of a common language, millions of individual acts of building will together generate a town which is alive, and whole, and unpredictable, without control. This is a the slow emergence of the quality without a name, as if from nothing. • 26. And as the whole emerges, we shall see it take that ageless character which gives the timeless way its name. This character is a specific, morphological character, sharp and precise, which must come into being any time a building or a town becomes alive: it is the physical embodiment, in buildings, of the quality without a name.

40. The Kernel of the Way • And yet the timeless way is not complete, and will not fully generate the quality without a name, until we leave the gate behind. • 27. Indeed this ageless character has nothing, in the end, to do with languages. The language, and the processes which stem from it, merely release the fundamental order which is native to us. They do not teach us, they only remind us of what we know already, and of what we shall discover time and time again, when we give up our ideas and opinions, and do exactly what emerges from ourselves.

41. Appendix Some Review Material (with an angle for testing)‏ (From 95.304…)‏

42. Divide and Conquer Strategy Faced with a complex problem our usual strategy is to Divide and Conquer • Break the problem into smaller, more manageable sub-problems and solve them separately. • There are different ways for dividing complex problems into simpler problems: -as steps or tasks to be carried out -as assignment of responsibility

43. SA/SD Modeling • Modeling tackled the system or a subsystem • testing was oriented towards testing of procedures (see next slide)‏ • Three kinds of models were used: • ER, DFD, FSM • Each model being used offered a view of the (sub)system being modeled as a whole: • e.g., Hatley/Pirbhai and Ward/Mellor advocated one FSM for the whole system • FSMs are apparently excellent for testing… • see Figure 7.3 p.182 for different FSM notations • Semantically, it was not clear how the different models being used should be integrated • e.g., were your DFD and FSM consistent?

44. Procedure Abstraction Programs = Data Structures + Algorithms Data Procedure Procedure Procedure Procedure Procedure Procedure Procedure

45. Data Abstraction Procedures and Encapsulated Data Stack Queue Employee Procedure Procedure Procedure Procedure Procedure Procedure Procedure

46. Modular Abstraction Objects Vehicle subclass: Car choke()‏ start()‏ injectFuel()‏ brake()‏ unlockBrake()‏ accelerate()‏ steer()‏ serialNumber engine wheels

47. Interacting Modules Object-Orientation Traffic Light Procedure Procedure Car Driver Procedure Procedure Procedure Procedure

48. Requirements Capture Analysis Design Main Models for OO Requirements Doc. Scenario Descriptions Domain Objects Requirements Models Structure Model Object Diagrams CRC Cards Analysis Models Scenario Model Interaction Diagrams Behavior Model Finite State Machines Design Models

49. A Few Facts • Notations are merely syntax: they don’t say how. • UML (Unified Modeling Language) is the dominant notation. • We will study Binder ’s criticism of UML wrt testability! • Methods edict rules: • no heuristic, let alone no rule, holds for all situations • OO system modeling is not a science but the realm of heuristics and patterns • Existing methods downplay validation and verification and generally ignore traceability • OO Assertional/Formal methods exist but: • are not very user-friendly • don’t necessarily scale up • are not very object-oriented • The semantics of inheritance complicates things! • and yet are at the basis of a multitude of testing papers...

50. Aiming for Traceability • Boehm insists that a development process must not only be iterative but also incremental. • Traceability is required to achieve convergence: • we must document the continuity that must exist between the work-products of different activities. • in turn, continuity enables completeness and consistency checks, which are at the basis of validation and verification. • Verification: Within a particular activity, the work-products must be consistent: • e.g., the structural, scenario and behavior models must be consistent • For example, if an interaction diagram shows an object receiving a message, then the FSM of this object must reflect this possibility • Validation: Work-products must also be complete with respect to the current requirements. • Traceability is key to successful change impact analysis, root cause analysis and regression testing.