Exam 2012

1. Exam 2012 Tor Stålhane IDI / NTNU

2. Unit Testing - Possible QA Killer – 1 • With unit testing you can properly check over 90 percent of your code, and, unlike QA’s manual testing tools, unit tests that are built correctly and tested automatically can evolve with your codebase, testing the code in real-time. • As a platform for testing code , unit testing makes sense, costs less, and gets better quality software quicker. Automated unit testing and TDD is the next best thing when it comes to making great software. It allows developers to adapt their code to new features and changes on-the-fly.

3. Unit Testing - Possible QA Killer – 2 • Companies are realizing the inability of a QA team to adjust to design changes. • With QA you have a department dedicated to sweeping for errors and then dumping the issues back on the developers for more iterations and fixes. • Unit testing with Agile development promises working software once it leaves the developer’s hands.

4. 1a-1 – 1 Identify three important issues that the blogger has missed or misunderstood. Write one paragraph for each issue, explaining why it is important. • He seems to believe that QA’s only responsibility is to do testing – and even that QA will do only manual testing.We need both integration testing – all units work together – and usability testing – that the customer can use the system as intended. • In agile theory the customer shall give continuous feedback to the developers • In the real world it has been observed that getting active customer participation from somebody who can commit the customer’s company is difficult to achieve.

5. 1a-1 – 2 • He has forgotten that we cannot test everything. Part of the customer’s confidence in the product stems from the fact at the process – e.g. Scrum – has been done according to the rules, without cutting corners such as daily stand-up meetings or sprint retrospectives. • He has ignored the need for companies to learn from mistakes and undertake continuous improvement. • Each person in a project learns a lot from the sprint retrospectives • Several companies have identified the problem that these lessons are not spread to the rest of the company. • Continuous learning is the most important characteristic of a successful company.

6. 1b-1 In some sense the blogger is correct – QA must change in order to provide maximum benefits to an agile development project. 1b-1: Select one of the issues identified in 1a-1. How can this issue be handled to better fit in with agile development?

7. 1b1 – Continuous process improvement • Traditional SPI: QA departments collect and analyse data to make suggestions for SPI. This can be either PMA or analysis of process data – e.g. • New SPI: The traditional approaches may or may not be applied. We need a way to make all experiences from the daily stand-ups and from the sprint retrospectives available to the rest of the company through an experience-sharing forum or an experience repository. Other important improvements based on agile development • customer contact – especially customer feedback on product quality • identification of new and better tools for e.g. unit testing.

8. 2a – Change management Amedium size Norwegian IT company - 50 employees. They have good manual unit testing procedures and are able to find and fix most errors in the coding phase. As the amount of code has grown and the need for a faster development process has been increasingly obvious, the company management has decided that they need to bring in a testing tool. The company has contacted you to act as a consultant for selecting and introducing the new unit testing tool. Price is an issue, although it is not on the top of the agenda.

9. 2a-1: Select a unit testing tool Start with identifying • The shortcomings of the current test process – too slow and not able to handle the current code volume. • Other, related problems that the developers experience. • The developers’ wishes – tool characteristics important to them Make a table where each tool get one line and each characteristic has one column. Agree with the developers on: • What kind of scoring scheme we should use – e.g. ++, +, 0, -, -- or 1 to 10. • The relative importance of each characteristic. Collect information on available tools – both commercial tools and open source tools. Remember to include maintenance costs and the cost of learning to use the new tool efficiently.

10. 2a-2: Introduce the new unit testing tool We need to know what the developers • Want to happen – info from the tool selection activity and the filled in selection table. • Fear will happen. We need to make sure that we can prevent such issues or at least reduce them to a low level. Ways to identify the “fear-issues” can be interviews or questionnaires. The rest of the process consists of two sets of activities: • Enables the company to reap the benefits • Prevent the “fear-issues” from materializing.

11. 2b – Follow-up activities – 1 2b-1: How would you follow up on the introduction of the new unit testing tool to see if it • Is used as planned. • Has the planned effect on the development process. It is of paramount importance that you get an agreed-upon definition of the effect that you want to achieve – most likely testing efficiency. Without this definition, the rest is a waste of time.

12. 2b – Follow-up activities – 2 Simple follow-up:Interview a sample of developers each week or month. Ask two questions: • Are you using the new tool? If the answer is “No”, why? • Is the testing now more efficient than before? If the answer is “No”, why? More advanced follow-up: • Install a counter in the tool so that we can see how often it is used. As an alternative we could stick to the weekly questionnaire. • Use e.g. GQM to get an agreed-upon definition of testing efficiency and a set of metrics. Analyse the metrics regularly – e.g. each month – to see if we have a trend in the right direction.

13. Problem 3 – Improvement methods Our company is a small Norwegian company – 10 employees. They have had a lot of trouble with meeting the agreed-upon delivery dates for their projects and their customers complain about the user interfaces – mostly for their web-systems. Since you are hired in as a consultant, you have done your home work and have come up with some suggestions for improvement. Your suggestions fall into two broad strategies – improvement through • a long sequence of small changes • an improvement realized in three large change steps.

14. 3a-1: Strong and weak points of the two improvement strategies – 1 A long sequence of small changes • Pro: short time to introduction => short periods of a chaotic state which are easier to control. Each change can quickly bring small but visible improvements => more enthusiasm and commitment • Con: it will take a long time before the final improvement goal is reached => large changes in the market place before we are fully ready with our new process, service or product and possible large total investments before we can reap the full benefits

15. 3a-1: Strong and weak points of the two improvement strategies – 2 A few large changes • Pro: short time to the final goal => short period before we reap the full benefits. A higher probability of hitting a window of opportunity. • Con: large changes are risky => high risk of failing and a considerable period where we will be in a chaotic state.

16. 3a-2: Which strategy would you choose and why I would choose a long sequence of small changes for the following reasons: • Smaller and thus more controllable risks • Enables a small improvement as “a first doable step”. This is easy to achieve while still having some observable benefit, thus strengthening the case for SPI in the company. • Can easily be undone or redone if the market and our targets change

17. 3b – Follow-up on process changes 3b-1: How would you follow up on the change to control that it has the intended affects – more timely delivery and better user interfaces. Define • A small set of parameters that can be used • How we can be collected data for these parameters and analyse them to see whether the company is improving.

18. 3b-1 – 1 The most efficient way of arriving at a useful set of parameters is using GQM. The following is just a small set of possible parameters that can be used. Timely delivery: • Delay in days • Delay in percentages of the total development calendar time Data for timely delivery can be collected in a straight forward manner by the QA department or the project manager. We should do • trend analysis based on earlier mean value and the R-value • trend plot and validate through visual inspection. • analysis of the correlation between project sequence number and the amount of delay for delivery using Spearman correlation.

19. 3b-1 – 2 Better user interface: • Number of customer complaints related to the use interface • Number of user interface errors found during acceptance testing • A user interface satisfaction score from a sample of users after having gone through a user course. Data used to check for a better user interface is a bit more laborious. • Counting the number of customer complaints, we can use a trend analysis just as for timely deliveries. • Apply customer questionnaires to see how satisfied the customers are with our user interfaces. A questionnaire will also enable us to collect improvement suggestions from our customers.