Usability with Project Lecture 19 – 19/11/08

1. Usability with ProjectLecture 19 – 19/11/08 Dr. Simeon Keates

2. Exercise • Start/continue your user trials • If you want feedback on your group presentation – then prepare slides for Friday • NOTE – this is optional, but recommended

3. Cost-justifying usability

4. The need to cost-justify • Not all companies (and managers) appreciate the benefits of usability • They may cite other factors as more important • Examples: • Tight deadlines • Functionality already developed • Limited money/resources

5. Cost-justifying usability • However, these are often “false economies” • Examples: • Deadlines: • Releasing the “wrong” product on time is as bad (or worse) as releasing the “right” product late • Existing functionality: • Existing functionality should have nothing to fear from usability, if it is the “right” functionality • Limited resources: • Putting the “right” resource in at the “right” time can make the overall project more efficient

6. Rosson and Carroll “Usability Engineering” • “Cost-benefit analysis of usability activities contributes to more systematic usability engineering …” • “… BUT benefits are difficult to quantify, so estimates will often be overly conservative.” Issues: • Benefits (e.g. customer satisfaction) are harder to quantify and predict accurately • Costs are very concrete and easy to identify • Thus, can be difficult to justify usability…

7. Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 1) • Development of requirements scenarios • Validation/refinements of scenarios with users and customers • Development of basic-level task scenarios • Refinement of design scenarios with development team and customers • Development of information model • Review with team members • Development of paper prototypes • Walk-throughs with users of paper prototypes • …

8. Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 2) • … • Analysis of transcripts/report preparation • Development of interaction model • Review with team members • Development of running prototypes • Formative evaluation • Analysis of transcripts/report preparation • Detailed design and prototype-driven iteration of previous three steps All of these steps have “usability” costs

9. Typical usability benefits • Fewer downstream design changes • Increased sales (and consequently reduced time to profitability) • Reduced need for user training • Enhanced customer productivity • Reduced resources spent on customer support • Increased loyalty in customer base (repeat and referral sales) Many of these benefits identified in airport case study from 2 lectures ago…

10. A more scientific approach • Mantei and Teorey(*) examined the cost/benefit analysis in 1988 • We will look at their calculations • But first we need to examine their methodology • NOTES: • Costs are based on 1988 prices and techniques • Not all costs are required for every project – choose which ones carefully • Costs are indicative, not definitive • Assumes 32,000 lines of code to be used by 250 employees (*) Mantei and Teorey (1988) Cost/benefit analysis for incorporating human factors in the software lifecycle. Communications of the ACM 31(4): 428-39

11. Typical development stages in a prototyping lifecycle • Feasibility study • Requirements definition • Global design • Prototype construction • User evaluation • System implementation • Testing • Update and maintenance

12. Typical development stages in a Human Factors software lifecycle • Market analysis • Feasibility study • Requirements definition • Product acceptance analysis • Task analysis • Global design • Prototype construction • User testing and evaluation • System implementation • Product testing • User testing • Update and maintenance • Product survey

13. A breakdown of tangible costs

14. Further assumptions • 40 hours per week • 4.3 weeks per month • =>172 hours per working month (WM) Examples: • $6880/WM for a salaried employee (at $40/hr) • i.e. 172 * 40 • $10,320/WM for an external contractor • i.e. 172 * 60

15. Costs added by Human Factors stages • 1 – Cost of running focus groups • 2 – Cost of building product mock-ups • 3 – Expense of the initial design of a prototype • 4 – Expense of making a prototyping design change • 5 – Expense of purchasing the prototyping software • 6 – Cost of running user studies • 7 – Cost of creating (or renting) a user study environment (laboratory) • 8 – Cost of conducting the user survey

16. Cost 1 – Focus groups cost breakdown • Time cost of the individuals involved + small equipment cost • Individuals involved: • Participants • Moderator/facilitator • Video-taping/recording personnel • Any other observers/data analysts ~10 people 1 person ~3 people

17. Focus groups cost breakdown • Focus groups typically take 3 hours to run + 1 day to set-up and 1 day to dismantle • Minimum of 2 days to analyse the data • Moderator: Time for focus group + analysis • Support staff: Time for set-up + focus group + dismantling • Participants: Time for focus group [+ any preparation time] • A complete focus group analysis (3 consecutive groups) takes ~2 weeks • Used for Market Analysis and Product Acceptance Analysis

18. Focus groups cost breakdown

19. Cost 2 – Estimating product mock-up costs • Building a product mock-up involves constructing a false UI scenario in software and video-taping the scenario • Script has to be written • Someone has to execute the scenario • Videotape needs to be professional • Videotape mock-up is used in Product Acceptance Analysis • To focus groups to initiate discussions • To target users who are then asked to complete questionnaires about the use of the product and their response to it • Can also be used in marketing

20. Estimating product mock-up costs

21. Estimating product mock-up costs • The techniques just described are somewhat out of date • More usually accomplished via early development cycle prototypes • e.g. alpha and beta versions • Flash movies, etc. • Q - What to do if no software written? • A – Wizard of Oz!

22. Wizard of Oz Do Action A {Receive response to Action A} Do Action B, etc.

23. Wizard of Oz (The Wizard unmasked!) Do Action A Create response to Action A {Receive response to Action A} Do Action B, etc. Create response to Action B, etc.

24. Wizard of Oz explained • User interacts with a UI mock-up only • There is no significant software back-end • Remote user (the Wizard) “pretends to be the system” • Wizard creates the response based on expected system performance • Still needs a script and plenty of preparation • Imagine if the Wizard does not know a response!

25. Cost 3 – Estimating user survey costs • Used in the Product Acceptance stage to assess users’ responses to the product mock-up • Also used in Product Survey (after launch) to: • Determine the difficulties users have with the working system • Examine the tasks the system is being used for • Gather suggestions for changes to system For a user population of 250 employees • Typically half (125) would receive a user survey • A typical survey is 4 pages in length… • … and takes 30 minutes to complete

26. Estimating user survey costs

27. Cost 4 – Estimating initial prototype building costs • Cost breakdown for building a prototype does NOT include the design time • Only the building time • Typically 4 weeks • Most prototyping systems involve 2 stages: • Stage 1 – specify the connections between the screen displays • Stage 2 – design each individual screen layout • Alternatively (and more modern) • Stage 1 – specify the states between user interactions • Stage 2 – design the individual states and the alterations that take place because of user actions

28. Estimating initial prototype building costs • As the UI grows more complex, time required to build the prototype also increases • Cost of building a prototype is: C = S × ( a + b D) Where • C = Cost • S = Number of states • D = Average number of new details per state • a = Constant reflecting the cost of building a single state • b= Constant reflecting the cost of adding a single detail

29. Estimating initial prototype building costs

30. Cost 5 – Estimating costs for design changes to original prototype • Once the prototype is built, user studies will uncover problems with it • Difficulties learning and using the system • Design change suggestions will be made … • … incorporated into the design … • … and evaluated again, etc. … • … until the number and severity of problems are “acceptable” • The initial user studies will usually find the most and most major issues • Later changes should be minimal • Especially if the prototype is close to the final product design … • … which it should be if task analyses, user surveys, etc. have been used • Also, design changes should simply be updates of parts of the prototype … • … and not a complete re-design

31. Estimating costs for design changes to original prototype • The initial user studies will usually find the most and most major issues • Later changes should be minimal • Especially if the prototype is close to the final product design … • … which it should be if task analyses, user surveys, etc. have been used • Also, design changes should simply be updates of parts of the prototype … • … and not a complete re-design

32. Estimating costs for design changes to original prototype • Since changes should be restricted to parts of the UI, should only take ~1 day per change • The number of changes expected and amount of time per change depends upon complexity of the interface • Similar equation to cost of building a prototype

33. Estimating costs for design changes to original prototype

34. Cost 6 – Estimating the cost of purchasing the prototyping software purchase • Actual purchase cost • Also time spent deciding which one … • … and then learning it • 1988 prices - $10,000 for the software ($2,500 to $15,000+) • 2008 prices - $600 per user (AXURE)

35. Estimating the cost of purchasing the prototyping software purchase + Time spent learning package

36. Cost 7 – Estimating costs of performing user studies • Preparation time can be substantial • Example: preparing a manual for a completely new UI • Manual need not be complete, but it needs to be sufficient and pilot-tested • Example: preparing the study protocol • Protocol needs to be complete and pilot tested • Costs of individual user studies are often independent of complexity of UI … • … however, the number of user studies increases with complexity • 3 types of user study in the Human Factors software lifecycle…

37. Estimating costs of performing user studies In Task Analysis: • Users are asked to perform the types of tasks the system should support • Aim is to build a model of how users approach the task In User Testing and Evaluation and in final User Testing: • More conventional user studies • Conducted on prototypes or final system • Final system evaluation is always needed • May be significant changes from the prototype behaviour

38. Estimating costs of performing user studies

39. Cost 8 – Estimating costs of construction of a user laboratory • A user lab can be a borrowed / rented or permanent office • Permanent office is recommended where significant usability activity is expected • User lab should be a mock-up of the environment of use for the product • The lab should allow space for an adjoining observation/recording room • Construction of a lab takes ~5 weeks

40. Estimating costs of construction of a user laboratory

41. Estimating costs – a summary

42. Estimating times – a summary

43. Common tangible benefits of Human Factors design • Direct benefits can be calculated by making several valid assumptions about the improvements to the UI, specifically: • (1) A reduction in user learning times • (2) A reduction in user errors • (3) A reduction in the cost of maintaining the system • Remember: for 32,000 line program for 250 employees…

44. Benefit 1 – Potential training cost savings • It is estimated that the learning time for new system will be 25% lower • Learning time is typically 2 weeks of classes • 10 working days / 80 working hours • Staff turnover is 50 hourly employees and 50 salaried employees per year (Savings / year) = (Turnover) × (Training time save) × (Wage) = 50 × 20 × $15 = $15,000 per year for HEs = 50 × 20 × $40 = $40,000 per year for SEs

45. Benefit 2 – Potential error reduction cost savings • User “traps” exist in all Uis • i.e. a standard sequence of user responses that always result in an error • Errors may be trivial and easily corrected … • … but are often annoying and time-consuming • These traps are almost always the result of the UI violating the learned behaviour of the user • Example: driving an automatic car

46. Potential error reduction cost savings • How many errors per year? Assume: • User has 0.025 chance of falling into a “trap” • Company has 250 employees using software 3 hrs/day and 20 scenarios/hr Errors/year = (# of employees) × p(Error) × (scenarios/hr) × (hrs/year) = 250 employees × 0.025 × 20 × (21.5 dy/mo × 3 hrs/dy × 12 mo/yr) = 96,750 errors per year

47. Potential error reduction cost savings • How much cost per year? Assume: • 96,750 errors per year • 2 minutes recovery time per error Cost/year = (errors/yr) × (hrs/error corr) × (wage/hr) = 96,750 × (2 min/err) × (1hr ÷ 60mins) × ($15 / hr) = $48,375 per year for hourly employees (HE) = 96,750 × (2 ÷ 60 ) × ($40 / hr) = $129,000 per year for salaried employees (SE)

48. Benefit 3 – Potential design change cost savings • It is hard to estimate maintenance savings • However, it is claimed that early changes cost ¼ of later design changes Assume: • 1 day (i.e. 8 hours) per change • 25 changes Early cost = (hr / change) × (# of changes) × (wage / hr) = 8 × 25 × $40 = $8,000

49. Potential design change cost savings Late cost = (early cost) × 4 = $8000 × 4 = $32,000 Design change savings = (late cost) – (early cost) = $32,000 – $8,000 = $24,000

50. Benefit 4 – System adoption • If the development of the system is planned to meet the needs and wants of the user … • … then the probability of acceptance and use is higher • The cost benefit here is the entire cost of the systems development … • … because it is not “lost”