Requirements Gathering & Task Analysis – Part 2 of 5

1. Requirements Gathering & Task Analysis – Part 2 of 5 Why, What and How – Methods This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Diane Gromala, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: Jan 2014.

2. How – Some (Not All) User & Task Analysis Methods Observation Interviews Ethnography Surveys/Questionnaires Focus groups Study documentation Study competitive products Data recording - automated Data recording – diary (log) study UI Design - Georgia Tech

3. Formative & Summative Evaluation • Formative evaluation • Conducting this process to help guide the formation (ie, design) of a UI • Summative Evaluation • Conducting this process to help summarize (sum up) the effectiveness of an existing or developmental UI • Many of the user & task analysis techniques can be used for both formative and summative evaluation • Our focus right now is on formative evaluation • Will revisit some of the methods again later UI Design - Georgia Tech

4. Observation - From a Distance • Watch users do what they do • Step-by-step procedures • Pauses to figure out what to do • Mistakes/errors • Do this today or tomorrow • Automatic Checkout, ATM, parking meter • Don’t scare off the observees!! • Record (with permission) UI Design - Georgia Tech

5. Observation – up Close • Often combined with “Thinking Out Loud” • Encourage user to verbalize what they are thinking • Put them at ease first! • Not every user is good at this • Hard for user to do for long time; need breaks • Record (with permission) • Up close and at a distance not mutually exclusive UI Design - Georgia Tech

6. Pros and Cons? • Pros • Cons UI Design - Georgia Tech

7. Interviews • Structured – “Just the facts” • Fixed set of questions • How do you know what questions to ask? • May miss subtleties and big picture • Unstructured – A conversation • Go with the flow • May learn what questions to ask • Semi-structured – start with focused questions, move to open-ended discussion • Good balance, often appropriate • Training – process + domain knowledge • Do some homework first UI Design - Georgia Tech

8. Interviews • Know why you are asking questions - don’t waste time • Plan for effective question types • How do you perform task x? • Why do you perform task x? • Under what conditions do you perform task x? • What do you do before you perform…? • What information do you need to…? • Whom do you need to communicate with …? • What do you use to…? • What happens after you…? • What is the result or consequence of…? • What is the result or consequence of NOT…? UI Design - Georgia Tech

9. Interviews: Typical Open-Ended Questions • Why do you do this (whatever the task is you are studying) • How do you do this? • Gets at task-subtask structure • Then ask about each subtask • Why do it this way rather than some other way? • Attempts to get user to explain method so you can assess importance of the particular way of doing task • What has to be done before you can do this? • To understand sequencing requirements UI Design - Georgia Tech

10. Interviews: More Typical Open-Ended Questions • Please show me the results of doing this • Do errors ever occur when doing this? • If answer is ‘yes,’ then learn why occur • How do you discover the errors, and how do you correct them? • Encourage digressions; ask for elaborations • Pause before asking next question • What else should I have asked you? UI Design - Georgia Tech

11. Interviews: Domain Experts • Expert may describe how it should be done • Not necessarily how it is done :-) UI Design - Georgia Tech

12. Interviews: Users vs. Management • Folks who are not in the trenches doing the work may have a different view than those who are actually doing whatever it is you are studying. • In organizations, can be political UI Design - Georgia Tech

13. Ethnography • Similar to Observation, but more intense • Observe people in their cultural context • Because behavior most meaningful in context • Immerse oneself in situation you want to learn about (has anthropological and sociological roots); use • In person observation • Interviews – formal and over coffee • Audio/video recording – after trust develops • Learn to do the work yourself! (biggest difference from observation) UI Design - Georgia Tech

14. Ethnography: Types of Findings • Can be both • Qualitative • Observe trends, habits, patterns, … • Quantitative • How often was something done, what per cent of the time did something occur, how many different … UI Design - Georgia Tech

15. Ethnography: Drawbacks • Time required • Can take weeks or months for large systems • Scale • Most use small numbers of participants just to keep somewhat manageable • Type of results • Highly qualitative, may be difficult to present/use • Acquired skill – “learn by doing” • Identifying and extracting “interesting” things is challenging UI Design - Georgia Tech

16. Questionnaires • Make questions clear and specific • Ask some closed questions with range of answers • Sometimes also have a no opinion option, or other answer option • Do test run with a few people • Use computer-based tools UI Design - Georgia Tech

17. Questionnaires – Likert Scale • Seven-point Likert Scale (use odd #) • Slightly leading question • Could be seen as favoring one answer • Help was worthwhile • Agent intrusive • To avoid issue…. UI Design - Georgia Tech

18. Questionnaires – Semantic Differential Annoying X Pleasing Easy to use Difficult to use X Attractive X Unattractive X Secure Not secure Helpful Unhelpful Modern Dated UI Design - Georgia Tech

19. Questionnaires – Typical Questions • Rank the importance of each of these tasks (give a list of tasks) • List the four most important tasks that you perform (this is an open question) • List the pieces of information you need to have before making a decision about X, in order of importance • Are there any other points you would like to make? (open-ended opinion question; good way to end) • Same questions can be used in interview and in questionnaire; difference is in follow-up opportunity UI Design - Georgia Tech

20. Questionnaire Pros/Cons Pros Cons UI Design - Georgia Tech

21. Focus Groups • Group of individuals - 3 to 10 • Use several different groups with different roles or perspectives • Careful about power relationships – separate groups • Careful about few people dominating discussion • Use structured set of questions • More specific at beginning, more open as progresses • Allow digressions before coming back on track • Relatively low cost, quick way to learn a lot • Audio or video record, with permission • Blog – open-ended focus group • Pros/cons? UI Design - Georgia Tech

22. Study Documentation • Similar in some ways to the expert interview • Often describe how things should be done rather than how they are done • Try to understand why not done “by the book” UI Design - Georgia Tech

23. Study Competitive Products • Looking for both good and bad ideas • Functionality • UI style • Try to understand why are successful or unsuccessful • Measure task completion times for new and experienced users • Upper bounds for new product UI Design - Georgia Tech

24. Data Recording – Time-stamped Event Log • High-level, such as • Web page visits • Commands used • Low-level, such as • Keystroke and mouse events • Device/person location; think Aware Home • Lots of data - need analysis tools • Pros/Cons? UI Design - Georgia Tech

25. Data Recording - Manual (Log Study) • Participants keep a log for days or weeks • Asked to record relevant events • May do follow-up interview • Lot of detailed text to analyze after the fact • Pros/cons UI Design - Georgia Tech

26. Methods to Use Depends on… • Resources • Current knowledge of tasks and users • Context • Can’t use talking out loud if tasks involve two people working together • Essential to use some methods • Not likely you will use all methods • Which best for class project? • Understand pros/cons of each • On own and relative to other methods UI Design - Georgia Tech

27. End of Part 2 UI Design - Georgia Tech

28. UI Design - Georgia Tech

29. Ethnography: Observations • Carefully observe everything about users and their environment • Structure and language used in work • Individual and group actions • Culture affecting work • Explicit and implicit aspects of work; what is said, what is left unsaid • Example: Office work environment • Business practices, rooms, artifacts, work standards, relationships between workers, managers, … UI Design - Georgia Tech

30. Why is Ethnography Useful? • Helps gain rich assessment of user needs • Help to define requirements • Uncovers true nature of user’s job • Discovers things that are outside of job description or documentation • Allows you to play role of end-user better • Can sit in when real users not available • Open-ended and unbiased nature promotes discovery UI Design - Georgia Tech