1 / 63

Collecting Data from Users

Collecting Data from Users. Uses. User and task analysis Prototype testing On-going evaluation and re-design. Data collection methods. Questionnaires Interviews Focus groups. Definitions. Survey:

harsha
Download Presentation

Collecting Data from Users

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Collecting Data from Users

  2. Uses • User and task analysis • Prototype testing • On-going evaluation and re-design

  3. Data collection methods • Questionnaires • Interviews • Focus groups

  4. Definitions • Survey: • (n): A gathering of a sample of data or opinions considered to be representative of a whole. • (v): To conduct a statistical survey on. • Questionnaire: (n) A form containing a set of questions, especially one addressed to a statistically significant number of subjects as a way of gathering information for a survey. • Interview • (n): A conversation, such as one conducted by a reporter, in which facts or statements are elicited from another. • (v) To obtain an interview from. • American Heritage Dictionary

  5. Surveying • Sample selection • Questionnaire construction • Data collection • Data analysis

  6. Surveys – detailed steps • determine purpose, information needed • identify target audience(s) • Select method of administration • design sampling method • design prelim questionnaire • including analysis • Often based on unstructured or semi-structured interviews with people like your respondents • pretest • Revise, pretest… • administer • analyze

  7. Why survey as method? • Answers from many people, including those at a distance • Relatively easy to administer, analyze • Can continue for a long time

  8. Surveys can collect data on: • Facts • Characteristics of respondents • Self-reported behavior • This instance • Generally/usually • Opinions and attitudes: • Preferences, opinions, satisfaction, concerns

  9. Some Limits of Surveys • Reaching users is easier than non-users • Relies on voluntary cooperation, possibly biasing the sample • Questions have to be unambiguous, amenable to short answers • You only get answers to the questions you ask; you generally don’t get explanations • The longer or more complex the survey the less cooperation

  10. Some sources of error • Sample • Question choice • Can respondents answer? • Question wording • Method of administration • Inferences from the data • Users’ interests in influencing results • “vote and view the results” CNN quick vote: http://www.cnn.com/

  11. When to do interviews? • Need details that can’t get from survey • Need more open-ended discussions with users • Small #s OK • Can identify and gain cooperation from target group • Sometimes: want to influence respondents as well as get info from them

  12. Sample selection

  13. Targeting respondents • About whom do you want information? • About whom can you get information? • E.g. non-users are hard to reach

  14. Sampling terminology • Element: the unit about which info is collected; basis of analysis. E.g., “User” • Universe: hypothetical aggregation of all elements. “All users” • Population: a specified aggregation of survey elements. “People who have used this service at least once in the last year.” • Survey population: aggregate of elements from which the sample is selected.“People who use the service at least once during the survey period.”

  15. Terminology, cont. • Sampling unit: elements considered for selection. • Sampling frame: list of sampling units. • Observation unit: unit about which data is collected. Often the same as unit of analysis but not always. E.g. one person (observational unit) may be asked about the household (unit of analysis). • Variable: a set of mutually exclusive characteristics such as sex, age, employment status. • Parameter: summary description of a given variable in a population. • Statistics: summary description of a given variable in a sample.

  16. Sample design • Probability samples • random • stratified random • cluster • Systematic • Size: if 10/90% split, 100; if 50/50, 400; • Yale’s slide • 30-50 in each cell • GOAL: Representative sample • Non-probability sampling • convenience sampling • purposive sampling • quota sampling

  17. Representative samples • Which characteristics matter? • Want the sample to be roughly proportional to the population in terms of groups that matter • E.g., students by gender and grad/undergrad status: • http://opa.vcbf.berkeley.edu/IC/Campus.Stats/CampStats_F00/CS.F00.Table.F2.htmgraduate/undergraduate…a

  18. Active vs passive sampling • active: solicit respondents • Send out email • Phone • Otherwise reach out to them • Follow up on non-respondents if possible • passive – e.g. on web site • Response rate may be unmeasurable • heavy users may be over-represented • Disgruntled and/or happy users over-represented

  19. Response Rates • low rates may > bias • Whom did you miss? Why? • How much is enough? • Babbie: 50% is adequate; 70% is very good • May help if they understand purpose • Don’t underestimate altruism • Incentives may increase response • Reporting back to respondents as a way of getting response

  20. Example of a careful sample design • http://www.pewinternet.org/reports/reports.asp?Report=55&Section=ReportLevel1&Field=Level1ID&ID=248

  21. RECAP • We collect data from users/potential users for: • User and task analysis • Prototype testing • On-going evaluation and re-design • Methods include: • Surveys, interviews, focus groups • Different methods useful for different purposes

  22. Some Issues Common to Different Methods • Know your purpose! • Match method to purpose and feasibility • Whom do you need/can you get to participate? • Population, sample composition, sample selection methods, size • Response rate, respondent characteristics (bias)

  23. Common Issues, cont. • What do you need to know? • What can/will respondents tell you? • How will it help? • How do you ask what you want to know? • Question construction, wording • Question ordering • What do you do with results? • Reporting and analysis

  24. Types of web surveys • Comprehensive • Quick polls – focused, one or few questions • http://www.gomez.com/ratings/index.cfm?topcat_id=19&firm_id=1768&CFID=295681&CFTOKEN=8136111 • Short, focused surveys • Guestbooks, user registration, user feedback • http://www.bookfinder.com/interact/comments/

  25. Uses of surveys of web sites • Identify users • Describe their characteristics • Describe their behavior • Ask their needs, preferences • Assess user satisfaction/response • Identify user problems, dissatisfactions • Solicit ideas for improvement

  26. Problems with Web Surveys • Population? • Size • Characteristics • Response rate? • Multiple responses from same person OK? • Biased sample? • Users competent to answer? • E.g., will they answer after they have used the site enough to be able to judge? • Non-users not represented; infrequent users under-represented?

  27. Questionnaire construction

  28. Questionnaire construction • Content • Goals of study: What do you need to know? • What can people tell you? • Conceptualization • Operationalization – e.g., how do you define “user”? • Question design • Question ordering

  29. Topics addressed by surveys • Respondent (user) characteristics • Respondent behavior • Respondent opinions, perceptions, preferences, evaluations

  30. Respondent characteristics • Demographics • General http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • Professional role (manager, student..) • User role (e.g., buyer, browser…) • Expertise – hard to ask • Domain • technology • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • System

  31. Behavior • Tasks (e.g., for what are they using this site?) • Site usage, activity • Frequency; common functions – hard to answer accurately • Self-reports vs observations • Web and internet use: Pew study

  32. Opinions, preferences, concerns • content • Organization, architecture • interface • perceived needs • concerns • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/privacy.html • Success, satisfaction • Subdivided by part of site, task, purpose… • other requirements • Suggestions

  33. Problems with questions • Social acceptability • Privacy • “Do you use the internet to view pornography?” • Difficulty wording unambiguously • International concerns • Language • Social acceptability

  34. Problems: topics hard to conceptualize, operationalize • e.g., “Why did you use the CDL today?” • Teaching, research… • my category of work or task • to find an electronic journal, locate a book, find a citation… • To locate kind of resource • to find material on x topic • the subject area • to save me time • i.e. I used this rather than another way of accessing same resource

  35. Question construction

  36. Question formulation • match respondents’ language • match respondents’ behavior • what do they want to tell you? • What can they tell you? • Recent CNN.com poll: “do you think the sentence given to x was too long, about right, too short.” What was the sentence? (What was the crime?) • Beware of compound questions, hidden assumptions: • ‘did you order something? If so, did you….’ – what if only browsing?

  37. Question formulation: simplicity and clarity Complete the following sentence in the way that comes closest to your own views: 'Since getting on the Internet, I have ...' • ... become MORE connected with people like me. • ... become LESS connected with people like me. • ... become EQUALLY connected with people like me. • ... Don't know/No answer. http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html

  38. Survey questions – format • open-ended http://www.bookfinder.com/interact/comments/ • “What is your job title?” _______ • “What do you find most difficult about your job?” • closed-ended (one answer; multiple answers) • http://www.useit.com/papers/surveys.html “Select the range that best represents the total number of staff…” 1-2 3-5 6-10… • paired characteristics/semantic differential Friendly__|___|___|___|____| Unfriendly

  39. Question format, cont. • Ordinal scale/ “Likert scale” Strongly agree, agree, neutral, disagree, strongly disagree http://www.amazon.com/exec/obidos/tg/stores/detail/-/books/073571102X/rate-this-item/104-0765616-4703139 • Rating scale – matrix (usability survey, qn 5) • Always include neutral, “other,” “N/A” (not applicable)

  40. Closed-ended questions • Answers need to be comprehensive and mutually exclusive; • OR • Allow people to give more than one answer • Tell them how to choose “Why did you come to SIMS?” • How many responses to allow? • As many as your respondent needs • Likert scale-like questions: • 5 OR 7 is usual; can respondent differentiate 7? • Very strongly agree; strongly agree; agree; neutral; disagree; strongly disagree; very strongly disagree? • Odd vs even: even allows neutral, odd forces a choices.

  41. Filter and Contingency questions • 1. “Have you ever used our competitor?” • “If no, go to question 3.” • 2. “If yes, how would you rate….” • Did you apply to any graduate program other than SIMS? • If yes… • If no…

  42. Layout: consistency Circle the answer that best matches your opinion. SIMS is a great place to study. Strongly agree Agree Neutral Disagree Strongly disagree UC Berkeley is a great university. Strongly agree Agree Neutral Disagree Strongly disagree IS214 is a great course. Strongly disagree Disagree Neutral Agree Strongly agree

  43. Include Instructions! • One answer to each question, or multiples? • If online, can program to test and let user know they have violated guidelines. • Respond every time you visit this site? • Be sure it’s clear what to do if a question does not apply. • “Why did you choose SIMS over any other programs that accepted you?”

  44. Question ordering • Group similar questions • Using headings to label parts of survey, topics • Funneling: • General to particular • Particular to general • Keep it short! Response falls off with length.

  45. Pre-test, pre-test, pre-test! • With people like expected respondents • Looking for: • Ambiguous wording • Missing responses • Mismatch between your expectations and their reality • Any other difficulties respondents will have answering, or you will have interpreting their responses

  46. Web Survey Problems • Who is the population? Self-selected sample. • Stuffing the ballot box • Cnn.com polls • How to know what response rate is • How to get responses (1) after they have used site (2) before they leave • What are you assessing and what are they responding to? • E.g., design of site, or content? Presentation of content, or content of content? • Loss of context – what exactly are you asking about, what are they responding to? • Are you reaching them at the appropriate point in their interaction with site?

  47. Web Survey Problems II • Incomplete responses -- avoiding blank responses – and annoying users • Respondents may not know how long the survey is • Let people know • Multiple submissions

  48. Survey administration

  49. Human subjects considerations • not hurt by the process itself (e.g., uncomfortable questions) • not hurt by uses of the data • job etc not in jeopardy • not at risk for criminal prosecution • confidentiality or anonymity • Consent, voluntary participation • http://cphs.berkeley.edu

  50. Ways of administering surveys • On a web site • Open access • Limited access • Email (with URL) • Phone • Paper • Mail • Users pick up on their own • Hand out in person • Fax • Face to face

More Related