1 / 37

Surveys

Surveys. HF302. Outline. Take and evaluate a survey Pros and cons of the survey as a research tool When to use a survey. To Get In the Mood. Complete as much of the technology survey as you can in 10 minutes.

emmet
Download Presentation

Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Surveys HF302

  2. Outline • Take and evaluate a survey • Pros and cons of the survey as a research tool • When to use a survey

  3. To Get In the Mood • Complete as much of the technology survey as you can in 10 minutes. • Spend a couple minutes writing notes about how to improve the survey (on the survey, to the side, please). • As you go • Or after you finish

  4. What’s wrong? • What can be improved in this survey? • What survey design problems have you encountered previously?

  5. Review of p.33, Stanton et al.Pros and Cons

  6. The Pros • Questionnaires offer a very flexible way of collecting large volumes of data from large participant samples • When the questionnaire is properly designed, the data analysis phase should be quick and very straightforward • Very few resources are required once the questionnaire has been designed • A number of HF questionnaires already exist… • Very easy to administer to large number of participants • Skilled questionnaire designers can use the questions to direct the data collection.

  7. The Cons • Designing, piloting, and analysing a questionnaire is time consuming. • Reliability and validity of questionnaires are low. • The design questionnaire process is taxing , requiring great skill… • Response rates are typically low, around 10% for postal questionnaires. • Causes sampling bias and distorted results. • The answers given are often rushed and non-committal. • Questionnaires are prone to biases, such as prestige bias. • Questionnaires can offer a limited output.

  8. Some More Disadvantages • Format discourages explanation or elaboration • “Ready to Fight” • Responses to open-ended questions can be ambiguous • Difficult to publish survey-based research

  9. Validity and Reliability

  10. Validity and Reliability

  11. What are two examples of strategies or frameworks that have been proposed as a means to help researchers and designers be better ‘systems thinkers’? 1. 2.

  12. Choose one of the strategies or frameworks you just listed. (A) How effective or useful has it been as a means for improving systems thinking? (B) Explain your answer to ‘A’. Chosen Strategy/Framework: ______________________________________ A. Very effective / Somewhat effective / Ineffective for the most part (Circle one.) B. Explain:

  13. When collecting data for research, why should you minimize the collection and use of opinions?

  14. Does someone’s opinion count as empirical data?

  15. Whether right or wrong… Experimental psychology is all about showing us that our assumptions are incorrect. • We cannot make ourselves stop thinking about white bears…or cigarettes. • You may think you were there and you may recall it in great detail; however… • We think we’re attracted to someone because they’re fascinating when really it’s just the shaky bridge we’re standing on. • We are happy not because we’ve made a good choice (e.g., to get married) but because we have fewer choices.

  16. What are some examples of other ways to obtain this information?

  17. And if you’re a philosophy buff… There’s the centuries-old concern about Ground Truth and Veridicality (veridical) coinciding with reality; "perceptual error...has a surprising resemblance to veridical perception” - F.A.Olafson http://wordnetweb.princeton.edu/perl/webwn?s=veridical

  18. And a centuries-old debate… • About what really matters in science? • Leading to factions: • Materialism, Idealism, Rationalism, Empiricism, Monism, Dualism, Modernism, Positivism, Pragmatism, Radical Empiricism, Phenomenolgy

  19. Example: Materialism, Positivism “In principle, any type of benefit is adequate to conduct a cost-benefit analysis, provided it is quantifiable.” - Sagar & Grier, 2005, p. 5

  20. When to Use a Survey • As a starting point • When sample size is very large • When participants are not accessible to you for observations, interviews experiments, etc. • When resources are limited • To collect background data about your participants

  21. Body Part Discomfort Survey Please indicate your level of job related pain by placing an X in one of the five boxes for each body region. If you know what causes the pain or discomfort please note this on the comments line. Comments If You Know Why Eyes Neck Shoulders Upper back Upper Arm Middle Back Elbow Lower back Lower Arm Wrists Hands Buttocks Thighs Knees Lower Legs Feet (None) (Just (Some Pain) (Moderate (Intolerable No Pain Noticeable) Restricts Some Pain) Pain) Pain Does Activity Restricts Restricts Not Restrict Most Activity All Activity Activity

  22. When to Not Use a Survey

  23. Neville’s Rules of Thumb • Always offer a “Comments” Line. Make it long enough for someone to write a comment. • Always pilot test your survey at least once with at least 10 representative participants. • Ethics: Always provide or offer your participants a debrief/explanation and a way to contact you. • Don’t report means and standard deviations of ratings. Report frequencies and quartiles instead.

  24. Box-and-Whisker Plot& Five-Number Summary lowest value highest value median 1st quartile 3rd quartile

  25. What is one strategy for improving the validity of a survey question or design?

  26. Back Up Slides

  27. Goods and Others?

  28. (None) (Moderate (Intolerable No Pain Pain) Pain) Body Part Discomfort Survey Please indicate your level of job related pain by placing an Xanywhere along the provided lines for each body region using the scale given below as a guide (No Pain, Moderate Pain, Intolerable Pain). If you know what causes the pain or discomfort (and if the pain/discomfort is on your right, left, or both sides of your body) please note this on the comments line. Comments If You Know Why Eyes Neck Shoulders Upper Back Upper Arm Middle Back Elbow Lower Back Lower Arm Wrists Hands Buttocks/Hips Thighs Knees Lower Legs Ankles Feet Example x Body Height Feet Body Weight (None) (Moderate (Intolerable No Pain Pain) Pain)

  29. (32 Questions) Assessing the Impact of Automation on Mental Workload - Long Version (AIM-L)

  30. Survey Design • Prepare • Construct • Evaluate and Improve • Address Challenges

  31. Survey Design • Prepare • Goals • Specific issues of interest • Target Population • How to reach target population? • In person, via internet, via mail, other?

  32. Survey Design • Construct • Introduction • Participant information • Information section • Debrief and contact information

  33. Survey Design • Evaluate and Improve • Response rate • Potential for misunderstanding • Potential for hasty inaccurate responses

  34. Survey Design • Address Challenges • Response rate • Objectivity • Try to disprove any expectations • Potential for misunderstanding • Potential for hasty inaccurate responses

More Related