Effects of Design in Web Surveys

2. Effects of Design in Web Surveys Vera Toepoel Tilburg University The Netherlands

3. CentERdata: Two Online Panels • 1. CentERpanel:l • Exists for 17 years • 2000 households • Respondents fill out questionnaires every week Online interviews as method, but: • Probability sample drawn from address sampling frame of Statistics Netherlands • Recruitment of new panel members address-based • Includes households without internet access (less than 20%): Equipment

4. CentERdata: Two Online Panels • 2. LISS Panel • Grant from The Netherlands Organisation for Scientific Research • 5000 households • Established in 2007 (we fielded 1st questionnaire!) • Respondents fill out questionnaires every month Online interviews as method, but: • Probability sample drawn from address sampling frame of Statistics Netherlands • Contacted by letter, telephone or visit • Includes households without internet access (less than 20%): Equipment

5. 1 item per screen

6. 4 items per screen

7. 10 items per screen

9. Answer categories

10. Open-ended

11. Vertical: positive to negative

12. Horizontal

13. Numbers 1 to 5

14. Numbers 5 to 1

15. Numbers 2 to -2

16. Trained Respondents: Panel conditioning • Content (knowledge on topics) • Prepare for future surveys • Develop attitudes • Procedure (question-answering process) • Learn how to interpret questions • Answer strategically • Speed through the survey

17. Procedure (answer process) • Differences between trained and fresh respondents with regard to web survey design choices • Items per screen • Response category effects • Question layout

18. Overall: • Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.

19. Experiment 1: Items per screen • Social Desirability Scale • 10 items • 3 different formats: • 1 item per screen • 5 items per screen • 10 items per screen

20. Experiment 1: Items per screen • Trained respondents had higher inter-item correlations for multiple-item-per-screen formats. • No significant difference in item non-response. • Mean score of the Social Desirability Scale showed no evidence for social desirability bias. • The mean duration to complete the ten social desirability items did not differ significantly between panels.

21. Experiment 2: Answer Categories

22. Experiment 2: Answer Categories

23. Experiment 2: Answer Categories • Category effect found • No difference in category effect between trained and fresh respondents

24. Experiment 3: Question Layout • Question: Overall, how would you rate the quality of education in the Netherlands? • Answer: 5-point scale • Six formats: • Reference format (decremental) • Reverse scale: incremental • Horizontal layout • Add numbers 1 to 5 to verbal labels • Add numbers 5 to 1 to verbal labels • Add numbers 2 to -2 to verbal labels

25. Experiment 3: Question Layout • Decremental vs. incremental: T+ F • Vertical vs. horizontal layout: - • No numbers vs. numbers 1 to 5:- • Numbers 1 to 5 vs. numbers 5 to 1: T+F • Numbers 5 to 1 vs. Numbers 2 to -2: T+F • Trained respondents more easily selected one of the first options. • T=significant differences in Trained panel • F=significant differences in Fresh panel

26. Design Effects in Web Surveys: Comparing Trained and Fresh Respondents • Overall little differences between trained and fresh respondents • Trained respondents are somewhat more sensitive to satisficing: • Shorter completion times • Higher inter-item correlations for multiple-items-per-screen formats • Select first response options more often

27. Current and Future Research • It has been little more than a decade since systematic research was begun on visual design effects in web surveys. • In the last decade dozens of studies have been conducted • It is now important that we begin to understand the importance of each of the visual effects • Can we reduce visual effects by effective question writing?!

28. Effective Question Writing • Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to: • Verbal language>numbers> visual cues • Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?

29. Experiment 1: Visual Heuristics(joint with Don Dillman) • Tourangeau, Couper, and Conrad (POQ 2004; 2007): • Middle means typical: respondents will see the middle option as the most typical • Left and top means first: the leftmost or top option will be seen as the ‘first’ in conceptual sense • Near means related: options that are physically near each other are expected to be related conceptually • Up means good: the top option will be seen as the most desirable • Like means close: visually similar options will be seen as closer conceptually • Experimental conditions: • Polar point or fully labeled scale • With or without numbers (1 to 5)

30. Middle Means Typical Fully labeled: even spacing Fully labeled: uneven spacing

31. Left and Top Means First Fully labeled with color: consistent ordering Fully labeled with color: inconsistent ordering

32. Near Means Related Polar point with numbers: separate screens Polar point with numbers: single screen

33. Up Means Good Polar point with numbers: incremental Polar point with numbers: decremental

34. Like Means Close Polar point Polar point with color

35. Like Means Close Polar point with numbers (1 to 5) Polar point with different numbers (-2 to 2)

36. Labels, numbers and visual heuristics: is there a hierarchy?

37. Experiment 2: Pictures in web surveys (joint with Mick Couper) • Replicate study Couper, Tourangeau, and Kenyon (POQ 2004) • 1. No Picture • 2. Low frequency picture • 3. High frequency picture • Add verbal instructions • A. No verbal instruction • B. Instruction to include both high and low frequency instances • C. Instruction to include only low frequency instances

38. Low and High frequency picture

39. Can verbal instructions reduce the effects of pictures? • MANOVA • main effect instructions lambda=.597, p<.0001  • main effect pictures lambda=.964, p<.0001 • interaction instructions*pictures lambda=.9691,p<.0001 • This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!

40. Future Research • How to reduce visual design effects in web surveys

41. LISS data • Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost • Proposals can be submitted through www.lissdata.nl • Existing data free available for academic use • longitudinal core studies • proposed studies • disseminated through www.lissdata.nl