Effects of Design in Web Surveys Vera Toepoel Tilburg University The Netherlands
CentERdata: Two Online Panels • 1. CentERpanel:l • Exists for 17 years • 2000 households • Respondents fill out questionnaires every week Online interviews as method, but: • Probability sample drawn from address sampling frame of Statistics Netherlands • Recruitment of new panel members address-based • Includes households without internet access (less than 20%): Equipment
CentERdata: Two Online Panels • 2. LISS Panel • Grant from The Netherlands Organisation for Scientific Research • 5000 households • Established in 2007 (we fielded 1st questionnaire!) • Respondents fill out questionnaires every month Online interviews as method, but: • Probability sample drawn from address sampling frame of Statistics Netherlands • Contacted by letter, telephone or visit • Includes households without internet access (less than 20%): Equipment
Trained Respondents: Panel conditioning • Content (knowledge on topics) • Prepare for future surveys • Develop attitudes • Procedure (question-answering process) • Learn how to interpret questions • Answer strategically • Speed through the survey
Procedure (answer process) • Differences between trained and fresh respondents with regard to web survey design choices • Items per screen • Response category effects • Question layout
Overall: • Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.
Experiment 1: Items per screen • Social Desirability Scale • 10 items • 3 different formats: • 1 item per screen • 5 items per screen • 10 items per screen
Experiment 1: Items per screen • Trained respondents had higher inter-item correlations for multiple-item-per-screen formats. • No significant difference in item non-response. • Mean score of the Social Desirability Scale showed no evidence for social desirability bias. • The mean duration to complete the ten social desirability items did not differ significantly between panels.
Experiment 2: Answer Categories • Category effect found • No difference in category effect between trained and fresh respondents
Experiment 3: Question Layout • Question: Overall, how would you rate the quality of education in the Netherlands? • Answer: 5-point scale • Six formats: • Reference format (decremental) • Reverse scale: incremental • Horizontal layout • Add numbers 1 to 5 to verbal labels • Add numbers 5 to 1 to verbal labels • Add numbers 2 to -2 to verbal labels
Experiment 3: Question Layout • Decremental vs. incremental: T+ F • Vertical vs. horizontal layout: - • No numbers vs. numbers 1 to 5:- • Numbers 1 to 5 vs. numbers 5 to 1: T+F • Numbers 5 to 1 vs. Numbers 2 to -2: T+F • Trained respondents more easily selected one of the first options. • T=significant differences in Trained panel • F=significant differences in Fresh panel
Design Effects in Web Surveys: Comparing Trained and Fresh Respondents • Overall little differences between trained and fresh respondents • Trained respondents are somewhat more sensitive to satisficing: • Shorter completion times • Higher inter-item correlations for multiple-items-per-screen formats • Select first response options more often
Current and Future Research • It has been little more than a decade since systematic research was begun on visual design effects in web surveys. • In the last decade dozens of studies have been conducted • It is now important that we begin to understand the importance of each of the visual effects • Can we reduce visual effects by effective question writing?!
Effective Question Writing • Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to: • Verbal language>numbers> visual cues • Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?
Experiment 1: Visual Heuristics(joint with Don Dillman) • Tourangeau, Couper, and Conrad (POQ 2004; 2007): • Middle means typical: respondents will see the middle option as the most typical • Left and top means first: the leftmost or top option will be seen as the ‘first’ in conceptual sense • Near means related: options that are physically near each other are expected to be related conceptually • Up means good: the top option will be seen as the most desirable • Like means close: visually similar options will be seen as closer conceptually • Experimental conditions: • Polar point or fully labeled scale • With or without numbers (1 to 5)
Middle Means Typical Fully labeled: even spacing Fully labeled: uneven spacing
Left and Top Means First Fully labeled with color: consistent ordering Fully labeled with color: inconsistent ordering
Near Means Related Polar point with numbers: separate screens Polar point with numbers: single screen
Up Means Good Polar point with numbers: incremental Polar point with numbers: decremental
Like Means Close Polar point Polar point with color
Like Means Close Polar point with numbers (1 to 5) Polar point with different numbers (-2 to 2)
Experiment 2: Pictures in web surveys (joint with Mick Couper) • Replicate study Couper, Tourangeau, and Kenyon (POQ 2004) • 1. No Picture • 2. Low frequency picture • 3. High frequency picture • Add verbal instructions • A. No verbal instruction • B. Instruction to include both high and low frequency instances • C. Instruction to include only low frequency instances
Can verbal instructions reduce the effects of pictures? • MANOVA • main effect instructions lambda=.597, p<.0001 • main effect pictures lambda=.964, p<.0001 • interaction instructions*pictures lambda=.9691,p<.0001 • This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!
Future Research • How to reduce visual design effects in web surveys
LISS data • Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost • Proposals can be submitted through www.lissdata.nl • Existing data free available for academic use • longitudinal core studies • proposed studies • disseminated through www.lissdata.nl