slide1
Download
Skip this Video
Download Presentation
Effects of Design in Web Surveys

Loading in 2 Seconds...

play fullscreen
1 / 42

Effects of Design - PowerPoint PPT Presentation


  • 300 Views
  • Uploaded on

Effects of Design in Web Surveys Vera Toepoel Tilburg University The Netherlands CentERdata: Two Online Panels 1. CentERpanel: l Exists for 17 years 2000 households Respondents fill out questionnaires every week Online interviews as method, but:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Effects of Design ' - emily


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
effects of design in web surveys

Effects of Design in Web Surveys

Vera Toepoel

Tilburg University

The Netherlands

centerdata two online panels
CentERdata: Two Online Panels
  • 1. CentERpanel:l
      • Exists for 17 years
      • 2000 households
      • Respondents fill out questionnaires every week

Online interviews as method, but:

    • Probability sample drawn from address sampling frame of Statistics Netherlands
    • Recruitment of new panel members address-based
    • Includes households without internet access (less than 20%): Equipment
centerdata two online panels4
CentERdata: Two Online Panels
  • 2. LISS Panel
  • Grant from The Netherlands Organisation for Scientific Research
      • 5000 households
      • Established in 2007 (we fielded 1st questionnaire!)
      • Respondents fill out questionnaires every month

Online interviews as method, but:

    • Probability sample drawn from address sampling frame of Statistics Netherlands
    • Contacted by letter, telephone or visit
    • Includes households without internet access (less than 20%): Equipment
trained respondents panel conditioning
Trained Respondents: Panel conditioning
  • Content (knowledge on topics)
    • Prepare for future surveys
    • Develop attitudes
  • Procedure (question-answering process)
    • Learn how to interpret questions
    • Answer strategically
    • Speed through the survey
procedure answer process
Procedure (answer process)
  • Differences between trained and fresh respondents with regard to web survey design choices
    • Items per screen
    • Response category effects
    • Question layout
overall
Overall:
  • Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel.
experiment 1 items per screen
Experiment 1: Items per screen
  • Social Desirability Scale
  • 10 items
  • 3 different formats:
    • 1 item per screen
    • 5 items per screen
    • 10 items per screen
experiment 1 items per screen20
Experiment 1: Items per screen
  • Trained respondents had higher inter-item correlations for multiple-item-per-screen formats.
  • No significant difference in item non-response.
  • Mean score of the Social Desirability Scale showed no evidence for social desirability bias.
  • The mean duration to complete the ten social desirability items did not differ significantly between panels.
experiment 2 answer categories23
Experiment 2: Answer Categories
  • Category effect found
  • No difference in category effect between trained and fresh respondents
experiment 3 question layout
Experiment 3: Question Layout
  • Question: Overall, how would you rate the quality of education in the Netherlands?
  • Answer: 5-point scale
  • Six formats:
  • Reference format (decremental)
  • Reverse scale: incremental
  • Horizontal layout
  • Add numbers 1 to 5 to verbal labels
  • Add numbers 5 to 1 to verbal labels
  • Add numbers 2 to -2 to verbal labels
experiment 3 question layout25
Experiment 3: Question Layout
  • Decremental vs. incremental: T+ F
  • Vertical vs. horizontal layout: -
  • No numbers vs. numbers 1 to 5:-
  • Numbers 1 to 5 vs. numbers 5 to 1: T+F
  • Numbers 5 to 1 vs. Numbers 2 to -2: T+F
  • Trained respondents more easily selected one of the first options.
  • T=significant differences in Trained panel
  • F=significant differences in Fresh panel
design effects in web surveys comparing trained and fresh respondents
Design Effects in Web Surveys: Comparing Trained and Fresh Respondents
  • Overall little differences between trained and fresh respondents
  • Trained respondents are somewhat more sensitive to satisficing:
    • Shorter completion times
    • Higher inter-item correlations for multiple-items-per-screen formats
    • Select first response options more often
current and future research
Current and Future Research
  • It has been little more than a decade since systematic research was begun on visual design effects in web surveys.
  • In the last decade dozens of studies have been conducted
  • It is now important that we begin to understand the importance of each of the visual effects
  • Can we reduce visual effects by effective question writing?!
effective question writing
Effective Question Writing
  • Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to:
  • Verbal language>numbers> visual cues
  • Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers?
experiment 1 visual heuristics joint with don dillman
Experiment 1: Visual Heuristics(joint with Don Dillman)
  • Tourangeau, Couper, and Conrad (POQ 2004; 2007):
  • Middle means typical: respondents will see the middle option as the most typical
  • Left and top means first: the leftmost or top option will be seen as the ‘first’ in conceptual sense
  • Near means related: options that are physically near each other are expected to be related conceptually
  • Up means good: the top option will be seen as the most desirable
  • Like means close: visually similar options will be seen as closer conceptually
  • Experimental conditions:
  • Polar point or fully labeled scale
  • With or without numbers (1 to 5)
middle means typical
Middle Means Typical

Fully labeled: even spacing

Fully labeled: uneven spacing

left and top means first
Left and Top Means First

Fully labeled with color: consistent ordering

Fully labeled with color: inconsistent ordering

near means related
Near Means Related

Polar point with numbers: separate screens

Polar point with numbers: single screen

up means good
Up Means Good

Polar point with numbers: incremental

Polar point with numbers: decremental

like means close
Like Means Close

Polar point

Polar point with color

like means close35
Like Means Close

Polar point with numbers (1 to 5)

Polar point with different numbers (-2 to 2)

experiment 2 pictures in web surveys joint with mick couper
Experiment 2: Pictures in web surveys (joint with Mick Couper)
  • Replicate study Couper, Tourangeau, and Kenyon (POQ 2004)
    • 1. No Picture
    • 2. Low frequency picture
    • 3. High frequency picture
  • Add verbal instructions
    • A. No verbal instruction
    • B. Instruction to include both high and low frequency instances
    • C. Instruction to include only low frequency instances
can verbal instructions reduce the effects of pictures
Can verbal instructions reduce the effects of pictures?
  • MANOVA
    • main effect instructions

lambda=.597, p<.0001 

    • main effect pictures

lambda=.964, p<.0001

    • interaction instructions*pictures lambda=.9691,p<.0001
    • This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures!
future research
Future Research
  • How to reduce visual design effects in web surveys
liss data
LISS data
  • Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost
    • Proposals can be submitted through www.lissdata.nl
  • Existing data free available for academic use
    • longitudinal core studies
    • proposed studies
    • disseminated through www.lissdata.nl
ad