Self-Administered Surveys: Mail Survey Methods

1. Self-Administered Surveys: Mail Survey Methods ChihChien Chen Lauren Teffeau Week 10

2. Dillman, 1991 Why MAIL surveys? • Lower costs • Perception of simple procedures • Large scale general public • Time • People with hearing problem

3. Dillman Cont’d • Sampling Error • Noncoverage Error – incomplete sampling frame • Nonresponse Error – people don’t answer • Measurement Error – instrument deficiencies Sources of Survey Error:

4. Reducing Nonresponse Error Why is increasing response rate important in mail survey design? • Hard to make the comparison between respondents and nonrespondents. • What and How do they differ?

5. Dillman Cont’d • Response Rate is indicator Reducing Nonresponse Error… How to improve Response Rates? • Subjective assessments: follow-up mailings, prior notice, incentives, return postage, sponsorship, personalization, interest in survey topic • Quantitative assessments via meta-analysis: number of contacts (initial and follow-up); salience of topic; government sponsorship; employee, school or army population; incentive on first contact – Negative effects: market research, sponsorship, general population & length

6. Understanding Mail Survey Response Behavior: A Meta-Analysis (Yammarino, Skinner & Childers, 1991) • To enhance understanding of mail survey response behavior by presenting a more comprehensive meta-analysis investigation of factors that affect mail survey response rates.

7. Yammarino et. al. Cont’d Feature 1: incorporate several aspects of a properly conducted meta-analysis Feature 2: combined meta-analysis procedures Features 3: more factors through a larger number of study outcomes Features 4: investigation of two moderator variables

8. Yammarino et. al. Cont’d Key results • a cover letter that includes appeals and a survey of less than four pages • works for all target population • a reply/return envelop and monetary incentives • are situation specific and need to consider other moderators • a preliminary notification • subject to time period. In the study: ’76-87 • a stamped/metered return postage • varied with time. In the study: ’40-70 • differed for samples of groups • a follow-ups/repeated contacts • differed from samples of groups) • Examine more moderators in subgroup • characteristics of respondents, additional details about samples, way of survey delivered and situation (time & location) in which surveys are administered/completed

9. Dillman Cont’d • Recommended procedures that should be used holistically versus a la carte • Based on Social Exchange theory • Results in high response rates Total Design Method – What is it? • What are some limitations? • Lack of empirical evidence • Ignores the effect of financial incentives • Over-reliance on social exchange? • No tailorization for different populations

10. Dillman Cont’d Only a perspective on how to stimulate responses, not definitive… Updating Total Design Method: Need for updating; hence, ‘Tailored’ Design Method, 2000 Tailored b/c computers make it easier to customize surveys to specific populations Attempts to create respondent trust, perceptions of increased rewards, and reduced social costs to respondents Increased emphasis on visual design

11. Dillman Cont’d • In mail surveys, measurement error usually stems from: • Item non-response • Poor open-ended responses • Low education of respondents • Complex skip patterns • Question order How to reduce Measurement Error? • Importance of pre-testing • Fostering response conditions: cover letter, follow-ups, incentives etc. • Know your population • Limit skips and craft clear questions (wording and visual placement on survey)

12. Dillman Cont’d How do Mail surveys impact respondents’ answers? • Less chance of interviewer subversion • Less social desirability pressure What special considerations must researchers make for Mail surveys? • Slower and potentially more costly data collection • Cannot forget self-administered environment: • - Clarity of questions and instructions paramount to lessen measurement error • - Issue of who is answering survey (may not be intended • respondent; salience of topic may influence returns)

13. Dillman Cont’d What are some advantages of Mail over Telephone surveys? • Self-administered lessens order effects • Mail respondents less extreme answers • Mail respondents less social desirability • Pacing of mail surveys – more cognition time vs off-the-top-of-the-head answers • What about Context effects? • Respondents don’t have to answer surveys linearly (Schwarz & Hippler – norm of evenhandedness) • If in doubt, use split-ballot to control for potential effects. • Use priming to your advantage if possible

14. Dillman Cont’d How to reduce Noncoverage Error? Have accurate sampling frame. What if you don’t? Find a different way to do your research OR create your sampling frame = much more cost and more lead time. • Important especially with mail because of reliance on household addresses, which are usually out-of-date • Area of further research and development of practices

15. Maximizing Response Rates • You get a national grant to administer a mail survey. What steps would you follow to get the highest response rate possible if expense was not an issue? • You find out that you will only get a fraction of the funding you need to administer a survey for your dissertation. What steps would you follow to get the highest response rate relative to your budget?

16. Web Surveys are Self-Administered Too! What is similar between web and mail surveying techniques? What what do you need to account for if doing a web instead of mail survey? What are the advantages of mixed methods?