Effectiveness of Monetary Incentives and Other Stimuli Across Establishment Survey Populations

1. Effectiveness of Monetary Incentives and Other Stimuli Across Establishment Survey Populations ICES III 2007 Montreal, Quebec Canada 4.30.07 Danna Moore and Mike Ollinger

2. Objective (1) Why is this study important? (2) Gaps in Establishment Survey Literature (3) Provide suggestions for implementing establishment surveys

3. Establishment Respondents: What type of respondent are they? Large business or Org Respondent Multiple locations Gate keepers Record system Household Respondent e.g. small farmer, small business owner Can vary along the continuum

4. Why is this important? • Questionnaire variation • Converting • Mandatory reporting • R’s Vary over time • Risk/Difficulty -- cash incentives • Population based sample sizes • Experimental design • Cost/Benefit • Generalize

5. Stimuli Tested • Cash Incentives • Cash plus special postage/package • Multiple modes • Mode sequencing • Mode preferences • Visual Design— color background vs. none • Answer boxes stand out

6. Some Answers Towards the Big Question Do incentives help or hinder in obtaining survey responses from businesses? • Crosses types of establishments and industries. • Experiment based • Population based • Random assignment

7. Understanding Why People Participate Several theories (Dillman, 1978; Gouldner, 1960; Biner and Kidd, 1994; Groves et al., 1999). • Social exchange Leverage Saliency Theory—Groves et al. (1999)  Decision to participate is a series of interactive additive factors.  Some are survey specific and others are person specific. Incentives are viewed as an inducement used to compensate for absence of some factors (i.e., saliency or sense of duty).

8. Leverage Saliency Theory Of Survey Participation 1999 POQ vol 64

9. One person selected to represent establishment. Burden increases as they answer as a representative. There is a respondent questionnaire interaction. 4.Respondent’s characteristics in relation to the establishment influences their ability to respond. 5. intermediary between the questionnaire and the characteristics of the record system. 6. Organizational environment 7. Extenuating survey situation Features of Establishment Surveys That Often Lead to Survey Errors

10. Factors Influencing Response • Businesses differ across industries by size, structure, and organizational environment. • Each survey may have a situation or circumstance that impedes contact. • These differences and circumstances often influence how well a survey request can penetrate an establishment

11. Gaps • Lack of monetary incentive studies: • Few experimental treatments for definitive comparisons and outcomes. • Few comparisons of cash versus “cash like” (e.g. checks, ATM cards) studies—could be especially important for government. • No empirical demonstrations of effectiveness of incentives across types of industries and firm size. • Few comparing effects of various size incentives. • Not much on effectiveness of survey modes and mode sequencing • Response attributable to survey mode sequencing.

12. 2001 HMO Survey Physicians N=1474 *** **

13. 2003 USDA Nationwide Meat Manufacturers N=1,705 Response rates achieved by experimental treatment group *** ***

14. 2006 Snake River Grain Warehouse and Shipper Survey Response by Experimental Group (n=424 elevators) N.S. *** N.S. Chi SQ 6.7 P < .01

15. 2007 Evaluation of WA Plastic Pesticide Container Use/Recycling Control versus Treatment, N=1,986 Chi Sq. 15.86 ***P<.001 Chi Sq. 33.5 ***P< .0001

16. 2006 Oregon Business Environmental Management Survey Type and Experimental Treatment Group Initial sample n=1964 ** *** *** *** *** ** **

17. 2006 Trade Adjustment Assistance Survey, All Qualifying Industries N=6,429 Exp. Incentive Treatment vs. Control ---Completion Rates *** Sig. Chi Sq. 51.07 P<.001 ***

18. Trade Adjustment Assistance (TAA) 2006 Survey of Program Nationwide – Fisheries N=5,592 *** *** ** *** **

19. Trade Adjustment Assistance (TAA) 2006 Survey of Program Nationwide – CommoditiesN=837 NS NS NS

20. TAA Survey—Percentage of Completes Associated with Incentive Experimental Assignment and Survey Mode Telephone Completes Web Completes Mail Completes

21. Suggestions for implementing establishment surveys Effective Practice: • Contact respondents multiple times • Contact respondents in multiple modes. • Allow for survey mode preference. • Design surveys that reduce burden • short, conditional branching, ease • Use leverages

22. Visual Design Effect – FARW Commercial Dealers N=1600Background Shading w/ Visible Answer BoxesDoes it make a difference?Is there an interaction effect w/ Incentives? *** ***

23. Visual Design TestFARW Growers With Pesticide License *** *** ** ***

24. All 3 survey modes generated completes • Exper. treatment (Cash incentive & priority mail) stimulated more responses in all 3 survey modes—large interaction effect. • Offering web as an alternative option garnered 15% • Telephone last 3% – still effective

25. Influential Circumstances Saliency: Topic interest • area of business emphasis for entity. • High personal interest for respondent. • High level of public or political concern Role of survey sponsor. • Regulator or source of certification. • Mandatory reporting. • Source of program $$ or sponsorship Response Burden • Complexity, length, multiple reports

26. What was learned from experimental trials • Token cash incentives were effective across types of establishment and industry populations. • 2-day Priority mail was more effective than first class mail. • Priority mail alone just slightly better than First class • Cash incentives combined with priority mail was synergistic. • Mixed mode strategies are very helpful and work. • Respondents may have mode preferences. • Establishment population characteristics and the selected respondent characteristics need to be considered jointly in explaining response. • The survey circumstances and situation impact establishment response.