Enhancing Response Rates in Online Surveys: Strategies and Considerations

1. The following lecture has been approved for University Undergraduate Students This lecture may contain information, ideas, concepts and discursive anecdotes that may be thought provoking and challenging It is not intended for the content or delivery to cause offence Any issues raised in the lecture may require the viewer to engage in further thought, insight, reflection or critical evaluation

2. Questionnaires & Surveys Delivery Design Development Improving Response rates Prof. Craig Jackson Head of Psychology Division Education Law & Social Sciences BCU

3. Are Postal Questionnaires Dead Yet? IT predicted death of postal surveys Use of IT at home and work increases survey methods Comparison of surveys using WWW or EMAIL or POSTAL Subjects – UK university staff 200 email questionnaires 200 emails with www url 100 postal questionnaires Asked 3 questions: teeth cleaning fruit walking Jones, R. and Pitt, N. 1999

4. Are Postal Questionnaires Dead? Results Days after sending email www post n=200 n=200 n=100 numbers responding 3 25 31 0 5 59 34 16 9 61 35 53 10 63 35 60 17 68 37 72 17 day response rate 34% 19% 72% cost per reply Actual cost 35p 41p 92p With 100% response 19p 7.5p 72p

5. What this means for surveys Golden age of communication? Postal methods still much better Novelty value of email is dead - therefore lower response rates Junk mail perceptions Email Filters are improving Postal letters demonstrate & emphasize their importance www surveys & email allow immediate data processing (software) Email & www have potential for low cost regular user surveys Intranet users benefit from e-surveys

6. Introduction Questionnaire is a fundamental component of most research projects Most MSc / PhD projects use questionnaire methods Largest reason for criticising projects - weak / dubious questionnaires Can be very efficient A Questionnaire is not: Constructed in 10 minutes An easy option A collection of simple questions A Questionnaire does: Take planning to get right Reward time spent on it Capture information

7. Basics of Measurement Measurement tools must be appropriate Psychometrics -- personality, attitudes, stress, symptoms, Physical measurements -- working, environment, symptoms Exposure assessments -- hazards, risks, ppm3, duration When measuring... Take multiple measurements (and take the mean) Under same conditions, but if not…. Statistical remedies to adjust e.g. age, time of day etc. Reliable & Validated tool Defined and Regular variables Well-defined standard of reference

8. Dilemma Achieving a high response rate to a questionnaire is vital But does not promise a normal distribution of responses? Postal questionnaires rarely get a response rate > 40%. Unless respondents have a vested interest in the outcome. Bias? Most efficient (best) response rates usually happen when respondents have to do very little to take part in the study. Multiple phase projects see a depletion in numbers at every stage. Quick “in and out” one-stop approach is best http://onlinestatbook.com/stat_sim/sampling_dist/index.html

9. Know your enemy

10. Diminishing returns of multi-stage recruitment Researcher Potential Sample 1000 people 540 consents 540 questionnaires Under-powered studyn = 210 Response rate of 21% 210 questionnaires

11. Structure Identifying Items Title Preamble Branch Instructions Research Items does not need to be honest some deception is necessary shorter is better

12. Identifying items Preliminary questions Collecting info necessary for screening: recording keeping tracking tracing data manipulation Ask only for relevant info - unethical Fewer items minimizes chance of alienating respondent Participant’s need for privacy & anonymity The need for rich info to improve the study

13. Encryption devices Steganography Secret communication of a message by hiding it’s existence Steganos, meaning covered. Gk Graphein, meaning to write, Gk If message is discovered it is easily read because of no encryption Cryptography Secret communication of a message by hiding it’s meaning Kryptos, meaning hidden. Gk Message established using a known protocol, to be decrypted by the receiver Steganography & Cryptography can be combined together if needed Steganogrpahy arouses less suspicion in questionnaire respondents

14. Encryption devices Steganography “This example of steganogrpahy may not work very well when projected onto a large screen, but it works very well on paper, such as questionnaires. Hopefully many of you will not notice the method of steganography used in this piece of text. Gosh how clever I am……” Cryptography drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui The above text (containing an encrypted name and address) looks suspicious and may be obliterated by the respondent drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui

15. Preamble An important introduction Frame of reference Without a FOR a respondent may base their answer on a wrong context “We would like to know if you have had any medical complaints and how your health has been in general, over the past few weeks. Please answer ALL the questions on the following pages by ticking the answer which best applies to you. Remember that we want to know about present and recent complaints, not those that you had in the past. It is important that you try to answer all questions”

16. Title Not to be underestimated – try to be user-friendly Sets the tone for the respondent The General Health Questionnaire vs The GHQ 28 Beware of abbreviated titles - may alienate respondents e.g. Agricultural Satisfaction Scale LEI BDI MMPI HAD etc “Simple language for simple people”

17. Research items • Set(s) of items to collect the “real” research data • Take many forms: • Item TypeData type • Fixed option items (ordinal or nominal) • Rating scale items (likert) • Yes / No items (binary) • True / False items (binary) • Likert scale items (likert) • Diagrams binary, ordinal, or nominal

18. Branch Instructions Guide the respondent through the items Maximises the efficiency of respondent’s efforts Avoids redundant items Focuses on worthwhile items Instructions can sometimes confuse the respondent so think carefully e.g “If you answered ‘NO’ to question 9, ignore 9b, 9c, and 9d, go to question 10.”

19. Ah, the Truth

20. Respondent “Outrage” increases with item intimacy Are your respondents telling the truth? Use a sprinkling of LIE DETECTOR items to assess reliability e.g “Have you ever taken anything without asking permission?” “Did you ever lie to your parents as a child?” “Have you ever visited a pornographic web-site?” Decide what to do with any respondent who “fails” the lie detector Keep them in? Exclude immediately? Retain but “flag” them?

21. Funnel Items Set funnel questions to try to narrow down a respondent to specific details e.g 6a “Did you write essays in college?” Yes / No 6b “What type of essays were they mostly?” Narrative / Descriptive / Persuasive 6c “How often did you write them?” ___ per term 1 per term / 2-3 per term / 3 or more

22. Response Set Bias Enables collection of detailed data easily Build a detailed data set from a simple binary item Randomise the layout / order of responses “true / false” e.g True False True False True False True False True False Respondents ticking the same items Swap the “True & False” responses sequence

23. happy sad Likert scales A visual linear scale for rating purposes X Swap around the words to avoid response-set bias Avoid using numbers on the scale -- can be (mis)leading

24. X X X X X X X X X X X X X X X X X X X X X happy sad Likert scales

25. happy sad Likert scales X X X X X X X X X X X X

26. Design Pointers Avoid colloquialisms or abbreviations Beware of local expressions Avoid words with double meanings e.g.”Fair....Dip....Lie....Well” Set a definition of specific terms e.g. “OK.....Average” Avoid long questions Specify exact time, place and context e.g “At school, did you ever...” Phrase items to make denial impossible e.g “When did you first...” Avoid numbers on your responses / scales Have items seemingly related to the research topic

27. Design Pointers Clearly phrase items Make items unambiguous Avoid leading items e.g. “Many people think...” Ask only what a respondent is qualified to answer Avoid socially loaded items e.g. “beggars” or “junkies” Do not use socially / religiously biased items Avoid phrases, clichés or sooth-sayings Ask for ages as a true number / D.O.B - - NOT in age groups Font Pictures

28. Most common errors “In your organization, do women have the same responsibilities as men, and should women have more?” “Preventing accidents in the workplace is vital, and more money should be spent on prevention.” “Training in risk assessment is something I would like to do, and I would like to see my colleagues do it to.” Only ask one question per item

29. Are life preservers and flares essential on-deck equipment ? agree disagree Most common errors What is your age ? (please tick) 16 - 26 26 - 36 36 - 46 46 - 56 What is your marital status ? (please tick) Married Single How many blood splashes have you had ? (please tick) 1 - 5 6 - 10 11 - 15 15 - 20 20 plus

30. Factorial approach The GHQ 28 A self-completion questionnaire assessing mental health How? 28 items 7 about Anxiety Anxiety score 3 7 about Severe depression Depression score 4 7 about Dysfunction Dysfunction score 5 7 about Somatic symptoms Somatic score 2 Global score 14 By summing the 4 factors there is a Global Mental Health score Statistics are performed on the factor scores and the global score NOT on each individual item

31. Designing factors Piloting items Set down as many items as possible concerning the topic Limit the questionnaire to 30 items Give the questionnaire to at least 20 people Score responses and place into SPSS spreadsheet Factor Analysis use statistics to “group” items together into factors (will do this as a group exercise in the SPSS lecture)

32. Creating factors Factor Analysis Results in manageable number of factors instead of 30 items Each factor comprised of a number of items that had similar answers from your pilot sample The similarity of responses to particular items is what SPSS uses to group items together into factors Identifying Items As a matter of routine: Get as much information as possible may be vital in later (unforeseen analyses) Sex DOB or Age (not age groups) Marital status / Domestic arrangements

33. Increasing Response Rates Incentives Appearance Delivery Origin Contact Content Communication

34. Increasing Response Rates - Incentives Money Vouchers Prize draw Ethical aspects Bias O.R Monetary incentive vs. None 2.02 Incentive with Q. vs. Incentive on return 1.71 Non-monetary incentive vs. No incentive 1.19 Edwards et al. 2002

35. Increasing Response Rates - Appearance O.R Shorter format vs. Longer format 1.86 Brown envelope vs. White 1.52 Coloured ink vs. Black 1.39 Folder / Booklet vs. Stapled pages 1.17 Personalised vs. Not personalised 1.16 ID feature on return vs. No ID 1.08 Coloured Q vs. White Q 1.06 Edwards et al. 2002

36. Increasing Response Rates – Delivery methods O.R Recorded delivery vs. Standard 2.21 Stamped return envelope vs. Business reply / franked 1.26 Q sent to work vs. Q sent to home 1.16 1st class outward mail vs. Other class 1.12 Pre-paid return envelope vs. Not pre-paid 1.09 Stamped outward envelope vs. Franked 0.95 Commemorative stamp vs. Ordinary stamp 0.92 Edwards et al. 2002

37. Increasing Response Rates – Origin & Contact Origin O.R University vs. Other organisation 1.31 Sent by senior persons vs. Juniors 1.13 Ethnically ambiguous name vs. Non-white name 1.11 Contact O.R Pre-contact vs. No contact 1.54 Follow up vs. No follow up 1.44 Postal follow up with Q vs. Without Q 1.41 Mentioning follow up vs. None 1.04 Pre-contact by telephone vs. Postal pre-contact 0.90 Edwards et al. 2002

38. Increasing Response Rates - Content O.R More interesting vs. Less interesting 2.44 User-friendly vs. Standard 1.46 Factual items only vs. Factual & attitude items 1.34 Relevant items first vs. Other items 1.23 Demographic items first vs. Other items 1.04 “Don’t know” boxes vs. no “Don’t know” boxes 1.03 Sensitive items vs. No sensitive items 0.92 General items first vs. Last 0.80 Edwards et al. 2002

39. Increasing Response Rates - Communication O.R Explain drop-out required vs. Not 1.32 Stresses benefit to respondent vs. Others 1.06 Stresses benefit to sponsors vs. Others 1.01 Stresses benefit to society vs. Others 1.00 Response deadline given vs. No deadline 1.00 Instructions given vs. No instructions 0.89 Choice to opt out given vs. No opt out 0.76 Edwards et al. 2002

40. Questionnaire Summary • Postal questionnaires widely used in data collection • Mark pre-pay or addressed envelopes • Use steganography over cryptography • Perform analyses on factors not individual items • Identifying items may be useful in later analyses • Think about scoring items before rushing ahead with the questionnaire • Store questionnaires securely until passing any viva voce • Collect as much info at source, parse it down later at discretion

41. Questionnaire Summary cont. • Perfect for epidemiological studies and health research • Non-response to postal questionnaires reduces sample & introduces bias • Identification of effective ways to increase postal response rates • Use existing metrics - pilot items if making new questionnaire • Use factor analysis • Keep it all as brief as possible • Don’t alienate respondents • Alternate types of items