Paul Krueger, Ph.D. Associate Professor Dept. of Clinical Epidemiology & Biostatisitcs

1. School of Nursing’s Research Workshop and Seminar SeriesSurvey Methods, Questionnaire Design & Implementation Paul Krueger, Ph.D. Associate Professor Dept. of Clinical Epidemiology & Biostatisitcs kruegerp@mcmaster.ca January 19, 2009

2. Surveys • Many study designs use surveys to collect information (exposures, confounders, outcomes, opinions, preferences, KABP) • Systematic method of collecting data • Structured and standardized questionnaire • Sample of the population

3. Advantages of Surveys • Most methods allow collection a lot of information, from many people, over a wide geographic area, in a relatively short time period • Everyone is asked the same questions in the same way (minimize interviewer bias) • If sampled appropriately, results can be representative of a larger population (within a certain degree of error)

4. Types of Surveys • Mail (e.g. survey of all ER and Family MDs about CAP) • focus of this presentation • Telephone (e.g. air pollution - random sample) • Becoming more difficult (e.g. cell phones, answering machines, call display, call blocking, unlisted, do not call lists) • Fax (e.g. physician survey) • Face-to-face (e.g. LTCF resident QOL/satisfaction) • Internet (e-mail and web based) • Combined methods (e.g. CAPIS, OARS…)

5. Dillman Approach toMail Surveys • Total Design Method (1978) • What aspects of the survey process affect quality or quantity of response • A detailed and systematic way to design and implement surveys • Social exchange theory (see below) • High quality mail surveys • Acceptable response rates

6. Dillman Approach Cont’d • The Tailored Design Method (2000) • Many changes since 1978 • Technology – computers, scanners… • More complete understanding of social exchange principles in survey research • Research on survey methods • Need to tailor the method vs same procedure • Note: Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method 3rd edition 2009

7. Social Exchange Theory • Theory of human behaviour • “The theory asserts that actions of individuals are motivated by the return these actions are expected to bring, and in fact usually do bring from others” (Dillman, 2000) • People will do things that are rewarding

8. Social Exchange Theory • Three elements for predicting action • Rewards • What one expects to gain from an activity • Costs • What one gives up/spends to obtain rewards • Trust • Expectation that in the long run, rewards of doing something will outweigh the costs of doing it

9. Application of Social Exchange Theory • In terms of questionnaire design & implementation: • Increase rewards for responding • Reduce perceived costs • Establish trust so the rewards outweigh costs of responding

10. Ways of Providing Rewards • Show positive regard • Show respect (e.g. give reason for survey, provide number to call, personally address letters) • Say thank you • e.g. “we appreciate your help”, “thanks in advance” • Ask for help, advice, assistance • Provides a sense of reward • Support group values • appeal to widely shared values – the study’s social usefulness

11. Ways of Providing Rewards • Give tangible rewards • Include a financial incentive (token) vs promise • Make the questionnaire interesting • Improve layout and design • Give social validation • Let them know that others have responded • Opportunities to respond are scarce • Respond quickly so information can be used

12. Ways of Reducing Social Costs • Avoid subordinating language • Ask for a favour e.g. to solve a problem vs it is necessary for you to complete this survey because…. • Avoid embarrassment • E.g. avoid complex questions and instructions at the start of the questionniare • Avoid inconvenience • E.g. include return postage paid envelope – real stamp

13. Ways of Reducing Social Costs • Make questionnaire appear short and easy • Lesson the perceived cost of responding • Minimize requests to obtain personal information • Offer explanations why info is important & ensure confidentiality

14. Ways of Establishing Trust • Provide a token of appreciation in advance • Small amount – gesture of trust – vs payment for your time • Including actual stamps on return envelope • Sponsorship by legitimate authority (e.g. university or gov’t sponsored vs marketing research) • Make the task appear important • Personalized cover letters on letterhead paper • Professional looking questionnaire • Something useful will be done with results

15. Dillman’s Five Needed Elements for High Response Rates • Respondent friendly questionnaire • Up to 5 (varied) contacts • Inclusion of stamped return envelopes • Personalized correspondence • Token financial incentive

16. Respondent Friendly Questionnaire • Easy to understand questions • Well ordered • Easy to follow layout • Overall appearance

17. Writing Questions • Develop questions that every potential respondent will: • Interpret the same way (and the same way as the investigators) • Be able to respond accurately • Be willing to answer

18. Dillman’s examples of things to do when writing questions • Use simple words • Do not be vague • Keep it short • Avoid being too specific • Do not talk down to respondents • Avoid bias • Avoid objectionable questions • Etc. • However, many of the above can be contradictory!

19. Things to do…. • Use equal numbers of pos and neg categories • Distinguish between undecided and neutral • Create mutually exclusive response categories • Avoid double-barrelled questions • Provide appropriate time referents • Etc. …… • See Dillman, Choi (and others) for many other things that should and should not be done when writing questions

20. Questionnaire Construction • Respondent friendly questionnaire design • Will improve response rates (and response rates among those least likely to respond) • Reduces measurement error (e.g. prevents items or answer categories from being missed) • Issues to consider: • Questionnaire format • Question order • Appearance of individual questions • Front and back covers

21. Questionnaire Format • Booklet format is recommended • Handled more or less automatically without error • People are familiar with them • See Dillman for design issues and rationale

22. Ordering Questions • A questionnaire is like a conversation • Choose the first question carefully • Should apply to everyone • Should be easy • Should be interesting • Should be connected to the study purpose • Needs to be a logical flow • Group questions with similar content • Sections with “transition statements” • More salient to least salient • Objectionable questions near end…….

23. Appearance of Questions • Respondents should see and understand every word of every question • Font, font size, bold, italics can ease the task of understanding and responding • Place instructions where needed • If combining questions with same answer categories – do it carefully • Number questions consecutively • More space between questions than subcomponents • List answer categories vertically (one column) • Etc……..

24. Front Cover • Opportunity to motivate and increase response rates • Title (short and simple) • Neutral graphic • Name and address of study sponsor • Extension of cover letter (well known and legitimate source)

25. St. Joseph’s Lifecare Centre 2007 FAMILY MEMBER SATISFACTION WITH CARE SURVEY The purpose of this survey is to collect information that will be used to help improve care and services provided for your loved one at the St. Joseph’s Lifecare Centre. Your participation is voluntary and all information will remain confidential. Answers will be pooled together so that no individual response can be identified. With this in mind, we are interested in your honest opinion, whether it be negative or positive. If you wish to comment on any question, please feel free to use the space in the margins. Your comments will be read and taken into account. Thank you in advance for your participation. Working together to enhance the quality of life by providing love, care, and dignity St. Joseph’s Lifecare Centre 99 Wayne Gretzky Parkway, Suite 105 Brantford, Ontario N3S 6T6 Attention: St. Joseph’s Health System Research Network

26. Back Cover • Keep it simple • Invitation to make additional comments (chance for people to vent) • Thank you • Lots of “white space” • Avoid questions on back cover • Last few questions tend to be the most objectionable (e.g. income)

27. Pretesting • Review by knowledgeable colleagues • Content validity (included all necessary questions; exclude unneeded questions; appropriate response categories….) • Diverse expertise (content + methodology) • Potential survey respondents • All words understood • All questions interpreted the same • All inclusive list of response categories • Mailout package create a positive attitude etc….

28. Pilot Testing • Revise questionnaire and then pilot test all proposed procedures • Examine response rates, item nonresponse, variable distributions • Make revisions based on findings • Then, have someone “fresh” that can review the questionnaire one last time Questionnaire design is much more complex and takes much longer than most people expect

29. Mail SurveyImplementation • Questionnaire design is only 1 part of a well designed survey • It is not the main determinant of high response rates • Survey implementation procedures collectively have a greater influence on response rates

30. Dillman Recommends5 Contacts Multiple, varied contacts are essential • Prenotice (advance notification) letter • Questionnaire mailout (package) • Thank you postcard (thank you / reminder letter) • Replacement questionnaire (package) • Final (special) contact

31. Prenotice Letter(First Contact) • Notice to expect a request to participate in an important study (all letters - low education level) • Brief, personalized, positively worded • Should build anticipation • Courtesy (people like to be told ahead of time) • Mention incentive • Mailed (first class) a few days to week before questionnaire

32. Prenotice Letter • Letterhead paper (helps to legitimize survey) • Actual Date / Respondent name and address / Dear… • What will happen (you should receive a questionnaire) • What it is about (brief description + sponsor) • Usefulness of survey (how it will help) • Thank you • Real signature + title and affiliations • P.S. Incentive notice (if one is to be offered)

34. Questionnaire Mailout(Second Contact) • Sent a few days to 1 week after prenotice • Includes: • Cover letter • Questionnaire (with an ID#) • Token of appreciation ($, TH coupons, pen etc.) • One of the best ways to increase response • Should accompany questionnaire vs afterwards • Return postage paid envelope (with stamps)

35. Cover Letter • 1 page on letterhead paper • Exact mailing date / Name & address / Dear… • Specifically written to the person vs a form letter • What this letter is about (your request) • Why were you selected (and how) • Usefulness of survey (how this will help) • Confidentiality • ID# on questionnaire (don’t hide it) • Voluntary (if refuse ask for reason)

36. Cover Letter Cont’d • Enclosures (stamped return envelope + token) • Who to contact with questions (accessibility) • Thank you for helping with this important study • Real signature (contrasting ink – pressed) • Postscript (thanks again - token)

38. Assembling and Mailing • Attention to detail • Questionnaire ID# matches the name on letter and mailing label • Everything is included • Dillman “stuffing procedure” to ensure that none of the contents are left in the envelope • Larger outgoing envelope (avoid folding) • 1st class mail • Select mailout time (avoid major holidays) and day

39. Thank you / Reminder(Third Contact) • Mail 1 week after questionnaire • Thank you / reminder post card (or letter) • Objective is to jog memories • Very short • Thanks those who completed and returned • Reminds others to “please do so today” • Statement of how important each recipient is to the success of the study • If you didn’t receive it, call and one will be sent • Signature

41. First Replacement Questionnaire(Fourth Contact) • Send 2 weeks after thank you / reminder • Only sent to non responders (need to track) • Cover letter - more persuasive/insisting tone • Questionnaire not yet received • Others have responded/importance to study (representative) • Not eligible? (return questionnaire) • Confidential / voluntary (still return it unanswered) • Note of appreciation • Personally signed / P.S. (any questions) • Replacement questionnaire (with ID#) • Return postage paid envelope - but no token

43. Special Contact(Fifth Contact) • 4 weeks later to non respondents [PK 2 wks] • Impact is due to a different “special” method which shows respondent & study are important • Telephone call (encourage/send new one/complete) • Registered mail (requires signature) • Special delivery • Courier • More expensive follow-up • “New” cover letter, questionnaire (ID#), postage paid envelope

44. Survey Error (Dillman) • Sampling error • Precision is limited by the number surveyed • Coverage error • Occurs when list does not include everyone • Measurement error • Results from poor wording and construction • Nonresponse error • When significant numbers don’t respond

45. Example Readings for Other Ideas • Increasing Response Rates to Postal Questionnaires (Edwards et al. 2002) • Systematic review of RCTs • 292 RCTs reviewed • 75 strategies for influencing response • A Catalog of Biases in Questionnaires (Choi and Pak, 2005) • Identified 48 common types of bias • Categorized into 3 main sources of bias • Question design • Questionnaire design • Questionnaire administration

46. Problems/Issues • Undelivered questionnaires • People move/incomplete address or errors • Attempt to correct problems and resend • New mail out dates – additional work • Handling respondent questions • Should attempt to answer all questions • Should anticipate questions (e.g. ID#, no proxies…) • Thank them for asking • Evaluating early returns • Identify problems (e.g. printing, item response, certain questions causing problems….)

47. Final Note AboutMail Survey Response Rates • Mail surveys often criticized for low response • Example Response Rates (PK using “Dillman approach”) • Surveys of physicians • Physician Survey (elder abuse) 65% • Physician Survey (pneumonia) 77% • LTC Medical Director Survey 61% • Surveys of other health professionals • LTC Director of Care Survey 76% • Canadian Optometrist Survey 90% • Other “general population” health surveys • Contact Lens Use Survey 93% • LTC Family Member Satisfaction Surveys (80-86%)