School of Nursing
1 / 47

Paul Krueger, Ph.D. Associate Professor Dept. of Clinical Epidemiology & Biostatisitcs - PowerPoint PPT Presentation

  • Uploaded on

School of Nursing ’s Research Workshop and Seminar Series Survey Methods, Questionnaire Design & Implementation. Paul Krueger, Ph.D. Associate Professor Dept. of Clinical Epidemiology & Biostatisitcs January 19, 2009. Surveys.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Paul Krueger, Ph.D. Associate Professor Dept. of Clinical Epidemiology & Biostatisitcs' - edan-velazquez

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Paul krueger ph d associate professor dept of clinical epidemiology biostatisitcs

School of Nursing’s Research Workshop and Seminar SeriesSurvey Methods, Questionnaire Design & Implementation

Paul Krueger, Ph.D.

Associate Professor

Dept. of Clinical Epidemiology & Biostatisitcs

January 19, 2009


  • Many study designs use surveys to collect information (exposures, confounders, outcomes, opinions, preferences, KABP)

  • Systematic method of collecting data

  • Structured and standardized questionnaire

  • Sample of the population

Advantages of surveys
Advantages of Surveys

  • Most methods allow collection a lot of information, from many people, over a wide geographic area, in a relatively short time period

  • Everyone is asked the same questions in the same way (minimize interviewer bias)

  • If sampled appropriately, results can be representative of a larger population (within a certain degree of error)

Types of surveys
Types of Surveys

  • Mail (e.g. survey of all ER and Family MDs about CAP)

    • focus of this presentation

  • Telephone (e.g. air pollution - random sample)

    • Becoming more difficult (e.g. cell phones, answering machines, call display, call blocking, unlisted, do not call lists)

  • Fax (e.g. physician survey)

  • Face-to-face (e.g. LTCF resident QOL/satisfaction)

  • Internet (e-mail and web based)

  • Combined methods (e.g. CAPIS, OARS…)

Dillman approach to mail surveys
Dillman Approach toMail Surveys

  • Total Design Method (1978)

    • What aspects of the survey process affect quality or quantity of response

    • A detailed and systematic way to design and implement surveys

    • Social exchange theory (see below)

    • High quality mail surveys

    • Acceptable response rates

Dillman approach cont d
Dillman Approach Cont’d

  • The Tailored Design Method (2000)

    • Many changes since 1978

      • Technology – computers, scanners…

      • More complete understanding of social exchange principles in survey research

      • Research on survey methods

      • Need to tailor the method vs same procedure

      • Note: Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method 3rd edition 2009

Social exchange theory
Social Exchange Theory

  • Theory of human behaviour

    • “The theory asserts that actions of individuals are motivated by the return these actions are expected to bring, and in fact usually do bring from others” (Dillman, 2000)

    • People will do things that are rewarding

Social exchange theory1
Social Exchange Theory

  • Three elements for predicting action

    • Rewards

      • What one expects to gain from an activity

    • Costs

      • What one gives up/spends to obtain rewards

    • Trust

      • Expectation that in the long run, rewards of doing something will outweigh the costs of doing it

Application of social exchange theory
Application of Social Exchange Theory

  • In terms of questionnaire design & implementation:

    • Increase rewards for responding

    • Reduce perceived costs

    • Establish trust so the rewards outweigh costs of responding

Ways of providing rewards
Ways of Providing Rewards

  • Show positive regard

    • Show respect (e.g. give reason for survey, provide number to call, personally address letters)

  • Say thank you

    • e.g. “we appreciate your help”, “thanks in advance”

  • Ask for help, advice, assistance

    • Provides a sense of reward

  • Support group values

    • appeal to widely shared values – the study’s social usefulness

Ways of providing rewards1
Ways of Providing Rewards

  • Give tangible rewards

    • Include a financial incentive (token) vs promise

  • Make the questionnaire interesting

    • Improve layout and design

  • Give social validation

    • Let them know that others have responded

  • Opportunities to respond are scarce

    • Respond quickly so information can be used

Ways of reducing social costs
Ways of Reducing Social Costs

  • Avoid subordinating language

    • Ask for a favour e.g. to solve a problem vs it is necessary for you to complete this survey because….

  • Avoid embarrassment

    • E.g. avoid complex questions and instructions at the start of the questionniare

  • Avoid inconvenience

    • E.g. include return postage paid envelope – real stamp

Ways of reducing social costs1
Ways of Reducing Social Costs

  • Make questionnaire appear short and easy

    • Lesson the perceived cost of responding

  • Minimize requests to obtain personal information

    • Offer explanations why info is important & ensure confidentiality

Ways of establishing trust
Ways of Establishing Trust

  • Provide a token of appreciation in advance

    • Small amount – gesture of trust – vs payment for your time

    • Including actual stamps on return envelope

  • Sponsorship by legitimate authority (e.g. university or gov’t sponsored vs marketing research)

  • Make the task appear important

    • Personalized cover letters on letterhead paper

    • Professional looking questionnaire

    • Something useful will be done with results

Dillman s five needed elements for high response rates
Dillman’s Five Needed Elements for High Response Rates

  • Respondent friendly questionnaire

  • Up to 5 (varied) contacts

  • Inclusion of stamped return envelopes

  • Personalized correspondence

  • Token financial incentive

Respondent friendly questionnaire
Respondent Friendly Questionnaire

  • Easy to understand questions

  • Well ordered

  • Easy to follow layout

  • Overall appearance

Writing questions
Writing Questions

  • Develop questions that every potential respondent will:

    • Interpret the same way (and the same way as the investigators)

    • Be able to respond accurately

    • Be willing to answer

Dillman s examples of things to do when writing questions
Dillman’s examples of things to do when writing questions

  • Use simple words

  • Do not be vague

  • Keep it short

  • Avoid being too specific

  • Do not talk down to respondents

  • Avoid bias

  • Avoid objectionable questions

  • Etc.

    • However, many of the above can be contradictory!

Things to do
Things to do….

  • Use equal numbers of pos and neg categories

  • Distinguish between undecided and neutral

  • Create mutually exclusive response categories

  • Avoid double-barrelled questions

  • Provide appropriate time referents

  • Etc. ……

    • See Dillman, Choi (and others) for many other things that should and should not be done when writing questions

Questionnaire construction
Questionnaire Construction

  • Respondent friendly questionnaire design

    • Will improve response rates (and response rates among those least likely to respond)

    • Reduces measurement error (e.g. prevents items or answer categories from being missed)

  • Issues to consider:

    • Questionnaire format

    • Question order

    • Appearance of individual questions

    • Front and back covers

Questionnaire format
Questionnaire Format

  • Booklet format is recommended

    • Handled more or less automatically without error

    • People are familiar with them

    • See Dillman for design issues and rationale

Ordering questions
Ordering Questions

  • A questionnaire is like a conversation

  • Choose the first question carefully

    • Should apply to everyone

    • Should be easy

    • Should be interesting

    • Should be connected to the study purpose

  • Needs to be a logical flow

  • Group questions with similar content

    • Sections with “transition statements”

  • More salient to least salient

  • Objectionable questions near end…….

Appearance of questions
Appearance of Questions

  • Respondents should see and understand every word of every question

  • Font, font size, bold, italics can ease the task of understanding and responding

  • Place instructions where needed

  • If combining questions with same answer categories – do it carefully

  • Number questions consecutively

  • More space between questions than subcomponents

  • List answer categories vertically (one column)

  • Etc……..

Front cover
Front Cover

  • Opportunity to motivate and increase response rates

  • Title (short and simple)

  • Neutral graphic

  • Name and address of study sponsor

  • Extension of cover letter (well known and legitimate source)

Paul krueger ph d associate professor dept of clinical epidemiology biostatisitcs

St. Joseph’s Lifecare Centre


The purpose of this survey is to collect information that will be used to help improve care and services provided for your loved one at the St. Joseph’s Lifecare Centre. Your participation is voluntary and all information will remain confidential. Answers will be pooled together so that no individual response can be identified. With this in mind, we are interested in your honest opinion, whether it be negative or positive. If you wish to comment on any question, please feel free to use the space in the margins. Your comments will be read and taken into account.

Thank you in advance for your participation.

Working together to enhance the quality of life by providing love, care, and dignity

St. Joseph’s Lifecare Centre

99 Wayne Gretzky Parkway, Suite 105

Brantford, Ontario

N3S 6T6

Attention: St. Joseph’s Health System Research Network

Back cover
Back Cover

  • Keep it simple

  • Invitation to make additional comments (chance for people to vent)

  • Thank you

  • Lots of “white space”

  • Avoid questions on back cover

    • Last few questions tend to be the most objectionable (e.g. income)


  • Review by knowledgeable colleagues

    • Content validity (included all necessary questions; exclude unneeded questions; appropriate response categories….)

    • Diverse expertise (content + methodology)

  • Potential survey respondents

    • All words understood

    • All questions interpreted the same

    • All inclusive list of response categories

    • Mailout package create a positive attitude etc….

Pilot testing
Pilot Testing

  • Revise questionnaire and then pilot test all proposed procedures

  • Examine response rates, item nonresponse, variable distributions

  • Make revisions based on findings

  • Then, have someone “fresh” that can review the questionnaire one last time

    Questionnaire design is much more complex and takes much longer than most people expect

Mail survey implementation
Mail SurveyImplementation

  • Questionnaire design is only 1 part of a well designed survey

  • It is not the main determinant of high response rates

  • Survey implementation procedures collectively have a greater influence on response rates

Dillman recommends 5 contacts
Dillman Recommends5 Contacts

Multiple, varied contacts are essential

  • Prenotice (advance notification) letter

  • Questionnaire mailout (package)

  • Thank you postcard (thank you / reminder letter)

  • Replacement questionnaire (package)

  • Final (special) contact

Prenotice letter first contact
Prenotice Letter(First Contact)

  • Notice to expect a request to participate in an important study (all letters - low education level)

  • Brief, personalized, positively worded

  • Should build anticipation

  • Courtesy (people like to be told ahead of time)

  • Mention incentive

  • Mailed (first class) a few days to week before questionnaire

Prenotice letter
Prenotice Letter

  • Letterhead paper (helps to legitimize survey)

  • Actual Date / Respondent name and address / Dear…

  • What will happen (you should receive a questionnaire)

  • What it is about (brief description + sponsor)

  • Usefulness of survey (how it will help)

  • Thank you

  • Real signature + title and affiliations

  • P.S. Incentive notice (if one is to be offered)

Questionnaire mailout second contact
Questionnaire Mailout(Second Contact)

  • Sent a few days to 1 week after prenotice

  • Includes:

    • Cover letter

    • Questionnaire (with an ID#)

    • Token of appreciation ($, TH coupons, pen etc.)

      • One of the best ways to increase response

      • Should accompany questionnaire vs afterwards

    • Return postage paid envelope (with stamps)

Cover letter
Cover Letter

  • 1 page on letterhead paper

  • Exact mailing date / Name & address / Dear…

    • Specifically written to the person vs a form letter

  • What this letter is about (your request)

  • Why were you selected (and how)

  • Usefulness of survey (how this will help)

  • Confidentiality

  • ID# on questionnaire (don’t hide it)

  • Voluntary (if refuse ask for reason)

Cover letter cont d
Cover Letter Cont’d

  • Enclosures (stamped return envelope + token)

  • Who to contact with questions (accessibility)

  • Thank you for helping with this important study

  • Real signature (contrasting ink – pressed)

  • Postscript (thanks again - token)

Assembling and mailing
Assembling and Mailing

  • Attention to detail

  • Questionnaire ID# matches the name on letter and mailing label

  • Everything is included

  • Dillman “stuffing procedure” to ensure that none of the contents are left in the envelope

  • Larger outgoing envelope (avoid folding)

  • 1st class mail

  • Select mailout time (avoid major holidays) and day

Thank you reminder third contact
Thank you / Reminder(Third Contact)

  • Mail 1 week after questionnaire

  • Thank you / reminder post card (or letter)

  • Objective is to jog memories

  • Very short

  • Thanks those who completed and returned

  • Reminds others to “please do so today”

  • Statement of how important each recipient is to the success of the study

  • If you didn’t receive it, call and one will be sent

  • Signature

First replacement questionnaire fourth contact
First Replacement Questionnaire(Fourth Contact)

  • Send 2 weeks after thank you / reminder

  • Only sent to non responders (need to track)

  • Cover letter - more persuasive/insisting tone

    • Questionnaire not yet received

    • Others have responded/importance to study (representative)

    • Not eligible? (return questionnaire)

    • Confidential / voluntary (still return it unanswered)

    • Note of appreciation

    • Personally signed / P.S. (any questions)

  • Replacement questionnaire (with ID#)

  • Return postage paid envelope - but no token

  • Special contact fifth contact
    Special Contact(Fifth Contact)

    • 4 weeks later to non respondents [PK 2 wks]

    • Impact is due to a different “special” method which shows respondent & study are important

      • Telephone call (encourage/send new one/complete)

      • Registered mail (requires signature)

      • Special delivery

      • Courier

    • More expensive follow-up

    • “New” cover letter, questionnaire (ID#), postage paid envelope

    Survey error dillman
    Survey Error (Dillman)

    • Sampling error

      • Precision is limited by the number surveyed

    • Coverage error

      • Occurs when list does not include everyone

    • Measurement error

      • Results from poor wording and construction

    • Nonresponse error

      • When significant numbers don’t respond

    Example readings for other ideas
    Example Readings for Other Ideas

    • Increasing Response Rates to Postal Questionnaires (Edwards et al. 2002)

      • Systematic review of RCTs

      • 292 RCTs reviewed

      • 75 strategies for influencing response

    • A Catalog of Biases in Questionnaires (Choi and Pak, 2005)

      • Identified 48 common types of bias

      • Categorized into 3 main sources of bias

        • Question design

        • Questionnaire design

        • Questionnaire administration

    Problems issues

    • Undelivered questionnaires

      • People move/incomplete address or errors

      • Attempt to correct problems and resend

        • New mail out dates – additional work

    • Handling respondent questions

      • Should attempt to answer all questions

      • Should anticipate questions (e.g. ID#, no proxies…)

      • Thank them for asking

    • Evaluating early returns

      • Identify problems (e.g. printing, item response, certain questions causing problems….)

    Final note about mail survey response rates
    Final Note AboutMail Survey Response Rates

    • Mail surveys often criticized for low response

    • Example Response Rates (PK using “Dillman approach”)

      • Surveys of physicians

        • Physician Survey (elder abuse) 65%

        • Physician Survey (pneumonia) 77%

        • LTC Medical Director Survey 61%

      • Surveys of other health professionals

        • LTC Director of Care Survey 76%

        • Canadian Optometrist Survey 90%

      • Other “general population” health surveys

        • Contact Lens Use Survey 93%

        • LTC Family Member Satisfaction Surveys (80-86%)