Survey design and implementation
Download
1 / 73

Survey Design and Implementation - PowerPoint PPT Presentation


  • 115 Views
  • Uploaded on

Survey Design and Implementation. Elizabeth C Wells Doctoral Candidate Ag & Extension Education http://www.msu.edu/~wellselz/. Why do people fill out surveys? What do you want to ask/measure? Accuracy/Survey Error Random Sampling Writing Questions Questionnaire Design

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Survey Design and Implementation' - yuval


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Survey design and implementation

Survey Design and Implementation

Elizabeth C Wells

Doctoral Candidate

Ag & Extension Education

http://www.msu.edu/~wellselz/


Why do people fill out surveys?

What do you want to ask/measure?

Accuracy/Survey Error

Random Sampling

Writing Questions

Questionnaire Design

Survey Implementation

Telephone, Interview and Internet Surveys

UCRIHS


Why people fill out surveys
Why People Fill Out Surveys

  • Social Exchange Theory

    Theory of human behavior in which actions of individuals are motivated by the return these actions are expected to bring.

    Elements of reward, cost and trust


Social exchange theory
Social Exchange Theory

  • Rewards

    show positive regard

    say thank you

    ask for advice

    support group values

    give tangible rewards

    make the questionnaire interesting


Social exchange theory1
Social Exchange Theory

  • Reduce Social Costs

    avoid subordinating language

    avoid inconvenience

    avoid embarrassment

    make it short and easy

    minimize requests for personal information

    keep requests consistent


Social exchange theory2
Social Exchange Theory

  • Establish Trust

    provide token of appreciation in advance

    sponsorship by legitimate authority

    make task appear important

    invoke other exchange


Constructs and measurement
Constructs and Measurement

  • What do you want to know?

  • What are the variables?

  • How will you measure it?

    behavior, attitudes, feelings, demographics


Survey error
Survey Error

  • Sampling Error – Who are you sampling?

  • Coverage Error – Does your list include everyone?

  • Measurement Error – Does everyone answer a question the same way?

  • Non-response Error – Why did respondent not answer:

    Instrument (whole questionnaire not returned)

    Item (question not answered)


Non response error
Non Response Error

  • Only an issue if non responders would answer differently and change results

    compare early to late respondents

    double dip


Random sampling
Random Sampling

  • How much sampling error can you tolerate?

  • How large is the population?

  • How varied is the population?

  • How confident do you wish to be about estimates made from the sample?


Random sampling chart
Random Sampling - Chart

Pop Size 50/50 split 80/20 split

  • 80 71

  • 132 111

  • 196 153

  • 234 175

  • 260 188

    1000 278 198

Sample size for a 95% confidence level

And a + 5% sampling error


Random sampling1
Random Sampling

  • http://www.randomizer.org

    A www site that will generate random numbers which you can use for random selection of participants on a mailing list.


Writing questions
Writing Questions

  • Everyone will interpret the same way

  • Respondents are able to respond accurately

  • Respondents are willing to answer


Writing questions1
Writing Questions

  • Use simple words

  • Keep it short

  • Be specific

  • Do not talk down

  • Avoid bias

  • Avoid hypothetical questions

  • Use complete sentences


Question criteria
Question Criteria

  • Does the question require an answer?

    If you fixed dinner last night, did you eat meat as part of that meal?

  • To what extent does the respondent have a ready made and accurate answer?

    Are tall people more likely to be elected President?

  • Can respondent accurately recall and report past behavior?

    How much television did you watch last month?


Question principles
Question Principles

  • Use simple words and phrases

    tired……………………………exhausted

    honest…………………………candid

    work……………………………employment

    most important………………..top priority

    free time……………………….leisure

    doctor………………………….physician

    your answers………………….your responses to this

    questionnaire


Question principles1
Question Principles

  • Avoid vague quantifiers

    How often did you attend religious services last year?

     Never

     Rarely

     Occasionally

     Regularly


Question principles2
Question Principles

How often did you attend religious services during the past year?

 Not at all

 A few times

 About once a month

 Two or three times a month

 About once a week

 More than once a week


Question principles3
Question Principles

  • Avoid bias by using both positive and negative sides in the question stem

    “To what extent do you agree….”

    vs

    “To what extent do you agree or disagree with this statement…. “


Question principles4
Question Principles

  • Avoid double-barreled questions

    16. Should the new school have a swimming pool that includes lanes for swimming laps and is not enclosed for winter use?


Question principles5
Question Principles

  • Avoid asking respondent to say yes in order to mean no.

    5. Do you favor or oppose not allowing the state to approve a casino without approval from 60% of the voters?

     Favor

     Oppose


5. Do you favor or oppose requiring 60% approval by voters in order for a casino to be built.

 Favor requiring 60% approval

 Oppose requiring 60% approval


Question principles6
Question Principles in order for a casino to be built.

  • Use equal numbers of positive and negative categories for scalar questions AND distinguish undecided from neutral by placing at the end of the scale.


To what extent do you agree or disagree with this statement: “Listening to classical music is good for a persons emotional health.”

 Strongly agree

 Somewhat agree

 Neither agree nor disagree

 Somewhat disagree

 Strongly disagree

 No opinion


Question principles7
Question Principles “Listening to classical music is good for a persons emotional health.”

  • Soften impact of potentially objectionable questions:

    Have you ever shoplifted something from a store?

    vs

    Have you ever taken something from a store without paying for it?


22. What was your total income from all sources in 2003? “Listening to classical music is good for a persons emotional health.”

______________ Total income for 2003

22. Which category best describes your total income from all sources in 2003?

 $10,000. or less

 $10,001. to $20,000.

 $20,001. to $35,000.

 $35,001. to $50,000.

 $50,001. or above


Question principles8
Question Principles “Listening to classical music is good for a persons emotional health.”

  • Don’t combine numbers and words

    Doing so may impact respondents perception of the scale.

    1= strongly agree, 2 = agree,

    3 = disagree, 4 = strongly disagree


Question principles9
Question Principles “Listening to classical music is good for a persons emotional health.”

  • Eliminate check-all-that-apply questions to reduce primacy and sufficiency effect.

    People tend to check the first ones more than the last ones and sometimes check a few and feel that is enough to satisfy the researcher.


Question principles10

Extension Agent “Listening to classical music is good for a persons emotional health.”

Other Cattlemen

Fertilizer and other salesmen

59% selected

Extension Agent

Other Cattlemen

Fertilizer and other salesmen

Extension Agent

41% selected

Extension Agent

Question Principles


Questionnaire design
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Paper and layout choice

    booklet format preferred

    one column for legal size

    two columns for full size

    avoid upper left staple design

    avoid unusual size, shape or fold


Order of questions
Order of Questions “Listening to classical music is good for a persons emotional health.”

  • A questionnaire is like a conversation. It evolves according to social norms. Constantly switching topics makes it appear as though the researcher is not listening to the respondents answers.

  • Doing otherwise results in answers that are less thought out when you want them to focus!


Order of questions1
Order of Questions “Listening to classical music is good for a persons emotional health.”

  • What was your family income in 2003?

  • Do you like to play golf?

  • What is your opinion on Global Warming?

  • Are you married?

  • How often have you gone fishing this year?

  • How adequate is your current health care?

  • How old are you?

  • What is your political party preference?

  • What is your occupation?


Question order
Question Order “Listening to classical music is good for a persons emotional health.”

  • Group alike questions together

  • Begin asking questions in the upper left corner

  • Ask one question at a time

  • Use item-in-a-series format


The first question
The First Question “Listening to classical music is good for a persons emotional health.”

  • Should clearly apply to everyone

  • Question should be easy

  • Question needs to be interesting

  • Connectedness between the question and the purpose of the survey.


Unacceptable first questions
Unacceptable First Questions “Listening to classical music is good for a persons emotional health.”

1. Please think about the things that make MSU Extension a pleasant place to work and write down the five most important aspects and rate them one to five in importance to you.

.


Unacceptable first questions1
Unacceptable First Questions “Listening to classical music is good for a persons emotional health.”

1. Please describe in your own words what you consider good about working for MSU Extension.

1. What year were you born?

_________year born


Good first question
Good First Question “Listening to classical music is good for a persons emotional health.”

1. Thinking about MSU Extension, how would you rate it as a place to work?

 Excellent

 Good

 Fair

 Poor


Graphic design
Graphic Design “Listening to classical music is good for a persons emotional health.”

  • Stimuli on page consists of two languages:

    written words

    graphic symbols and arrangements


Graphic design1
Graphic Design “Listening to classical music is good for a persons emotional health.”

  • Desired navigational path

  • Create visual navigational guides

  • Develop additional guides for redirecting during skip patterns

  • Be consistent through out your instrument


Questionnaire design1
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Instructions need to be exactly where needed not at the beginning of the whole questionnaire.

  • Use italics or bolding or underline to emphasize instructions and directions


Questionnaire design2
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Pre-attentive processing is 210 degrees

  • Focused reading is 2 degrees or about 8 to 9 characters

  • Use font, size, contrast, underline, spacing to lead the respondent through the questionnaire


Start here: “Listening to classical music is good for a persons emotional health.”

  • Which of the following is your main work activity?

    Research

     Teaching

     Administration

     Other (Please Specify) _________________


START HERE: “Listening to classical music is good for a persons emotional health.”

Which of the following is your main work activity?

 Research

 Teaching

 Administration

 Other (Please specify) ____________________


Questionnaire format
Questionnaire Format “Listening to classical music is good for a persons emotional health.”

  • For paper questionnaires the standard font is 12 pt Times New Roman – also consider a larger font for senior audiences

  • For on-line questionnaires the standard font is 10 pt Ariel


Questionnaire design3
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Use a square for your answer and ask for an “X” in the square.

    [ ]

    1


Questionnaire design4
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Place instructions exactly where needed within the question

  • Differentiate with font, bolding, underlining etc


5. “Listening to classical music is good for a persons emotional health.”How many months have you worked at your current job?

_________Number of months

Please be as specific as possible in answering the next question, including any area of specialization. Example: Extension Agent – Agriculture. If you had more than one job, answer for the job for which you worked the most hours.

6. What kind of work do you do in your current job?

_________________________________ Kind of work


5. “Listening to classical music is good for a persons emotional health.”How many months have you worked in your current job?

_____________________ Number of months

6. What kind of work do you do in your current job? Please be as specific as possible in answering. Include any are of specialization, for example: “Extension Agent – Agriculture”. If you had more than one job, answer for the job which you worked the most hours.

______________________________Kind of work


Questionnaire design5
Questionnaire Design “Listening to classical music is good for a persons emotional health.”

  • Vertical alignment eases response task

  • Don’t double or triple bank answers


1 “Listening to classical music is good for a persons emotional health.”. Which one of the following best describes the 4-H project with the largest enrollment within your club?

  • market beef

  • lambs

  • hogs

  • dogs

  • cats

  • rabbits

  • goats

  • poultry

  • dairy


1. Which one of the following best describes the 4-H project with the largest enrollment within your club?

  • beef

  • hogs

  • lambs

  • dogs

  • cats

  • rabbits

  • goats

  • poultry

  • dairy


Questionnaire design6
Questionnaire Design with the largest enrollment within your club?

  • Major changes are needed for skip patterns

  • Skip patterns are difficult to follow

  • Skip patterns often lead to item non response


  • Normally do you eat breakfast every day? with the largest enrollment within your club?

  •  Yes (Go to 13)

  •  No (Go to 18)

  • 13. How many times a week do you drink orange juice with your breakfast?

    __________ Times a week


12. Normally do you eat breakfast every day? with the largest enrollment within your club?

 No (Skip to 18)

 Yes

13. (If Yes) How often do you drink orange juice with your breakfast?

____________ Times a week


Questionnaire design7
Questionnaire Design with the largest enrollment within your club?

  • Demographic data questions

    Place at end of questionnaire but not on

    back page

    Only ask what is necessary


Questionnaire design8
Questionnaire Design with the largest enrollment within your club?

  • Front cover needs to stand out

    Consider simple art work

    Include name of survey

    Include name and address for return


Questionnaire design9
Questionnaire Design with the largest enrollment within your club?

  • Back cover needs to be simple with white space

    Try not to ask specific questions

    May place open ended question here

    Give opportunity to add comments

    Conclude with a thank you


Implementation
Implementation with the largest enrollment within your club?

  • Respondent friendly questionnaire

  • Multiple contacts

  • Token pre-paid incentive

  • SASA for return

  • Personalized correspondence


Implementation1
Implementation with the largest enrollment within your club?

  • Four contacts plus one “special” = 5 contacts

    Brief pre notice letter

    Questionnaire mailing with SASE

    Thank you postcard

    Replacement questionnaire

    Final contact with “special” mailing


Implementation contact 1
Implementation – Contact 1 with the largest enrollment within your club?

  • Pre-notice letter:

    Date at top

    Personalized letter

    What will happen

    What it is about

    Usefulness of survey

    Thank you

    Real signature

    P.S. Token incentive

Mail out a few days to one week before mailing instrument

4-6% higher response rate


Implementation contact 2
Implementation – Contact 2 with the largest enrollment within your club?

  • Mail the questionnaire

    Include personalized cover letter

    Questionnaire

    Self addressed stamped envelope with real stamp

    Enclose incentive

    Use first class postage – plain envelope


Implementation cover letter
Implementation – Cover Letter with the largest enrollment within your club?

  • Inside address

  • Request

  • Why you were selected

  • Usefulness of survey

  • Confidentiality

  • Token of appreciation

  • Willingness to answer questions

  • Thank you

  • Real signature in contrasting ink


Implementation token incentive
Implementation – Token Incentive with the largest enrollment within your club?

  • Cash in advance $1. or $2. = 12% to 19% increase in response rate

  • Less of an increase for larger amount later

  • Increasing incentive not cost effective

  • Non dollar incentive not as effective and may be effective because of packaging


Implementation contact 3

Postcard to all who were mailed a questionnaire thanking them

Serves to jog memory

Gives opportunity to call for replacement if needed

Implementation – Contact 3

Mail one week after questionnaire

Increase up to 13%


Implementation contact 4
Implementation – Contact 4 them

  • Replacement questionnaire to those who have not yet responded.

  • Letter reminding them of the survey and asking again for their assistance.

Sent only to non-respondents

Two to four weeks after 3rd contact


Implementation contact 5
Implementation – Contact 5 them

  • Special mailing to those who have not yet responded

  • Use unique method of mailing – certified, priority etc

  • May also telephone and fill out over the phone

Final contact is a week or more after 4th contact


Tracking responses
Tracking Responses them

  • Assign each questionnaire a number

  • Assign each name and address a number

  • Send respondents their individually numbered questionnaire

  • Have assistant mark off numbers as they are returned

  • Note surveys are confidential not anonymous

  • Keep completed questionnaires in secure location


Mixed mode surveys
Mixed Mode Surveys them

Responses differ by mode:

“Excellent Health” = 44% by personal interview

= 37% by telephone

= 30% by mail

social desirability

acquiescence

primacy vs recency effect


Telephone surveys
Telephone Surveys them

  • Difficult to obtain phone numbers

  • Caller id and answering machines may screen calls

  • Need well trained interviewers

  • Limited number of questions and answers


Internet surveys
Internet Surveys them

  • Consider coverage error. Does your entire audience have and use the internet on a regular basis?

  • E-mail addresses change often

  • Difficult to include incentive

  • Shorter less detailed questionnaire

    Survey Monkey


Research ethics
Research Ethics them

Anonymous – No way to know who returned the survey

Confidential – Only researcher can connect the respondent with their answers. All answers remain confidential.


Research ethics1
Research Ethics them

  • If you wish to publish results you MUST get approval from UCRIHS before you send out surveys.

    University Committee on Research Involving Human Subject


References
References them

Mail and Internet Surveys – The Tailored

Design Method by Don Dillman 2000


ad