Asking the right questions developing effective surveys
Download
1 / 48

Asking the Right Questions: Developing Effective Surveys - PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on

Asking the Right Questions: Developing Effective Surveys. An evaluation capacity-building training from the Tobacco Control Evaluation Center by Robin Kipke & Travis Satterlund June 10, 2011. What We’ll Be Covering. 10:00 Introductions, take knowledge pre-test 10:40 End-use strategizing

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Asking the Right Questions: Developing Effective Surveys' - dara


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Asking the right questions developing effective surveys
Asking the Right Questions:Developing Effective Surveys

An evaluation capacity-building training

from the Tobacco Control Evaluation Center

by Robin Kipke & Travis Satterlund

June 10, 2011


What we ll be covering
What We’ll Be Covering

10:00 Introductions, take knowledge pre-test

10:40 End-use strategizing

11:35 Question types

11:50 Dillman principles for writing questions

12:15 Lunch break

1:15 Dillman continued

1:35 Writing Questions -- Practice

2:25 Survey sequencing & construction

3:00 Field testing surveys

3:10 Learning recap, setting action plans

3:20 Complete exit survey

3:30 Optional consultation with TCEC associates


Training objectives
Training Objectives

Participants can explain and apply these concepts:

  • A survey is like a conversation

  • The 1st step of developing a survey is end-use strategizing

  • The aim of survey design is to reduce non-response and measurement error


A survey is like a conversation in that

Is communication with a purpose

Begins with an introduction

Needs to capture respondent’s interest

Starts with easy-to-answer ?s, builds to more substantial

Follows logical order, uses transitions to change topic

Finds a delicate way to raise sensitive issues

Winds down with less consequential subject

Indicates end with a sign off

A survey is like a conversation in that…

Photo by Robert Thivierge


Where survey design fits in
Where Survey Design Fits In

Source: Youth Media Evaluation Toolkit 2005


What surveys can tell you
What Surveys Can Tell You

To inform your project about

  • Need for education or outreach

  • Community priorities or policy options

  • Level of support or opposition among stakeholder groups

  • Makeup of local populations

  • Extent of any change effected

  • Satisfaction with services



Using reverse logic to develop questions
Using Reverse Logic to Develop Questions

Apply process to evaluation planning and development of data collection instruments

  • Purpose of data

  • Target audience for the data

  • How information to be used

  • Likely critics

  • Credibility threshold

  • Pieces of data

  • Data sources and formats


How it works
How It Works

An example

Try it out – pair activity

Think of the project you work on and how a survey could inform your efforts

Discuss with your team what you might want a survey to tell you, how it could be used

Work together to fill in each of the boxes of the End-use Strategizing worksheet

9


Questions information types
Questions –Information Types

  • Attitudes—What one wants or prefers

  • Beliefs—What one thinks to be true

  • Behavior—What one does or has done

  • Attributes—What one is


Questions structure types
Questions—Structure Types

  • Open-ended—No answer choices are offered

  • Closed-ended—Answer choices are offered


Open ended questions
Open-Ended Questions

Examples:

  • Short Answer

    • How long have you lived in this apartment?

  • Clarification (as part of skip pattern)

    • If you answered “yes” to the previous question, please explain why...

  • Comments

    • Please write any additional comments you may have about the potential smoke-free policy.


Open ended questions1
Open-Ended Questions

Drawbacks:

  • Respondents may find it difficult to express their feelings

  • They take more time

  • Can yield inadequate answers without probing, follow-up questions

  • Analysis is time consuming and difficult


Closed ended questions
Closed-Ended Questions

Examples:

  • Yes/No questions

    • Have you used any tobacco products in the last 30 days?

       Yes  No

  • Multiple Choice

    • How many bedrooms does this apartment have?

       Zero (Studio)  One  Two  Three

       Other _________________


Closed ended questions1
Closed-Ended Questions

  • Likert-type Scales

    • To what degree would you favor or oppose a policy to make at least half of the individual units in this apartment complex non-smoking?

    • How often do you attend coalition meetings?


Closed ended questions2
Closed-Ended Questions

Examples (continued):

  • Ranking

    • On a scale of 1 to 5, rank the issues that matter most to you with 1 being most important.

      ___ Health care

      ___ Environmental protection

      ___ Safe neighborhoods

      ___ Quality of education

      ___ Fair wages


“Survey design is all about motivating people adequately so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

~ Don Dillman


The dillman principles
The Dillman Principles so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • The aim of survey design is to minimize measurement and non-response error

  • Measurement error: poor question wording or formatting leads to inaccurate answers

  • Non-response error: people who respond to the survey are different from those who did not

    Don A. Dillman. 2007. Mail and Internet Surveys: The Tailored Design Method, 2nd ed. Hoboken, NJ: John Wiley & Sons.


Creating respondent buy in
Creating Respondent Buy-in so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • Motivate people to begin and complete survey

    • Use introduction/cover letter

    • Give compelling reason to participate

    • Ask interesting first question

  • Build trust by demonstrating competence

    • Employ good survey design

    • Make it easy to understand what to do


Example introductions
Example Introductions so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • The Bonanza County Public Health Department is interested in finding out about how tenants feel about being around tobacco smoke. There are no right or wrong answers and they will be kept anonymous. We hope you will share your opinions with us.

  • To help protect Bonanza County multi-unit housing (MUH) residents from the dangers of secondhand smoke, the county Tobacco Prevention Program will be working with MUH owners and managers to adopt and implement a voluntary policy that prohibits smoking in…

  • The Bonanza County Tobacco Education Coalition is seeking public opinion on the effects of exposure to secondhand smoke and possible smoking policies. Your opinions are very important to us.

  • Smoke-Free Apartment Complexes – Tenant Survey


Managing cognitive energy
Managing Cognitive Energy so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • Respondents will devote only a limited amount of mental effort to complete your survey

  • Make survey easy to understand, navigate and respond to

  • Cluster related topics

  • Cluster similar response types

  • Weigh need for survey length vs. complexity of questions


Keep population of interest in mind
Keep Population of Interest in Mind so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • What cultural characteristics might affect their ability to understand the survey?

  • What is their literacy level?

  • What language do they feel most comfortable with?

  • How much time will they be willing to spend on taking the survey?

  • Is this a topic that interests them?


Dillman 1 keep it simple
Dillman #1: Keep It Simple so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • Choose simple rather than specialized words

  • Use as few words as possible

  • (Sometimes these two rules conflict)


Dillman 2 say what you mean
Dillman #2: Say What You Mean so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • Use complete sentences to ask questions(even when they seem self-evidently clear)

  • Please check one:  Male  Female

  • Age: ____

  • What is your gender?  Male  Female

  • How old are you? _____ years

Problematic:

Revised:


Dillman 3 it s all about timing
Dillman #3: It’s All about Timing so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • Avoid vague qualifiers

  • Instead use more specific frames

    • How often do you dine out at a restaurant?


Dillman 4 balancing act
Dillman #4: Balancing Act so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

  • To avoid bias, state both sides of the attitudinal scale in the question stem

    • Would you favor or oppose a city policy to make all parks non-smoking?

  • Balance scales with equal number of positive and negative choices

    • Very likely, somewhat likely, not very likely

  • Neutral position is different from undecided

    • Neither supportive nor unsupportive vs. don’t know

  • Problematic:


    Balancing act example
    Balancing Act Example so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    Good Example:

    To what extent do you agree or disagree with the statement “Smokers have a right to smoke”?

    • Strongly agree

    • Somewhat agree

    • Neither agree nor disagree

    • Somewhat disagree

    • Strongly disagree

    • I don’t know


    Dillman 5 pick me
    Dillman #5: Pick Me! so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Beware of primacy effects in “mark all that apply” lists

    • Lists are OK for factual questions but not attitudes or preferences

    • Make each issue a yes/no or scalar question

      • Which type of organization do you represent in the coalition? (mark all that apply)

        Local lead agency  Service organization

         Law enforcement  Educational institution

         Other ____________  I don’t represent an organization

    Ok for list


    • Problematic: so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Which of the following areas in the apartment complex do you think should be made non-smoking? (Mark all that apply)

    • courtyard  pool area  barbeque areas

    • balconies/patios  mailboxes  laundry rooms

      Revised:

    • Would you like any of the following areas of your apartment complex to be made non-smoking?

    • Courtyard  yes  no

    • Pool area  yes  no

    • Barbeque areas  yes  no

    • Balconies/patios  yes  no

    • Mailboxes  yes  no

    • Laundry rooms  yes  no


    Dillman 6 framing the issue
    Dillman #6: Framing the Issue so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Use cognitive techniques to improve recall

      • Guided imagery

      • Layer questions

        Problematic:

        When you have watched movies where actorswere smoking, did it make smoking seem more appealing to you?


    Framing the recall
    Framing the Recall so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    Revised:

    Think about the last three movies you saw.

    1. What type of movies were they? (mark all that apply)

     Action  Comedy  Drama  Documentary

    2.In any of those movies, did any of the characters smoke?

     Yes

     No (go to question 4)

    3. Did seeing the characters smoking make cigarettes seem more appealing to you?

     Yes

     No


    Framing sensitive issues
    Framing Sensitive Issues so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Save more sensitive questions towards the end

      • Once you’ve established a rapport

      • In case respondents refuse to proceed

    • Soften the impact of potentially objectionable questions

      • Preface personal questions with more general ones about the issue or ask about other people


    • Problematic so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”:

    • Have you ever shoplifted any tobacco products from a store?

    • Revised Versions:

    • The questions which follow are being asked to help us understand where young people get tobacco products when they are underage. We really appreciate your help and that of students all over the state who have been asked to complete this survey honestly.

    • V1. Have you ever taken any tobacco products from a store without paying for them?

    • V2. Have you ever obtained tobacco products from…

    • a friend or relative?  yes  no

    • a store without paying?  yes  no


    Dillman 7 six of one half dozen of the other
    Dillman #7: Six of One, Half Dozen of the Other so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Make sure answer choices are mutually exclusive (numbers, conceptual overlaps)

      Problematic:

      How old are you?

       15-18 yrs. old 18-30 yrs. old  30-50 yrs. old  50+ yrs. old

      When you tried to quit, where did you turn to for help?  my doctor  support group

       online service  quitline

       clinic or hospital  on my own


    Dillman 8 it doesn t add up
    Dillman #8: It Doesn’t Add Up so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Ensure that respondents can answer the question

    • Is it beyond the knowledge of respondents?

      • Were you aware that the California Air Resources Board has declared secondhand smoke to be a toxic air contaminant?

      • If the city were to pass a tobacco retail ordinance, how much should a license cost?

    • Avoid excessive specificity

      • How many cigarettes have you smoked within the last 30 days?


    It doesn t add up continued
    It Doesn’t Add Up continued so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Use timeframes within memory

      In the last year, how many tenants complained about drifting tobacco smoke?

    • Avoid unnecessary calculations

      How long have you lived in this apartment?

      ______ months


    Dillman 9 don t be negative
    Dillman #9: Don’t Be Negative! so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Avoid asking respondents to say “yes” in order to mean “no”

      Problematic:

      In which areas of the apartment complex should people not be able to smoke?

      Revisions:

      v1 Which areas of the apartment complex would you like to be non-smoking?

      v2 In which areas of the apartment complex should smoking be prohibited?


    Dillman 10 over a barrel
    Dillman #10: Over a Barrel so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Avoid double-barreled questions where two things are being asked in the same question

    • Watch out for “and”

    • Instead collapse into illustrative category, split into two questions, use “or”


    Over a barrel
    Over a Barrel? so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    Is either of these double-barreled?

    • If the downtown area was free of secondhand smoke and cigarette litter, do you think you would be more or less likely to shop and attend events there?

    • Do you think retailers should have to pay for a license to sell tobacco which would earmark a portion of the funds to cover the expense of enforcement?


    Now you try it
    Now You Try it so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    Write one question using each of these formats:

    Yes/no

    Multiple choice

    Scalar

    Open-ended

    40


    Survey design considerations
    Survey Design Considerations so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Create a clear navigational path

    • Provide signposts to guide respondents

    • Use graphic elements as clues

      START ❶ Circle the response

    • Keep format visually uncluttered


    Survey design considerations1
    Survey Design Considerations so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Cluster like topics (e.g., knowledge, preferences, support)

    • Also cluster similar response formats (Likert scales, statements of agreement, etc.)

    • Use consistent scale directions throughout

    • Organize choices vertically, not horizontally

    • Beware of response set effect

    • Use pamphlet layout


    Sequencing questions
    Sequencing Questions so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Remember a survey is like a conversation

    • First engage interest, build trust, ensure success

    • Move from easy-to-answer to complex

    • General to more specific

    • Less personal to more sensitive

    • Balance open-ended vs. closed-ended

    • Save demographic questions for the end


    Field testing your survey
    Field Testing Your Survey so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Why it’s important

    • Who to involve

    • How to go about it

    • What to look for

    • What to do with the feedback


    Tcec your resource center
    TCEC: Your Resource Center so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Recorded webinars & training modules:

      • Online Surveys: Techniques & Tips (12/2/10)

      • Reducing Error: Designing Surveys that Work (9/30/10)

      • Journey of a Survey (1/28/10)

      • Developing a Survey Instrument (3/26/09)

      • End-use Strategizing for Creating DCIs (12/10/08)

      • Public Opinion Surveys (mini training)

    • End Use Strategizing Checklist

    • Tips & Tools #2 on writing questions

    • Hundreds of survey instruments in repository

    • Individualized help from Evaluation Specialists


    Reflecting on learning
    Reflecting on Learning so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    • Surveys are more than a brainstormed list of ???s

    • Start with end-use strategizing process

    • A survey is like a conversation

    • Follow design principles to reduce error

    • Need to motivate respondents

    • Manage finite amount of cognitive energy

    • Create navigational path

    • Make it easy to complete!


    To reach us: so they complete the cognitive steps necessary for answering questions accurately and return the questionnaire.”

    Email: [email protected]

    http://programeval.ucdavis.edu

    Main phone line: 530.752.9951


    ad