Developing effective surveys
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23

Developing Effective Surveys PowerPoint PPT Presentation


  • 179 Views
  • Uploaded on
  • Presentation posted in: General

Developing Effective Surveys. Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) [email protected] September, 2009. Today’s Agenda. Discuss survey design as a process Provide tips to enhance the quality of the process and therefore the data.

Download Presentation

Developing Effective Surveys

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Developing effective surveys

Developing Effective Surveys

Steve Culver, Ph.D.

Associate Director

Office of Academic Assesment

122 Hillcrest (0157)

[email protected]

September, 2009


Today s agenda

Today’s Agenda

  • Discuss survey design as a process

  • Provide tips to enhance the quality of the process and therefore the data


Definitions

Definitions

  • A survey is a data-gathering and analysis approach in which respondents answer questions or respond to statements developed in advance.

  • IRB – Institutional Review Board must approve “research”


Types of surveys

Types of surveys

  • Face to face (usually most expensive)

    more control, sometimes can collect more data

  • Telephone

    fairly controlled setting, can ask for clarification, but phone screening, fewer hours at home

  • Mail (takes longer)

    Respondent can complete at leisure, can include pictures; no control over who completes survey, potential illegible data; bias - literacy

  • Web (e-mail survey or survey posted on web)

    more consistent (reliable) than human interviewer, but no human element to identify problems, no control over data errors; respondents give longer answers to open-ended questions.


Steps in the process

Steps in the process

  • Identify research objective

  • Identify & characterize the target audiences

  • Design the sampling plan

  • Design & write the questionnaire

  • Pilot test the questionnaire

  • Distribute the questionnaire

  • Analyze the results & write the report


Identify research objective

Identify research objective

  • Define precisely the information desired

  • Try to write as few questions as possible to obtain that information

  • A clear-cut need for every question should be established


Identify characterize the target audiences

Identify & characterize the target audiences

  • Is it a wide range of people or a niche group?

  • What is there interest in the topic?

  • What is their reading/educational level?

  • Are there other important characteristics to consider that relate to your study? (how would you know?)


Design the sampling plan

Design the sampling plan

  • Address how a sample of people should be selected to meet the study objectives and to obtain reliable information.

  • Need to identify the accessible population.

  • Sampling methods: random, systematic, stratified

  • Follow up with the nonrespondents.


Improving your response rate

Improving your response rate

  • Increase perceived rewards

    say thank you, ask them for advice, make questionnaire interesting

  • Decrease perceived costs

    avoid subordinating language, don’t embarrass the respondent, make it quick and easy to respond

  • Establishing trust

    make the questionnaire seem important, provide a sense of legitimate authority, provide a token


Design write the questionnaire

Design & write the questionnaire

  • As simple, as short as possible

  • Unambiguous items

  • No leading questions

  • No double-barreled questions

  • Questions follow a logical sequence

  • Put personal or confidential questions at the end.


Layout design

Layout & Design

  • Clear introductory statement & contact information

  • KISS & lessen clutter

  • Number items and be consistent with wording, font (serif font like Times easier to read than a san serif font).

  • Use italics and bold judiciously and with a purpose.

  • Careful with color & limit graphics


Particularly for web surveys

Particularly for web surveys…

  • Design the survey as multiple pages, with a submit button at end of each section (not one long page)

  • Use a progress button.

  • Decide if respondents can exit and re-enter survey.

  • Design so that it takes no longer than 20 minutes to complete the survey.


Question order

Question order

  • Begin with questions that raise interest but are easy to answer

  • Then ask questions of interest that are harder to answer

  • Personal questions at end after respondent has committed to answering (though completion of web surveys is an issue, so maybe move some to middle)

  • Make logical groupings


Types of questions

Types of questions

  • Open-ended vs. closed-ended

    • CE = easy to standardize, analyze

    • OE = easier to write, draw in respondent

  • Single vs. multiple response

    • Choose one; choose all that apply

  • Ranked responses

    • Indicate order of importance of a list

  • Rated responses

    • Likert scales


Writing tips

Writing tips

  • Avoid double barreled questions

    • Do you think college students today should eat less and exercise more?

  • Avoid questions with negatives

    • Are you against a ban on smoking?

  • Ask for precise answers

    • Your age on January 1, 2009


More writing tips response scales

More writing tips – Response scales

--allow for maximum variability (no more than 10; less than h.s. educ., 5)

--use a balanced scale

--“neutral” or “no opinion” vs. “don’t know” (odd/even number of responses)

--Order response categories in a logical way


Pilot test the questionnaire

Pilot test the questionnaire

  • Field pre-testing

  • Focus group discussions

  • Individual interviews – “think aloud”

  • Behavior coding

    does the respondent ask for clarification?

    How long does it take to answer each question?


Distribute the questionnaire

Distribute the questionnaire

  • After revision from pilot test, distribute the questionnaire

  • Document the how, where, when

  • Document response times, nonrespondents

  • Follow up nonrespondents

  • So, distribution is seen as a longitudinal process


Analyze the results write the report

Analyze the results & write the report

  • think in terms of useful information directed toward change, rather than data

  • Clear presentation – not excessive pretty charts and graphs


In summary

In summary . . .

  • Be mindful that you are using someone else’s time for your survey

  • Think of the survey process as a process, not just an instrument

  • Be careful of extending your results to make them mean more than they do


References

References

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. NY: Wiley.

Porter, S. (2004). Pros and cons of paper and electronic surveys, overcoming survey research problems new Directions for Institutional Research, 121, 91-99.

Frary, R. B. (1996). Brief guide to questionnaire development. Washington, DC: ERIC Clearinghouse on Assessment and evaluation.


Helpful websites

Helpful websites

  • Resources for program evaluation and social research methods http://gsociology.icaap.org/methods/

  • Guide to the design of questionnaires http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires


Questions comments

Questions/Comments?

Please take a few minutes to

complete the session feedback

Form.

Thank you!


  • Login