Developing effective surveys
1 / 23

Developing Effective Surveys - PowerPoint PPT Presentation

  • Updated On :

Developing Effective Surveys. Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) [email protected] September, 2009. Today’s Agenda. Discuss survey design as a process Provide tips to enhance the quality of the process and therefore the data.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Developing Effective Surveys' - edana

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Developing effective surveys

Developing Effective Surveys

Steve Culver, Ph.D.

Associate Director

Office of Academic Assesment

122 Hillcrest (0157)

[email protected]

September, 2009

Today s agenda
Today’s Agenda

  • Discuss survey design as a process

  • Provide tips to enhance the quality of the process and therefore the data


  • A survey is a data-gathering and analysis approach in which respondents answer questions or respond to statements developed in advance.

  • IRB – Institutional Review Board must approve “research”

Types of surveys
Types of surveys

  • Face to face (usually most expensive)

    more control, sometimes can collect more data

  • Telephone

    fairly controlled setting, can ask for clarification, but phone screening, fewer hours at home

  • Mail (takes longer)

    Respondent can complete at leisure, can include pictures; no control over who completes survey, potential illegible data; bias - literacy

  • Web (e-mail survey or survey posted on web)

    more consistent (reliable) than human interviewer, but no human element to identify problems, no control over data errors; respondents give longer answers to open-ended questions.

Steps in the process
Steps in the process

  • Identify research objective

  • Identify & characterize the target audiences

  • Design the sampling plan

  • Design & write the questionnaire

  • Pilot test the questionnaire

  • Distribute the questionnaire

  • Analyze the results & write the report

Identify research objective
Identify research objective

  • Define precisely the information desired

  • Try to write as few questions as possible to obtain that information

  • A clear-cut need for every question should be established

Identify characterize the target audiences
Identify & characterize the target audiences

  • Is it a wide range of people or a niche group?

  • What is there interest in the topic?

  • What is their reading/educational level?

  • Are there other important characteristics to consider that relate to your study? (how would you know?)

Design the sampling plan
Design the sampling plan

  • Address how a sample of people should be selected to meet the study objectives and to obtain reliable information.

  • Need to identify the accessible population.

  • Sampling methods: random, systematic, stratified

  • Follow up with the nonrespondents.

Improving your response rate
Improving your response rate

  • Increase perceived rewards

    say thank you, ask them for advice, make questionnaire interesting

  • Decrease perceived costs

    avoid subordinating language, don’t embarrass the respondent, make it quick and easy to respond

  • Establishing trust

    make the questionnaire seem important, provide a sense of legitimate authority, provide a token

Design write the questionnaire
Design & write the questionnaire

  • As simple, as short as possible

  • Unambiguous items

  • No leading questions

  • No double-barreled questions

  • Questions follow a logical sequence

  • Put personal or confidential questions at the end.

Layout design
Layout & Design

  • Clear introductory statement & contact information

  • KISS & lessen clutter

  • Number items and be consistent with wording, font (serif font like Times easier to read than a san serif font).

  • Use italics and bold judiciously and with a purpose.

  • Careful with color & limit graphics

Particularly for web surveys
Particularly for web surveys…

  • Design the survey as multiple pages, with a submit button at end of each section (not one long page)

  • Use a progress button.

  • Decide if respondents can exit and re-enter survey.

  • Design so that it takes no longer than 20 minutes to complete the survey.

Question order
Question order

  • Begin with questions that raise interest but are easy to answer

  • Then ask questions of interest that are harder to answer

  • Personal questions at end after respondent has committed to answering (though completion of web surveys is an issue, so maybe move some to middle)

  • Make logical groupings

Types of questions
Types of questions

  • Open-ended vs. closed-ended

    • CE = easy to standardize, analyze

    • OE = easier to write, draw in respondent

  • Single vs. multiple response

    • Choose one; choose all that apply

  • Ranked responses

    • Indicate order of importance of a list

  • Rated responses

    • Likert scales

Writing tips
Writing tips

  • Avoid double barreled questions

    • Do you think college students today should eat less and exercise more?

  • Avoid questions with negatives

    • Are you against a ban on smoking?

  • Ask for precise answers

    • Your age on January 1, 2009

More writing tips response scales
More writing tips – Response scales

--allow for maximum variability (no more than 10; less than h.s. educ., 5)

--use a balanced scale

--“neutral” or “no opinion” vs. “don’t know” (odd/even number of responses)

--Order response categories in a logical way

Pilot test the questionnaire
Pilot test the questionnaire

  • Field pre-testing

  • Focus group discussions

  • Individual interviews – “think aloud”

  • Behavior coding

    does the respondent ask for clarification?

    How long does it take to answer each question?

Distribute the questionnaire
Distribute the questionnaire

  • After revision from pilot test, distribute the questionnaire

  • Document the how, where, when

  • Document response times, nonrespondents

  • Follow up nonrespondents

  • So, distribution is seen as a longitudinal process

Analyze the results write the report
Analyze the results & write the report

  • think in terms of useful information directed toward change, rather than data

  • Clear presentation – not excessive pretty charts and graphs

In summary
In summary . . .

  • Be mindful that you are using someone else’s time for your survey

  • Think of the survey process as a process, not just an instrument

  • Be careful of extending your results to make them mean more than they do


Dillman, D. A. (1978). Mail and telephone surveys: The total design method. NY: Wiley.

Porter, S. (2004). Pros and cons of paper and electronic surveys, overcoming survey research problems new Directions for Institutional Research, 121, 91-99.

Frary, R. B. (1996). Brief guide to questionnaire development. Washington, DC: ERIC Clearinghouse on Assessment and evaluation.

Helpful websites
Helpful websites

  • Resources for program evaluation and social research methods

  • Guide to the design of questionnaires

Questions comments

Please take a few minutes to

complete the session feedback


Thank you!