developing effective surveys l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Developing Effective Surveys PowerPoint Presentation
Download Presentation
Developing Effective Surveys

Loading in 2 Seconds...

play fullscreen
1 / 23

Developing Effective Surveys - PowerPoint PPT Presentation


  • 114 Views
  • Uploaded on

Developing Effective Surveys. Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) sculver@vt.edu September, 2009. Today’s Agenda. Discuss survey design as a process Provide tips to enhance the quality of the process and therefore the data.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Developing Effective Surveys' - leola


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
developing effective surveys

Developing Effective Surveys

Steve Culver, Ph.D.

Associate Director

Office of Academic Assesment

122 Hillcrest (0157)

sculver@vt.edu

September, 2009

today s agenda
Today’s Agenda
  • Discuss survey design as a process
  • Provide tips to enhance the quality of the process and therefore the data
definitions
Definitions
  • A survey is a data-gathering and analysis approach in which respondents answer questions or respond to statements developed in advance.
  • IRB – Institutional Review Board must approve “research”
types of surveys
Types of surveys
  • Face to face (usually most expensive)

more control, sometimes can collect more data

  • Telephone

fairly controlled setting, can ask for clarification, but phone screening, fewer hours at home

  • Mail (takes longer)

Respondent can complete at leisure, can include pictures; no control over who completes survey, potential illegible data; bias - literacy

  • Web (e-mail survey or survey posted on web)

more consistent (reliable) than human interviewer, but no human element to identify problems, no control over data errors; respondents give longer answers to open-ended questions.

steps in the process
Steps in the process
  • Identify research objective
  • Identify & characterize the target audiences
  • Design the sampling plan
  • Design & write the questionnaire
  • Pilot test the questionnaire
  • Distribute the questionnaire
  • Analyze the results & write the report
identify research objective
Identify research objective
  • Define precisely the information desired
  • Try to write as few questions as possible to obtain that information
  • A clear-cut need for every question should be established
identify characterize the target audiences
Identify & characterize the target audiences
  • Is it a wide range of people or a niche group?
  • What is there interest in the topic?
  • What is their reading/educational level?
  • Are there other important characteristics to consider that relate to your study? (how would you know?)
design the sampling plan
Design the sampling plan
  • Address how a sample of people should be selected to meet the study objectives and to obtain reliable information.
  • Need to identify the accessible population.
  • Sampling methods: random, systematic, stratified
  • Follow up with the nonrespondents.
improving your response rate
Improving your response rate
  • Increase perceived rewards

say thank you, ask them for advice, make questionnaire interesting

  • Decrease perceived costs

avoid subordinating language, don’t embarrass the respondent, make it quick and easy to respond

  • Establishing trust

make the questionnaire seem important, provide a sense of legitimate authority, provide a token

design write the questionnaire
Design & write the questionnaire
  • As simple, as short as possible
  • Unambiguous items
  • No leading questions
  • No double-barreled questions
  • Questions follow a logical sequence
  • Put personal or confidential questions at the end.
layout design
Layout & Design
  • Clear introductory statement & contact information
  • KISS & lessen clutter
  • Number items and be consistent with wording, font (serif font like Times easier to read than a san serif font).
  • Use italics and bold judiciously and with a purpose.
  • Careful with color & limit graphics
particularly for web surveys
Particularly for web surveys…
  • Design the survey as multiple pages, with a submit button at end of each section (not one long page)
  • Use a progress button.
  • Decide if respondents can exit and re-enter survey.
  • Design so that it takes no longer than 20 minutes to complete the survey.
question order
Question order
  • Begin with questions that raise interest but are easy to answer
  • Then ask questions of interest that are harder to answer
  • Personal questions at end after respondent has committed to answering (though completion of web surveys is an issue, so maybe move some to middle)
  • Make logical groupings
types of questions
Types of questions
  • Open-ended vs. closed-ended
    • CE = easy to standardize, analyze
    • OE = easier to write, draw in respondent
  • Single vs. multiple response
    • Choose one; choose all that apply
  • Ranked responses
    • Indicate order of importance of a list
  • Rated responses
    • Likert scales
writing tips
Writing tips
  • Avoid double barreled questions
    • Do you think college students today should eat less and exercise more?
  • Avoid questions with negatives
    • Are you against a ban on smoking?
  • Ask for precise answers
    • Your age on January 1, 2009
more writing tips response scales
More writing tips – Response scales

--allow for maximum variability (no more than 10; less than h.s. educ., 5)

--use a balanced scale

--“neutral” or “no opinion” vs. “don’t know” (odd/even number of responses)

--Order response categories in a logical way

pilot test the questionnaire
Pilot test the questionnaire
  • Field pre-testing
  • Focus group discussions
  • Individual interviews – “think aloud”
  • Behavior coding

does the respondent ask for clarification?

How long does it take to answer each question?

distribute the questionnaire
Distribute the questionnaire
  • After revision from pilot test, distribute the questionnaire
  • Document the how, where, when
  • Document response times, nonrespondents
  • Follow up nonrespondents
  • So, distribution is seen as a longitudinal process
analyze the results write the report
Analyze the results & write the report
  • think in terms of useful information directed toward change, rather than data
  • Clear presentation – not excessive pretty charts and graphs
in summary
In summary . . .
  • Be mindful that you are using someone else’s time for your survey
  • Think of the survey process as a process, not just an instrument
  • Be careful of extending your results to make them mean more than they do
references
References

Dillman, D. A. (1978). Mail and telephone surveys: The total design method. NY: Wiley.

Porter, S. (2004). Pros and cons of paper and electronic surveys, overcoming survey research problems new Directions for Institutional Research, 121, 91-99.

Frary, R. B. (1996). Brief guide to questionnaire development. Washington, DC: ERIC Clearinghouse on Assessment and evaluation.

helpful websites
Helpful websites
  • Resources for program evaluation and social research methods http://gsociology.icaap.org/methods/
  • Guide to the design of questionnaires http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires
questions comments
Questions/Comments?

Please take a few minutes to

complete the session feedback

Form.

Thank you!