Developing Effective Surveys

1. Developing Effective Surveys Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) sculver@vt.edu September, 2009

2. Today’s Agenda • Discuss survey design as a process • Provide tips to enhance the quality of the process and therefore the data

3. Definitions • A survey is a data-gathering and analysis approach in which respondents answer questions or respond to statements developed in advance. • IRB – Institutional Review Board must approve “research”

4. Types of surveys • Face to face (usually most expensive) more control, sometimes can collect more data • Telephone fairly controlled setting, can ask for clarification, but phone screening, fewer hours at home • Mail (takes longer) Respondent can complete at leisure, can include pictures; no control over who completes survey, potential illegible data; bias - literacy • Web (e-mail survey or survey posted on web) more consistent (reliable) than human interviewer, but no human element to identify problems, no control over data errors; respondents give longer answers to open-ended questions.

5. Steps in the process • Identify research objective • Identify & characterize the target audiences • Design the sampling plan • Design & write the questionnaire • Pilot test the questionnaire • Distribute the questionnaire • Analyze the results & write the report

6. Identify research objective • Define precisely the information desired • Try to write as few questions as possible to obtain that information • A clear-cut need for every question should be established

7. Identify & characterize the target audiences • Is it a wide range of people or a niche group? • What is there interest in the topic? • What is their reading/educational level? • Are there other important characteristics to consider that relate to your study? (how would you know?)

8. Design the sampling plan • Address how a sample of people should be selected to meet the study objectives and to obtain reliable information. • Need to identify the accessible population. • Sampling methods: random, systematic, stratified • Follow up with the nonrespondents.

9. Improving your response rate • Increase perceived rewards say thank you, ask them for advice, make questionnaire interesting • Decrease perceived costs avoid subordinating language, don’t embarrass the respondent, make it quick and easy to respond • Establishing trust make the questionnaire seem important, provide a sense of legitimate authority, provide a token

10. Design & write the questionnaire • As simple, as short as possible • Unambiguous items • No leading questions • No double-barreled questions • Questions follow a logical sequence • Put personal or confidential questions at the end.

11. Layout & Design • Clear introductory statement & contact information • KISS & lessen clutter • Number items and be consistent with wording, font (serif font like Times easier to read than a san serif font). • Use italics and bold judiciously and with a purpose. • Careful with color & limit graphics

12. Particularly for web surveys… • Design the survey as multiple pages, with a submit button at end of each section (not one long page) • Use a progress button. • Decide if respondents can exit and re-enter survey. • Design so that it takes no longer than 20 minutes to complete the survey.

13. Question order • Begin with questions that raise interest but are easy to answer • Then ask questions of interest that are harder to answer • Personal questions at end after respondent has committed to answering (though completion of web surveys is an issue, so maybe move some to middle) • Make logical groupings

14. Types of questions • Open-ended vs. closed-ended • CE = easy to standardize, analyze • OE = easier to write, draw in respondent • Single vs. multiple response • Choose one; choose all that apply • Ranked responses • Indicate order of importance of a list • Rated responses • Likert scales

15. Writing tips • Avoid double barreled questions • Do you think college students today should eat less and exercise more? • Avoid questions with negatives • Are you against a ban on smoking? • Ask for precise answers • Your age on January 1, 2009

16. More writing tips – Response scales --allow for maximum variability (no more than 10; less than h.s. educ., 5) --use a balanced scale --“neutral” or “no opinion” vs. “don’t know” (odd/even number of responses) --Order response categories in a logical way

17. Pilot test the questionnaire • Field pre-testing • Focus group discussions • Individual interviews – “think aloud” • Behavior coding does the respondent ask for clarification? How long does it take to answer each question?

18. Distribute the questionnaire • After revision from pilot test, distribute the questionnaire • Document the how, where, when • Document response times, nonrespondents • Follow up nonrespondents • So, distribution is seen as a longitudinal process

19. Analyze the results & write the report • think in terms of useful information directed toward change, rather than data • Clear presentation – not excessive pretty charts and graphs

20. In summary . . . • Be mindful that you are using someone else’s time for your survey • Think of the survey process as a process, not just an instrument • Be careful of extending your results to make them mean more than they do

21. References Dillman, D. A. (1978). Mail and telephone surveys: The total design method. NY: Wiley. Porter, S. (2004). Pros and cons of paper and electronic surveys, overcoming survey research problems new Directions for Institutional Research, 121, 91-99. Frary, R. B. (1996). Brief guide to questionnaire development. Washington, DC: ERIC Clearinghouse on Assessment and evaluation.

22. Helpful websites • Resources for program evaluation and social research methods http://gsociology.icaap.org/methods/ • Guide to the design of questionnaires http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires

23. Questions/Comments? Please take a few minutes to complete the session feedback Form. Thank you!