ARL Workshop on User Surveys

1. ___________________________ARL Workshop on UserSurveys Measurement Scales and Types of Questions Brinley Franklin

3. ___________________________Importance Of This Section Reliable and valid survey questions are a cornerstone of effective survey research Done correctly, questions can produce valuable information Done incorrectly, questions can gather inappropriate information, introduce bias, or even invalidate the user survey

4. ___________________________Topics To Be Covered The effects of attitudes, behavior, knowledge, and demographics on question construction Understanding the survey�s context Guidelines for asking survey questions Open or closed questions? Response choices/measurement scales

5. ___________________________The Effects of Attitudes, Behavior, Knowledge and Demographics onQuestion Construction Attitudes Behavior Knowledge Questions Demographic Considerations

6. __________________________Attitudes Attitudes are complex and are difficult to define and measure Attitude questions often have two components: how respondents feel; and how strongly they feel

7. ___________________________Behavior Questions refer to what respondents actually do Generally concerned with identification, quantity, time, duration, or frequency (what, how many, when, how long, how often) Asking specific questions about a long period of time yields imprecise answers Lists help respondents recall things they might have done

8. ___________________________Knowledge Questions Can determine if respondents know enough about a topic to warrant asking their opinion May identify gaps in knowledge that need to be addressed Sometimes help to explain attitudes and behavior

9. ___________________________Demographic Considerations Knowing something about the survey�s targeted respondents can help frame the questions and determine an appropriate level of specificity If specific responses are sought and the range of answers is large, ask for the exact information; don�t present a multiple choice question Use current words and terms Decide if comparability is sought. If so, standardize questions

10. ___________________________Understanding the Survey�s Context Focus the questions on the survey�s specific purpose(s) or information sought Agree upon the precise data being sought in advance Clarify or define all potentially imprecise or ambiguous terms Terminology, language level, and time frame should be appropriate for respondents Stick with the least number of questions needed for adequate measurement

11. ___________________________Understanding the Survey�s Context (Continued) Standardize survey(or)s Use anonymous surveys to ask sensitive questions about personal behaviors or attitudes

12. ___________________________Guidelines For Asking Survey Questions Ask concrete questions that are precise and unambiguous use time periods that are related to the importance of the question Use conventional grammar, punctuation, and spelling Use the proper context when constructing survey questions

13. ___________________________Using Conventional Grammar, Punctuation, and Spelling Use complete sentences Avoid abbreviations, slang and colloquialisms Avoid jargon and technical expressions Have the questions reviewed by experts in survey question writing Have the questions reviewed by potential respondents Adopt or adapt questions used successfully in other surveys

14. ___________________________Using Proper Context Shorter questions take less time and often lead to shorter answers Use longer questions with lead-ins to provide background information and collect data on sensitive concerns (e.g., journal cancellations) Use loaded questions cautiously Avoid biasing words or phrases Avoid two-edged questions (avoid �and�) Avoid negative questions or ones that require an exercise in logic

15. ___________________________Open Or Closed Questions? Open questions are used to solicit answers in the respondent�s own words Closed questions, where responses are pre-selected, are considered more efficient and reliable Use a checklist to decide between open and closed questions

16. ___________________________Open Questions Are useful in getting unanticipated answers Describe the respondent�s, not the surveyor�s world Provide quotable material Are primarily used to make decisions about individuals, not groups may be difficult to compile, compare and interpret

17. ___________________________Closed Questions Provide standardized data that can be analyzed statistically Answers have a better chance of relevance and consistency over time Are more difficult to write and validate May overlook things respondents might volunteer in open questions

18. ___________________________Checklist For Deciding Between Open And Closed Questions

19. ___________________________Response Choice/Measurement Scales Nominal/Categorical response choices Ordinal response choices Numerical response choices

20. ___________________________Nominal/Categorical ResponseChoices Are not numerical or preferential values Often used to categorize respondents� characteristics Findings are usually reported as percentage groupings

21. ___________________________Suggestions For Constructing Nominal/Categorical Responses Generally try to be exhaustive Try to make choices mutually exclusive Try to use categories that are meaningful to respondents and will make sense when results are reported

22. ___________________________Ordinal Response Choices Respondents are asked to rate or rank choices Are often used more than the other types of questions Are typically used to measure satisfaction Survey findings are usually reported as an average score or percentage

23. ___________________________Suggestions For Constructing Ordinal Responses Consider five types of response options: Endorsement/Frequency/Intensity/Influence/ Comparison Determine through pre-testing if a neutral response is called for Use a 5point to 7point scale (either odd or even) Know when to put the negative end of the scale first

24. __________________________Numerical Response Choices Ask respondents to quantify a response Provide completely objective response Allow many statistical analyses to be performed and reported

25. ___________________________Suggestions For Constructing Numerical Responses Decide if discrete (i.e. exact) numbers work best Consider using intervals if precision is not needed Decide if it is most appropriate to offer numerical responses along a continuum

26. ___________________________Conclusion - You Should Now Have a Better Understanding Of: Why survey questions are important Characteristics of the user population that influence the question asked The context for framing survey questions Guidelines for asking survey questions The difference between open and closed questions, and when to use each type The three types of response choices: nominal/categorical; ordinal; and numerical

27. ___________________________Related Readings Babbie, Earl R. Survey research methods. Belmont, CA: Wadsworth Press, 1990. Bradburn, Norman M. and Sudman, Seymour. �The current status of questionnaire design�. In Paul P. Biemer et al, Measurement errors in surveys. New York: John Wiley, 1992. Converse, Jean M. Survey questions: handcrafting the standardized questionnaire. Beverly Hills, CA: Sage Publications, 1986. Fink, Arlene and Kosecoff, Jacqueline. How to ask survey questions. Volume 2 of The survey kit. Thousand Oaks, CA: Sage Publications, 1985. Fowler, Floyd J. Survey research methods. Newbury Park, CA: Sage Publications, 1993. Kosecoff, Jacqueline and Fink, Arlene. Evaluation basics: a practitioner�s manual. Beverly Hills, CA: Sage Publications, 1982. Schuman, Howard and Presser, Stanley. Question and answers in attitude surveys. New York: Academic Press, 1981. Sudman, Seymour and Bradburn, Norman M. Asking questions. San Francisco, CA: Jossey Bass, 1982.