Questionnaire Development

1. Questionnaire Development Health Survey Research Methods Susan Sherman December 6, 2010

2. Lecture Objectives • Learn about the sampling universe, sample, and sampling methodology • Random • Nonrandom • Describe strengths and weaknesses of different approaches. • To conceptualize, operationalize, and specify research questions;

3. Learning Objectives 4. To understand how questions, responses, instructions and the questionnaire can effect the meaning of a question. 5. To recognize and revise unbalanced, loaded, or double-barreled questions. 6. Hear about the block

4. Study Population • The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated • Members of the population are elements • Often cannot sample elements directly (not available, too expensive), but are associated with other units, enumeration units or listing units.

5. Sampling

6. Why sample? • Economy! • No need to determine all possible responses • Can use probability and statistics to assist making an informed judgment • Sampling frames are out of date as soon as they are developed

7. The population and the sample • The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated • Members of the population are elements • Often cannot sample elements directly (not available, too expensive), but are associated with other units, enumeration units or listing units.

8. The sample Probability and non-probability sampling • Probability sample: every element in the population has a known, nonzero probability of being included in the sample • Non-probability sampling does not have this feature, but is commonly used in market research and public opinion polls (time, expense, not feasible) – quota surveys

9. Sampling frame • Reliance on known probability of being selected (vs. marketing research) • Provides means of identifying and locating population elements. • Often contains additional information that can be used for stratification and clustering • Organization of frame exerts strong influence over sample design.

10. Sampling frame • “Ideal frame” lists each population element once and contains no other listings (rare) • Kish’s classification of possible frame problems: • Missing elements (not in the frame) • Clusters • Blanks and foreign elements • Duplicate listings

11. Problems constructing sampling frames • Household information may be dated • New households may have been added • Older houses may have been demolished • Block listings date quickly • Organizational lists may be fraught with missing information • Clustering • Substitution

12. Forms of Probability Sampling • Simple random sampling (rarely done) • Systematic sampling • Stratified sampling • Cluster Sampling • Serpentine fashion order

13. Unknown Source Population • Absence of a sampling frame; unknown boundaries and size of target population • Privacy concerns; illegal or stigmatized activities i.e., Hidden populations • Men who have sex with men (MSM) • Injection drug users (IDU) • Commercial sex workers • Migrant workers • Relatively small groups

14. Sampling Methods • Facility-Based • Snowball • Targeted • Time-Location (TLS) or Venue-Based (VBS) • Respondent-Driven (RDS)

15. Facility-Based Sampling • Sample of clients from facilities serving the target population • Examples: • Jails/Prisons • STI clinics • Drug Treatment Centers • Biased sample based on service seeking • Considered convenience sample

16. Snowball Sampling • “Random” selection of “seeds” • Seeds refer others with outcome/exposure of interest • Endpoint: sample size or sample saturation • Considered convenience sample • Example: ALIVE Study looking at the natural history of HIV/AIDS among IDU

17. Targeted Sampling • Formative research to identify networks of outcome/exposure of interest • Different networks treated as sampling strata • Systematic sampling within strata • Practically treated as a convenience sample • Heavily dependent on extensive formative research

18. Venue-Based Sampling (VBS) • Sampling of physical venues attended by target population • Formative research identifies public/private venues and days/times of attendance • Venue-Day-Times (VDT) enumerated for eligibility and viability • Sampling frame consists of VDTs; random selection of VDTs to construct sampling event calendar • Individuals systematically recruited at sampling events

19. Respondent-Driven Sampling (RDS) • Type of chain referral sampling to reach hidden populations (Markov chain) • Begin with a set of non randomly selected seeds • Seeds recruit peers, who recruit peers, etc. • Recruits are linked by coupons with unique identifying numbers • Recruitment quota through coupons • Incentives provided for completed survey and for each successful recruit Heckathorn 1997; Heckathorn & Salganik, 2004; Broadhead et al. 1998

20. RDS Recruitment Network

21. Bias (systematic error) • Sampling and non-sampling biases • Sampling biases come from the sampling processes themselves or from the statistical estimation process • Frame biases are the most problematic • Inappropriate selection procedure • Elements appear > 1 time • Non-random ordering

22. Non-sampling biases • Account for largest source of total survey error, most often ignore, unappreciated • Observational biases: caused by obtaining and recording observations incorrectly • Field errors (data collection, enumeration, measurement) • Processing and data analysis errors

23. Overall Conclusions • Sampling methods have improved ability to arrive at valid inferences • Each method has to be considered and applied based on objectives and target population • Formative research is vital for implementation and interpretation • These active surveillance and research efforts can greatly supplement and enhance passive surveillance • Infrastructures can be used for prevention efforts

24. Research Questions and measurement

25. To start: what are your goals? Research Question: A statement that identifies the phenomenon to be studied. • What are the units or entities being studied? • What variables will be compared across those units? • What relationships do you want to examine?

26. What relationships do you want to examine? • Associative • correlate • Causal • question a direction • temporality • Mediating

27. Example To determine if the relationship between exercise and obesity varies by race/ethnicity among 15-24 year olds in Baltimore, MD. • What are the units? • What are the variables of interest? • What are the relationships between variables of interest?

28. Measurement Process Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties. (Singleton, ‘93) 3 steps… Conceptual Definition Operational Definition Variable Definition

29. 1. Conceptual Definition Process of formulating and clarifying concepts of interest Refines problem statements or hypotheses which can be vague Example: Obesity

30. 2. Operational Definition Questions asked to obtain information on concept or issue Example: 1. Do you consider yourself overweight, underweight, or just about right? 2. a. About how tall are you without shoes? b. About how much do you weigh without shoes?

31. Measurement Process Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties. (Singleton, ‘93)

32. 3. Variable Definition Variable constructed from questions to be used in the analysis of the data. Example: 1. Obesity: 1 = overweight 2 = underweight 3 = about right 2. Obesity: Construct index of obesity based on BMI, calculated as weight divided by height squared.

33. Note of Caution No indicator can perfectly represent a single concept. No two indicators measure a single concept exactly the same.

34. Developing survey questions

35. New vs. Existing Questions Pros of Existing: enhances quality and applicability of items, enables comparisons across studies Consider: • Tested for validity and reliability • Evidence of methodological problems—missing values • How questionnaire was delivered (self administered, ACASI, etc) • Study population • Social changes (time period)

36. Four elements that effect meaning • Questions • Responses • Instructions • Questionnaires

37. 1. Questions • Wording • Unclear word choice • Phrasing • Unbalanced question • Loaded question • Double barreled question • Sentence • Wordiness (short vs. long) • Question • Irrelevant to population or research question

38. 1a. Wording • Avoid jargon • Avoid acronyms • Be precise • Ensure there is adequate knowledge

39. 1b. Phrasing • Unbalanced Question • Double-barreled Question • Loaded Question

40. Unbalanced Question Definition: Both sides of a question are not adequately represented. Example: Do you agree that medical marijuana is bad?

41. Loaded Question Definition: A question that encourages participants to respond to the question in a certain way Example: There are many people who believe that medicinal marijuana should e available. Are you one of them? Revised: Medicinal marijuana has positive medical properties. [strongly agree… strongly disagree]

42. Double-barreled Question Definition: A question that has more than one question embedded within it. A red flag is the word “AND” Example: Do you agree that medical marijuana should be legal and that you would vote for it on a ballet?

43. 1c. Sentences Avoid wordiness and confusing sentence structure. Example: Do you believe that the parking situation on campus is problematic or difficult because of the lack of spaces and the walking distances or do you believe that the parking situation on campus is ok?

44. 1d. Question Be sure that questions directly relate to research questions. Example (for research on parking): Do you like or dislike the bus system?

45. 2. Responses • Open ended • comprehensiveness • Close ended (rating, ranking) • Should be completely exhaustive • Should be mutually exclusive

46. Open-ended Enables participant to talk about what comes to mind first Respondents can provide a comprehensive and diverse array of answers Difficult to code and process Closed-ended Dependent on structure of responses: some categories may be inadvertently omitted May be effected by # and type of response categories, presence of a neutral, “don’t know” category Easier to code and may be more reliable across respondents and interviewers Open-ended vs. Closed-ended

47. 3. Instructions Purpose • to ensure that the question or questionnaire is answered in the way that it should be. Usage • As part of question itself • To introduce or close questionnaire • To make meaningful transitions between topics • To guide respondent/interviewer on skip patterns Designation • parentheses, all capital letters or some other type face

48. To introduce or close questionnaire We are now done with the questionnaire. If you have any questions about this interview, please feel free to ask me now. [PAUSE FOR QUESTIONS]. If there are no questions, I'd like to thank you for participating in this interview.

49. To make meaningful transitions between topics The next set of questions is about using drugs. Please remember that your answers are strictly confidential. Your name is not on this form. No one can trace these answers back to you. If you do not want to answer certain questions you don’t have to, but please answer all the questions that you can.

50. 4. Questionnaire The order and context in which items are placed has an impact on the meaning of certain questions and how respondents answer them.