questionnaire development n.
Skip this Video
Loading SlideShow in 5 Seconds..
Questionnaire Development PowerPoint Presentation
Download Presentation
Questionnaire Development

Loading in 2 Seconds...

play fullscreen
1 / 59

Questionnaire Development - PowerPoint PPT Presentation

  • Uploaded on

Questionnaire Development. Health Survey Research Methods Susan Sherman December 6, 2010. Lecture Objectives. Learn about the sampling universe, sample, and sampling methodology Random Nonrandom Describe strengths and weaknesses of different approaches.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Questionnaire Development' - zudora

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
questionnaire development

Questionnaire Development

Health Survey Research Methods

Susan Sherman

December 6, 2010

lecture objectives
Lecture Objectives
  • Learn about the sampling universe, sample, and sampling methodology
    • Random
    • Nonrandom
  • Describe strengths and weaknesses of different approaches.
  • To conceptualize, operationalize, and specify research questions;
learning objectives
Learning Objectives

4. To understand how questions, responses, instructions and the questionnaire can effect the meaning of a question.

5. To recognize and revise unbalanced, loaded, or double-barreled questions.

6. Hear about the block

study population
Study Population
  • The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated
    • Members of the population are elements
  • Often cannot sample elements directly (not available, too expensive), but are associated with other units, enumeration units or listing units.
why sample
Why sample?
  • Economy!
  • No need to determine all possible responses
  • Can use probability and statistics to assist making an informed judgment
  • Sampling frames are out of date as soon as they are developed
the population and the sample
The population and the sample
  • The population (universe, target population) is the entire set of individuals to which findings of the survey are to be extrapolated
  • Members of the population are elements
  • Often cannot sample elements directly (not available, too expensive), but are associated with other units, enumeration units or listing units.
the sample
The sample

Probability and non-probability sampling

  • Probability sample: every element in the population has a known, nonzero probability of being included in the sample
  • Non-probability sampling does not have this feature, but is commonly used in market research and public opinion polls (time, expense, not feasible) – quota surveys
sampling frame
Sampling frame
  • Reliance on known probability of being selected (vs. marketing research)
  • Provides means of identifying and locating population elements.
    • Often contains additional information that can be used for stratification and clustering
    • Organization of frame exerts strong influence over sample design.
sampling frame1
Sampling frame
  • “Ideal frame” lists each population element once and contains no other listings (rare)
  • Kish’s classification of possible frame problems:
    • Missing elements (not in the frame)
    • Clusters
    • Blanks and foreign elements
    • Duplicate listings
problems constructing sampling frames
Problems constructing sampling frames
  • Household information may be dated
  • New households may have been added
  • Older houses may have been demolished
  • Block listings date quickly
  • Organizational lists may be fraught with missing information
  • Clustering
  • Substitution
forms of probability sampling
Forms of Probability Sampling
  • Simple random sampling (rarely done)
  • Systematic sampling
  • Stratified sampling
  • Cluster Sampling
  • Serpentine fashion order
unknown source population
Unknown Source Population
  • Absence of a sampling frame; unknown boundaries and size of target population
  • Privacy concerns; illegal or stigmatized activities

i.e., Hidden populations

    • Men who have sex with men (MSM)
    • Injection drug users (IDU)
    • Commercial sex workers
    • Migrant workers
  • Relatively small groups
sampling methods
Sampling Methods
  • Facility-Based
  • Snowball
  • Targeted
  • Time-Location (TLS) or Venue-Based (VBS)
  • Respondent-Driven (RDS)
facility based sampling
Facility-Based Sampling
  • Sample of clients from facilities serving the target population
  • Examples:
    • Jails/Prisons
    • STI clinics
    • Drug Treatment Centers
  • Biased sample based on service seeking
  • Considered convenience sample
snowball sampling
Snowball Sampling
  • “Random” selection of “seeds”
  • Seeds refer others with outcome/exposure of interest
  • Endpoint: sample size or sample saturation
  • Considered convenience sample
  • Example: ALIVE Study looking at the natural history of HIV/AIDS among IDU
targeted sampling
Targeted Sampling
  • Formative research to identify networks of outcome/exposure of interest
  • Different networks treated as sampling strata
  • Systematic sampling within strata
  • Practically treated as a convenience sample
  • Heavily dependent on extensive formative research
venue based sampling vbs
Venue-Based Sampling (VBS)
  • Sampling of physical venues attended by target population
  • Formative research identifies public/private venues and days/times of attendance
  • Venue-Day-Times (VDT) enumerated for eligibility and viability
  • Sampling frame consists of VDTs; random selection of VDTs to construct sampling event calendar
  • Individuals systematically recruited at sampling events
respondent driven sampling rds
Respondent-Driven Sampling (RDS)
  • Type of chain referral sampling to reach hidden populations (Markov chain)
  • Begin with a set of non randomly selected seeds
  • Seeds recruit peers, who recruit peers, etc.
  • Recruits are linked by coupons with unique identifying numbers
  • Recruitment quota through coupons
  • Incentives provided for completed survey and for each successful recruit

Heckathorn 1997; Heckathorn & Salganik, 2004; Broadhead et al. 1998

bias systematic error
Bias (systematic error)
  • Sampling and non-sampling biases
  • Sampling biases come from the sampling processes themselves or from the statistical estimation process
  • Frame biases are the most problematic
    • Inappropriate selection procedure
    • Elements appear > 1 time
    • Non-random ordering
non sampling biases
Non-sampling biases
  • Account for largest source of total survey error, most often ignore, unappreciated
  • Observational biases: caused by obtaining and recording observations incorrectly
    • Field errors (data collection, enumeration, measurement)
    • Processing and data analysis errors
overall conclusions
Overall Conclusions
  • Sampling methods have improved ability to arrive at valid inferences
  • Each method has to be considered and applied based on objectives and target population
  • Formative research is vital for implementation and interpretation
  • These active surveillance and research efforts can greatly supplement and enhance passive surveillance
  • Infrastructures can be used for prevention efforts
to start what are your goals
To start: what are your goals?

Research Question: A statement that identifies the phenomenon to be studied.

  • What are the units or entities being studied?
  • What variables will be compared across those units?
  • What relationships do you want to examine?
what relationships do you want to examine
What relationships do you want to examine?
  • Associative
    • correlate
  • Causal
    • question a direction
    • temporality
  • Mediating

To determine if the relationship between exercise and obesity varies by race/ethnicity among 15-24 year olds in Baltimore, MD.

  • What are the units?
  • What are the variables of interest?
  • What are the relationships between variables of interest?
measurement process
Measurement Process

Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties.

(Singleton, ‘93)

3 steps…

Conceptual Definition

Operational Definition

Variable Definition

1 conceptual definition
1. Conceptual Definition

Process of formulating and clarifying concepts of interest

Refines problem statements or hypotheses which can be vague

Example: Obesity

2 operational definition
2. Operational Definition

Questions asked to obtain information on concept or issue


1. Do you consider yourself overweight, underweight, or just about right?

2. a. About how tall are you without shoes?

b. About how much do you weigh without shoes?

measurement process1
Measurement Process

Measurement is the process of assigning numbers or labels to units of analysis in order to represent conceptual properties.

(Singleton, ‘93)

3 variable definition
3. Variable Definition

Variable constructed from questions to be used in the analysis of the data.


1. Obesity:

1 = overweight

2 = underweight

3 = about right

2. Obesity: Construct index of obesity based on BMI, calculated as weight divided by height squared.

note of caution
Note of Caution

No indicator can perfectly represent a single concept.

No two indicators measure a single concept exactly the same.

new vs existing questions
New vs. Existing Questions

Pros of Existing: enhances quality and applicability of items, enables comparisons across studies


  • Tested for validity and reliability
  • Evidence of methodological problems—missing values
  • How questionnaire was delivered (self administered, ACASI, etc)
  • Study population
  • Social changes (time period)
four elements that effect meaning
Four elements that effect meaning
  • Questions
  • Responses
  • Instructions
  • Questionnaires
1 questions
1. Questions
  • Wording
      • Unclear word choice
  • Phrasing
      • Unbalanced question
      • Loaded question
      • Double barreled question
  • Sentence
      • Wordiness (short vs. long)
  • Question
      • Irrelevant to population or research question
1a wording
1a. Wording
  • Avoid jargon
  • Avoid acronyms
  • Be precise
  • Ensure there is adequate knowledge
1b phrasing
1b. Phrasing
  • Unbalanced Question
  • Double-barreled Question
  • Loaded Question
unbalanced question
Unbalanced Question

Definition: Both sides of a question are not adequately represented.

Example: Do you agree that medical marijuana is bad?

loaded question
Loaded Question

Definition: A question that encourages participants to respond to the question in a certain way

Example: There are many people who believe that medicinal marijuana should e available. Are you one of them?

Revised: Medicinal marijuana has positive medical properties. [strongly agree… strongly disagree]

double barreled question
Double-barreled Question

Definition: A question that has more than one question embedded within it. A red flag is the word “AND”

Example: Do you agree that medical marijuana should be legal and that you would vote for it on a ballet?

1c sentences
1c. Sentences

Avoid wordiness and confusing sentence structure.

Example: Do you believe that the parking situation on campus is problematic or difficult because of the lack of spaces and the walking distances or do you believe that the parking situation on campus is ok?

1d question
1d. Question

Be sure that questions directly relate to research questions.

Example (for research on parking): Do you like or dislike the bus system?

2 responses
2. Responses
  • Open ended
    • comprehensiveness
  • Close ended (rating, ranking)
    • Should be completely exhaustive
    • Should be mutually exclusive
open ended vs closed ended

Enables participant to talk about what comes to mind first

Respondents can provide a comprehensive and diverse array of answers

Difficult to code and process


Dependent on structure of responses: some categories may be inadvertently omitted

May be effected by # and type of response categories, presence of a neutral, “don’t know” category

Easier to code and may be more reliable across respondents and interviewers

Open-ended vs. Closed-ended
3 instructions
3. Instructions


  • to ensure that the question or questionnaire is answered in the way that it should be.


  • As part of question itself
  • To introduce or close questionnaire
  • To make meaningful transitions between topics
  • To guide respondent/interviewer on skip patterns


  • parentheses, all capital letters or some other type face
to introduce or close questionnaire
To introduce or close questionnaire

We are now done with the questionnaire. If you have any questions about this interview, please feel free to ask me now. [PAUSE FOR QUESTIONS]. If there are no questions, I'd like to thank you for participating in this interview.

to make meaningful transitions between topics
To make meaningful transitions between topics

The next set of questions is about using drugs. Please remember that your answers are strictly confidential. Your name is not on this form. No one can trace these answers back to you. If you do not want to answer certain questions you don’t have to, but please answer all the questions that you can.

4 questionnaire
4. Questionnaire

The order and context in which items are placed has an impact on the meaning of certain questions and how respondents answer them.

sequencing of questions
Sequencing of Questions
  • Begin with behavioral questions about the present and then the past.
  • Start with questions that make them feel comfortable.
  • Place sensitive questions in the middle.
categories of health questions
Categories of Health Questions
  • Socio-demographics: e.g., age
  • Behavior: e.g., condom use
  • Knowledge: e.g., HIV transmission,
  • Attitudes: e.g., towards HIV positive individuals
1 socio demographics
1. Socio-demographics
  • Use of standardized measures:
    • Reduce time and effort
    • Permit direct comparison
    • May have documented validity and reliability
  • Adapt US Census, federally sponsored surveys
2 behaviors
2. Behaviors
  • Prone to over and under reporting due to recall errors
  • Recall errors are a function of:
    • time period over which events are to be remembered
    • Salience or significance of event to person
minimizing recall bias memory aid procedures
Minimizing Recall Bias: Memory Aid Procedures
  • Aided recall: Clues provided in question
  • Records: Ask respondents to consult personal records, e.g. checkbooks, doctor or hospital bills
  • Diaries: Respondents use calendars to record relevant events; respondent consults diary when interviewed about these events
3 4 knowledge and attitudes
3 -4. Knowledge and Attitudes
  • Knowledge: more objective b/c a “right” answer is presumed to exist
  • Opinion: refers to views about particular objects such as a person or policy
  • Attitude: Bundle of opinions that are more or less coherent and are about some complex object
minimize threat knowledge questions
Minimize Threat: Knowledge Questions
  • Threat stems from fear of being perceived as ignorant
  • To minimize threat, phrase questions as opinions (e.g., “do you think…”)
  • To accurately capture knowledge:
    • Ask more than one question
    • Keep responses open ended
rules to write by
Rules to Write By
  • Be mindful of 4 elements that effect the meaning of a question: question, response, instructions, questionnaire
  • Write in everyday terms
  • Avoid unbalanced questions
  • Avoid loaded questions
  • Avoid double-barreled questions
  • Write short, simple questions
  • Ensure questions are relevant to research question
  • Minimize social desirability bias
  • Minimize recall bias