Research Methods

Research Methods Survey Research

Outline: Survey Research • I. Sampling • II. Survey Contents

I. Sampling from a Population “Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” • Population - complete set of individuals having some common characteristic – e.g., Australians • Sampling frame – subset of the population from which the sample is actually drawn – e.g., White pages • Sample – the set of people included in the study (i.e., selected from the sampling frame) – e.g., Every 1000th person in the white pages

Population and Sample Population Sample Use statistics to summarize features Use parameters to summarize features Inference on the population from the sample

Confidence Intervals • The problem is to use the sample to make inferences about the population • If we were to repeat the entire process of drawing a sample and computing the statistic many, many times, we would find that the statistic varies some • A confidence interval takes advantage of those variations and allows us to specify a range that probably contains the true value of the parameter • Technically, for an X% confidence interval, the parameter will fall inside the range X% of the time in repetitions of the study • Confidence intervals usually vary between 90% and 99.9%

Define a Population • The first step in choosing a sample is to define the population, or the overall set of cases that we’re interested in • If we want to predict the results of an election, then our population is “likely voters” • If we want to understand who decides to vote, then our population is “citizens eligible to vote” • If we want to understand who decides to use marijuana in the United States, our population is probably “everyone in the United States”

Types of Samples Used • Nonprobability Sample • Items included are chosen without regard to their probability of occurrence • Probability Sample • Items in the sample are chosen on the basis of known probabilities

Types of Sampling Methods Samples Probability Samples Non-Probability Samples Simple Random Stratified Convenience Quota Cluster Systematic

Probability Sampling • Subjects of the sample are chosen based on known probabilities Probability Samples Simple Random Systematic Stratified Cluster

Simple Random Samples • Every individual or item from the frame has an equal chance of being selected • Selection may be with replacement or without replacement • Samples obtained from table of random numbers or computer random number generators

N = 64 n = 8 k = 8 First Group Systematic Samples • Decide on sample size: n • Divide frame of N individuals into groups of k individuals: k=N/n • Randomly select one individual from the 1st group • Select every k-th individual thereafter

Stratified Samples • Population divided into two or more groups according to some common characteristic • Simple random sample selected from each group • The two or more samples are combined into one

Stratified Sampling Example Population Cash holdings of All Financial Institutions in the United States Financial Institutions Stratified Population Large Institutions Medium Size Institutions Small Institutions Stratum 1 Select n1 Stratum 2 Select n2 Stratum 3 Select n3

Cluster Samples • Population divided into several “clusters,” each representative of the population • Randomly select certain clusters • The samples are combined into one Population divided into 4 clusters.

Algeria California Alaska New York Idaho Mexico Australia 25 105 20 36 152 76 37 Cluster Sampling Example Mid-Level Managers by Location for Morrison-Knudsen Construction Company Illinois Scotland Florida 42 22 52

Cluster Sampling Example Mid-Level Managers by Location for Morrison-Knudsen Construction Company Illinois Scotland Florida 42 22 52 All members selected from these clusters

Types of Survey Errors • Coverage error • Non response error • Sampling error • Measurement error Excluded from frame. Follow up on non responses. Chance differences from sample to sample. Bad Question!

II. Survey Contents • Remember that your respondents will be “Lazy Thinkers.”

Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover

Open-ended A survey question to which the respondent replies in his or her own word, either by writing or by talking difficult to analyze, subjective analysis time consuming rich information useful for descriptive, exploratory work Closed-ended or fixed-choice A survey question that provides preformatted response choices for the respondent to circle, check, mark, etc. easier to analyze efficient useful for hypothesis testing important info may be lost forever Survey Contents: Type

Closed-ended Questions • Closed-ended or fixed-choice • Response options are limited • Yes/no • True/false • Multiple choice with an “other” option • Race: White, Black, Hispanic, Asian, Other • Likert Scales 1 2 3 4 5 Strongly Agree Neither Disagree Strongly Agree A nor D Disagree • Rating Scales • Ask respondents to rate something like a person, place, thing, idea, attitude, etc. on a numbered scale, often Likert scales • Semantic Differential a fifth grader is Bad Good Awful ⊲ ⊲ ⊲ ⊲ ⊳ ⊳ ⊳ ⊳ Nice

Survey Contents: Wording • Wording Clarity of questions and response options • Respondents should understand your questions • Pre-testing is a very effective way to see if they do • Examples of bad questions: • Do you hate taxes? • The answer will tell you little because our society has many different taxes, loves to complain about them, but thinks they are necessary • Do you hate getting up in the morning?

Survey Contents: Wording • Guidelines for good questions: • Be direct • Maintain simplicity • Be specific • Take the role of your respondent The following points elaborate on these four main themes.

Survey Contents: Wording • No complex rhetoric, syntax, or disciplinary slang or jargon. • Do not expect them to learn new information just answer a question. (Sometimes, you may establish context with a short paragraph then ask a series of short, specific questions.) • Avoid phrasing questions to seem too personal or direct, especially when dealing with culturally sensitive issues. For example, “Do you abuse your kids?”

Survey Contents: Wording • Avoid Double-barreled questions. They contain two questions in one. For example, “Do you think that students and Professors should be given discounts on sports tickets?” • Avoid Double-negative questions. For example, “Do you disagree that professors should not be required to help students outside of class?” • Avoid hypothetical questions (unless you are studying hypothetical situations). You will not gain useful information from these questions. For example, “If men could have children, would your husband or boyfriend stay with you?”

Survey Contents: Wording • Avoid ambiguous questions. • “Do you teach your children to effectively function?” “Does your boss engage you in interactive dialogue?” “Effectively function” and “interactive dialogue” are subject to interpretation. • Words such as "usually" or "normally" mean different things to different people. “Do your customers normally complain?” has a variety of interpretations. • Avoid biased questions. Avoid making one response option look more suitable than the other, using emotionally loaded terms, or using unbalanced response categories. • “Don’t you think that suffering terminal cancer patients should be allowed to be released from their pain by choosing death?” • Virginia pays teachers more than similar states. Virginia should: spend more, keep spending the same, reduce spending little, reduce spending some, reduce spending a lot, dramatically reduce spending

Survey Contents: Order • Order Relative placement of questions in the instrument • Opening questions should be simple and introduce the topic of the survey. • Try not to mix topics. Put like things into sets of questions. • Avoid framing later questions with topics that can be linked to them in previous questions. For example, one should not ask about attitudes toward crack use right before asking about attitudes toward the urban poor. This will invoke stereotypes about the poor.

Survey Contents: Order • Sensitive questions should never be at the beginning. Put in middle. May need opening paragraph for them. • Consider need to transition between types of questions. Write transitory questions or directions. • Place your most important questions earlier—R’s may fatigue before answering them if they are later.

Survey Contents: Order • Do not put in too many complex format questions (e.g., “skipping” or “go to next section”). R’s will get confused/frustrated. • Closed-ended questions are easier to answer. Put open-ended questions later in survey. Investment of time up to that time will likely keep them through the open-ended.

Survey Contents: Content • Content Topics the questions cover • You should make the topic of the survey clear to R’s. Do not surprise them with questions about unrelated topics. • If including sensitive questions, provide a rationale for asking them. Explain how honest answers will be helpful to others. • Questions relevant to deviance should include normalizing statements. For example, “Many people use drugs for a variety of reasons. Have you ever used ecstasy (or x, e, MDMA) to feel closer to other people?”

Survey Contents: Content • The more sensitive the topic, the more reassurance of confidentiality should be stated. • Never open a survey with questions about sensitive topics, and do not end surveys with questions that will negatively linger in R’s minds. • Continually reassure R’s that there are no right or wrong answers to questions—just truthful or not. • Questions that require lots of specific details or a good memory are typically useless. If they are necessary, employ techniques to prompt recollection.

Survey Contents • General Advice • Always consult other surveys first • They will give you good ideas for exploring your topic • They may provide good examples of how to ask questions • You avoid “reinventing the wheel” • Especially useful for validated scales • Be sure that you have permission to use • Always pre-test your instrument

Survey Contents • Consider these issues: • Not everyone will have an opinion on every topic. Consider “no opinion” as an option. • People want to appear to agree. Consider negative and positive statements. • R’s may lose track and choose salient options (such as first in series). Keep questions simple. • Response set problem: In a series with the same response options, R’s may just quickly check the same response for all questions.

Survey Contents • Consider these issues: • Bad memory leads to: • Forward telescoping (reporting that events occurred more recently than in reality) • Backward telescoping (reporting events further back than in reality). • Salient events are overreported • Mundane events are underreported • “Habitual” events will fill in for lost information. Provide aides to recall such as reference points, landmark events, etc. Use limited time frames in questions.

Class Exercise • Split into groups • Select a topic for a survey • Develop 2 closed-ended and 2 open-ended questions. For each question, write a ‘poor’ version and a ‘good’ version • Administer the survey to member of another group. Critique each other’s questions

Research Methods