CH 6: The Survey

1. CH 6: The Survey (pp. 144-174)

2. WHAT IS A SOCIAL SURVEY? • Social surveys are the most widely used data-gathering technique in the social sciences and other fields • The survey technique gives us data based on self-reports • In a social survey, a large number of people (called respondents because they respond to a query) hear/read and provide answers to the exact same questions • Opinion polls are one type of survey, which ask about opinions on current issues • Besides opinions, surveys may ask about knowledge, social background and characteristics, general beliefs or behaviors

3. Survey data are considered correlational • Survey data are sometimes called correlational because they are strongest at meeting the 2nd condition of causality – association/correlation • correlational research: any non-experimental study in which correlations in data are examined and cause-effect relations are shown indirectly

4. Cause-effect explanations based on survey data • Cause-effect explanations based on survey data differ somewhat from other research techniques, such as the experiment • Recall from Ch. 2 that if you want to say that one variable causes another, you must meet 3 conditions • The IV must come before the DV in time • The 2 variables must be associated, or correlated with one another; and • No alternative cause for the relation exists, or there is no spuriousness spuriousness: when two variables appear to be causally connected but in reality, they are not because an unseen third factor is the true cause

5. Condition 1: time order? • Meeting the 1st condition – time order – is complicated in survey data collected at a single point in time • Without data at multiple time points, you must logically show that information from one survey question (e.g., experience of severe trauma) occurred earlier in time than information from another question (e.g., suicidal ideation)

6. Condition 3: no alternative cause? • To meet the 3rd condition, no alternative cause, you must think of variables that are possible alternative causes and measure them in a survey  control variables • control variable: a variable that represents a possible alternative explanation to the main hypothesis being tested; often control variables are measured in survey research with questions in addition to measures of the independent and dependent variables

7. HOW TO CONDUCT A SOCIAL SURVEY • 3 stages /6 steps • 1. Start-up stage – plan and prepare survey questionnaire (steps 1, 2, 3) • 2. Execution stage – collect and record data (step 4) • 3. Data analysis stage – analyze and interpret data and report final results (steps 5, 6)

8. Start-up • In this stage, you address the following 3 questions: • Who will be respondents? • What information do you want to learn from them? • How can you effectively get that information? -What will be your data collection instrument?

9. Start-up: Step 1 • Develop hypotheses • Decide on type of survey: • questionnaire: a fixed collection of questions used in a social survey that respondents answer • interview schedule: a questionnaire specifically designed for an interviewer asking respondents the questions • Face-to-face or by phone/online? • Write survey questions • Decide on response categories • Design layout

10. Start-up: Steps 2 and 3 • Step 2 • Plan how to record data • Pilot test the survey instrument • Step 3 • Decide on target population • Get sampling frame • Decide on sample size • Select sample

11. Execution: Step 4 • Locate respondents • Conduct interviews • Carefully record data

12. Data analysis: Steps 5 and 6 • Step 5 • Enter data into computers • Recheck all data • Perform statistical analysis on data • Step 6 • Describe methods and findings in research report • Present findings to others for critique and evaluation

13. WRITING GOOD SURVEY QUESTIONS • Two core principles: 1) Avoid confusion, e.g., double-barreled questions • double-barreled question: a confusing survey question that includes two or more 2) Keep the respondent’s perspective in mind • Dilemma: You want each respondent to hear the exact same question (b/c you want to measure the same thing) but if respondents have diverse backgrounds, the same wording may not have the same meaning for everyone.

14. What Are Leading Questions? • A good survey question is one in which respondents do not know which answer you expect, and they feel free to state what they really think or feel • leading question: a survey question worded such that respondents are pushed to a specific answer or position.

15. Some leading questions • “Stable gay and lesbian couples have the same rights as any other law-abiding citizens. Do you agree that your fellow citizens who are gay or lesbian have the same right to the benefits and responsibilities of a legal marriage as anyone else?” •  leading to overstate support • “Should we alter the law so that sexually deviant people get the privileges and benfits of a marriage, or should we uphold the natural basis of a marriage as between one man and one woman as it has been for centuries in custom, law and religious teachings?” •  leading to understate support

16. Should you use open or closed format? • open-ended question format:allowsrespondents to give any answer • closed-ended question format:respondents must choose among fixed answer choices

17. Writing Good Closed-Format Response Choices • Writing good answer choices is just as important as writing a good question. Answer choices should have 3 features: • Mutually exclusive: response categories don’t overlap • Exhaustive: each respondent has a choice • Balanced: favorable and unfavorable choices are equally available in set of responses • e.g., ‘outstanding, excellent, very good, and satisfactory’ is an unbalanced set of responses to rate performance

18. Should You offer a ‘Don’t Know’ or ‘No Opinion’ Response Choice? • Opinions differ, but you want to avoid two errors: • a) getting ‘no opinion’ or ‘don’t know’ response when a respondent actually holds a non-neutral opinion • b) forcing a respondent to choose a position when s/he has no opinion on an issue or knows nothing about it • standard-format question:a closed-ended survey question that does not offer a DK or no opinion option • quasi-filter question:a closed-ended survey question that includes a choice for respondents who have no opinion or do not know about an issue • full filter question:a contingency survey question that first asks whether a respondent knows about an issue, then only asks those who do

19. Factors that interfere with respondent recall of past events • A sensitive or threatening topic • Events that occurred simultaneously • Events that occurred after that being recalled • An issue or event that was not significant • A person’s need to be consistent and not appear to contradict him/herself

20. Asking Respondents about Sensitive Issues • People often underreport having an illness or disability or engaging in illegal or deviant behavior—and overreport positive, desirable behavior • social desirability bias: a tendency for survey respondents to answer in a way that conforms to social expectations or makes them look good rather than to answer honestly

21. Contingency questions • contingency question: a two-part question in which a first question screens who gets the second question, e.g., • Did you vote in the midterm elections last week? [] Yes (Go to Q2) [] No (Go to Q3) • Which gubernatorial candidate did you vote for? • Had you voted in the 2008 elections?

22. Learning from History: The Power of Words • ‘forbid’ vs. ‘not allow’ • spending ‘to help the poor’ vs. ‘for welfare’ • ‘inner city’ • ‘impartial’?

23. EFFECTIVE QUESTIONNAIRE DESIGN TIPS • The length of a survey or questionnaire depends on the format of your survey • Organize questions to minimize discomfort and confusion for respondents • Reduce order effects • order effects: when the ordering of survey questions influences how respondents answer them • Reduce context effects by being aware how preceding questions and the interview setting may influence responses. Ask more general questions before specific ones.

24. Mail & Self-Administered Questionnaires • Advantages • Easy and inexpensive • If mailed, cover wide geographical area and allows respondents to check personal records at home • Offer anonymity and avoid interviewer bias • Disadvantages • Low response rates • No control over survey conditions • Unsuitable for Qs that require visual aids, open-ended Qs, many contingency Qs, and complex questions • Ill-suited for the illiterate or semi-literate

25. Telephone Interviews • Advantages • Cover wide geographical area • Flexible, with most strengths of F2F interviews • Can use contingency Qs effectively, esp w/CATI • Supervisors can monitor quality by listening in • Disadvantages • Higher cost • Limited interview length • Use of interviewer reduces anonymity and introduces potential interviewer bias • Cooperation with telephone interviews has steadily declined

26. Face-to-Face Interviews • Advantages • Highest response rates • Allow longest questionnaires • Interviewers can observe nonverbal communication • Interviewers can ask all types of Qs, complex Qs, and can use extensive probes • Disadvantages • Training, travel, supervision, and personnel costs are high • Interviewer bias • Less supervision than for telephone interviews

27. Web Surveys • Advantages • Lowest cost format • Fastest • Span geographic space • Can include visual materials • Disadvantages • Unrepresentative samples • Cannot control conditions of survey • No one present to clarify or probe

28. SURVEY INTERVIEWING • The Interviewer’s Role • Interview Stages • Training Interviewers • Using Probes • Interviewer Bias • Computer-Assisted Telephone Interviewing

29. The Interviewer’s Role • Survey research interviewing is specialized interview in which the primary goal is to obtain accurate information from another person • It is a social relationship with expectations, social roles, and norms • Interviewers must obtain cooperation and build rapport yet remain neutral and objective • A survey interviewer is nonjudgmental and never reveals his or her opinions, verbally or nonverbally • The interviewer helps define the situation for respondents, determining whether they have the information sought, understand what is expected , are motivated, and provide relevant and serious answers

30. Interview Stages • Introduction and entry • The main part of the interview probe:a neutral request made by an interviewer to clarify an ambiguous answer, complete an incomplete answer or obtain a relevant response • The exit

31. Training Interviewers • A professional-quality interview requires carefully selected and trained interviewers • Researchers consider the interviewer’s physical appearance, age, race, sex, languages spoken, even the voice

32. Using Probes • Probes are not substitutes for writing clear questions or creating a framework of understanding for the respondent • Unless carefully stated, probes might shape respondent’s answers • Yet, flexible, conversational interviewing using many probes can improve accuracy on questions about complex issues

33. Interviewer Bias • An interviewer’s uncontrolled visible characteristics, including race and gender, often affect interviews and respondent answers • Thus, you should note the race and gender of both interviewers and respondents • In general, interviewers of the same gender or ethnic-racial group as the respondent get the most accurate answers

34. Computer-Assisted Telephone Interviewing • computer-assisted telephone interviewing (CATI): telephone survey technology that integrates interviewing over the phone with a computer for the questionnaire and data entry.