SURVEY RESEARCH

1. SURVEY RESEARCH KRISTOPHER, KIM, SHELLEY AND JILLIAN

2. Getting Started… • “A telephone survey says that 51 percent of college students drink until they pass out at least once a month. The other 49 percent didn't answer the phone. “-Craig Kilborn

3. The Following Topics will be Discussed: • Theoretical Tradition and Epistemological Paradigm • Most Appropriate Types of Research Questions • Major Features of the Method • Major Criteria for Rigour • Strengths and Weaknesses • Designing a Project using the Method

4. Theoretical and Epistemological Background • In general, a survey involves the collection of information from a large group of people or a population, commonly via: • Opinion Surveys • Political Polls • TV Viewing Polls • However…

5. Theoretical and Epistemological Background • …our focus is on survey research, which is conducted to advance scientific knowledge or develop theory. • Survey research is frequently utilized within the Social Sciences, including disciplines like Psychology, Marketing, and Organizational Behaviour.

6. Positivism • Positivism has become a dominant institutional form in social research. • However, during the 1970's and 1980's prominent concerns were raised about the limits of quantitative data and methods often associated with positivism; including survey research designs. • Eg: Concerns explained.

7. Positively More Positivism • Assumption: Objective world which science can “mirror” with privileged knowledge. • Key Focus: Search for contextual and organizational variables which cause organizational actions. • Key Theories in Paradigm: Contingencytheory, systems theory, population ecology, transaction cost economics of organizing, dustbowl empiricism, etc. • Goal of Paradigm:Uncover truth and facts as quantitatively specified relations among variables.

8. Positively More Positivism II • Criteria for Assessing Research: • Prediction=Explanation, Rigour; internal/external validity, reliability. • Unit of Analysis: The variable. • Research Methods and Types of Analysis: • Experiments, questionnaires, secondary data analysis, quantitatively coded documents. • Quantitative: Regression, Likert scaling, structural equation modelling. • Qualitative: Grounded theory testing.

9. Major Features of the Method • Variables are often operationalized when researchers ask people questions as a way of getting data for analysis and interpretation (the questions are either asked by an interviewer, or written down and given to respondents for completion) • 2 Forms of surveys: • Questionnaire: document containing questions and others types of items designed to solicit information appropriate for analysis • In dept Interviewing: ask people questions in order to gather data • Survey research is especially appropriate for making descriptive studies of large populations • Data collected may be used for explanatory purposes

10. The Beauty of Questionnaires • It provides a method of collecting Data by either asking people questions or asking them to agree or disagree with statements representing different points of view • Used primarily in survey research, but also in experiments, field research and other modes of observations • Questions can be open ended (respondents supply their own answer-similar to a short answer question on an exam), answers can be in the form of writing on paper or verbally reporting answers to an interviewer • Example: what is your opinion on abortion? • It can also be closed ended (select from a list of answers provided) This type of questionnaire is more common than open-ended ones because they provide greater uniformity of responses, and are more easily processed • Example: From a range of 1-5, how would you rate this hotel? With 1 being poor, and 5 as excellent

11. A Closer Look at Closed Ended Questions • Two structural requirements for closed ended questions: • 1. Response categories provided should be exhaustive (should include all possible responses that might be expected-even if answers may be obscure to you and me). Usually researchers try to ensure this by adding a category such as “other” and may follow “please specify” • 2. The answer category must be mutually exclusive: respondent should not be compelled to select more than one to ensure this, we must carefully consider each combination of categories, could the respondent reasonably choose more than one answer? • If our survey is constructed carefully, we usually do not need to add an instruction to tell respondents to select the “best answer”.

12. Things to Note for whenAsking Questions • Make Items Clear: questionnaire items should be precise so that the respondent knows exactly what the researcher is asking • Avoid Double-Barrelled Questions: Researchers asking respondents for a single answer to question that actually has multiple parts • Respondents must be competent to answer: When asking respondents to provide information, must keep in mind whether they can do so reliably • Respondents must be willing to answer: Often, we would like to learn things from people that they are unwilling to share with us

13. Continued… • Questions should be relevant: Questionnaire should be relevant to most respondents • Shorts Items are the best: respondents are often unwilling to study an item in order to understand it, respondent should be able to read something quickly, understand the main point, and select an answer without difficulty, answers should not be misinterpreted • Avoid Negative Items/”Negative” words: appearance of a negation in a questionnaire items paves the way for easy misinterpretation • Avoid Biased Items and Terms: Meaning of someone’s response to a question depends in large part on its wording, some questions seem to encourage particular response more than do other questions

14. Three Main Methods for Administering Survey Questionnaires • Self administered questionnaires: respondents are asked to complete the questionnaire themselves (example: Mail survey), administer questionnaire to a group of respondents gathered at the same place at the same time. • Surveys administered by interviewers in face-to-face encounters • Surveys conducted by the telephone

15. Mail Distribution and Return • Research worker can either hand deliver questionnaires, requesting that respondents mail the completed questionnaires to the research office or after questionnaires are mailed, researcher visits homes to pick them up and check for completeness • Completion rate seems to be higher when research worker delivers the questionnaire, picks it up or both. • Basic method of collecting data through mail is to send questionnaire, with a letter of explanation and a self addressed, stamped envelope for retuning (for example, like the surveys we receive in our mailboxes, many often have the “postage paid” indicated in the corner of the envelope) • The Main reason for not returning questionnaires is that it’s TOO MUCH TROUBLE!

16. Acceptable Response Rates • Response rates: the number of people participating in a survey divided by the number selected in the sample (percentage form) • Inferential statistics used with survey analysis assumes that all members of the initial sample complete and return their questionnaires • Because that is very rare to happen, response bias is a concernresearchers often hopes for the possibility that the respondents look essentially like a random sample of the initial sample, and a somewhat smaller random sample of the total population

17. Continued… • Overall response rate is one guide to the representativeness of the sample reponsdentshigh response rates means less chance of significant response bias, with low response rates , correspondents are likely to differ from the respondents in ways other than their willingness to participate in the survey • Rough guides (with no statistical basis) to what is a good response rate: 50% is adequate for analysis and reporting, 60% is good and 70% is very good

18. Interview Surveys • Interviewers ask questions orally and record respondents answers • Usually done in a face-to-face encounter, but telephone interviewing follow similar guidelines • Most interviews require more than one interviewers, but small scale interviews can be done by one person • Researchers must assume that a questionnaire item will mean the same thing to every respondent and every given response must mean the same when given by different respondents • The interview’s presence should not affect a respondent's perception of a question or the answer given • To save time and money, a given interviewer is typically assigned to complete all the interviews in a particular geographical area

19. General Guidelines for Survey Interviewing • Appearance and Demeanor: interviewers should dress in a fashion similar to that of the people they’ll be interviewing (cleanliness and neatness in modest apparel). Dress and grooming are typically regarded as signs of a person’s attitudes and orientations. Interviewers should be pleasant, must communicate interest in getting to know the respondent without appearing to spy (never too casual or clingy). Interview will be more successful if the interviewer can become the kind of person the respondent is comfortable with, respondents deserve the most enjoyable experience the researcher can provide

20. Continued… • Familiarity with Questionnaire: must be able to read questionnaire items to respondents without error and stumbling over words, lines must be read as though they are part of a natural conversation • Interviewer must be familiar with the specifications prepared in conjunction with the questionnaire • Some questions will not fit a given respondent's situation, interviewer must determine how the question should be interpreted in that situation • It would be better for the interviewer to leave a question unanswered than to spend a period of time searching though the specifications for clarifications or trying to interpret the relevant instructions • Following Question Wording Exactly • Recording Responses Exactly • Probing for Responses: Sometimes respondents will give an inappropriate/incomplete answer, request for an elaboration can be useful (Probe). Probes must be completely neutral, must not affect the nature of subsequent response

21. Advantages of Doing an Interview Survey • They typically attain high response rates (a properly designed and executed interview survey ought to achieve a completion rate of at least 80-85%) • Presence of and interviewer decreases the number of “don’t knows:” and “no answers”, since clarification can easily be done on the spot • Interviewers can clarify matters if respondent clearly misunderstands the intent of the question • Can observe respondents as well as ask questions

22. Telephone Surveys:Positive Factors (+) • Saves both money and time • May dress anyway you please without affecting the answers respondents give • Respondents may be more honest in giving socially disapproved answers (no eye to eye contact) • Interviewers can communicate a lot about themselves over the phone, even though they can’t be seen • Allow greater control over data collection if several interviewers are engaged in the project

23. Telephone Surveys:Negative Factors (-) • Some phone numbers are unlisted (but this has been erased through a technique called random digit dialling) • Bogus surveys-ones that are actually sales campaigns disguised as research • The ease in which people can hang up • Answering machines (using machines to screen calls)however research has showed that this had not yet had a significant effect on the ability of telephone researchers to contact prospective respondents

24. Comparing theThree Different Methods

25. Computer Assisted Telephone Interviewing (CATI) • Increasingly used by academic, government and commercial survey researchers • Central computer is programmed to select a telephone number at random and dials it • Interviewer introduces study and asks the question displayed on the screen and then types that answer into the computer terminal (depending on the question-open/close ended) • Computer automatically prepares the data for analysis, or researcher can begin analysing data before the interview is complete (gaining and advanced view of how the analysis will turn out)

26. New Technologies andSurvey Research • CAPI (Computer assisted personal interviewing): face to face interviews rather than over the phone • CASI (Computer assisted self interviewing): respondent reads questions on the computer screen and enters his/her own answers • CSAQ (Computerized self administered questionnaire): respondent receives the questionnaire on a floppy disk, bulletin board, or other means and completes the questions, software then accepts the answers. Respondent then returns the data file • TDE (Touchtone data entry): initiates process by calling a number at the research organization, prompts a series of computerized questions, respondent answers by pressing the keys on the telephone pad • VR (Voice recognition): same as TDE, but the system accepts spoken responses • Internet and world wide web

27. DO Consistent wording between the invitation and the survey Use plain simple language Offer to share selected results from others who also have completed the survey Plan the time of day and day of week to mail (when will respondents most likely be reading mail at home?) Be aware of technical limitations (Will respondents have programs needed to access) Test incentives, rewards and prize drawings to determine the optimal response Limit studies to >15 minutes DO NOT Use terms such as unique ID number in the invitation, then ask respondents to type “password” when they get to the survey Force the respondent to scroll down the screen for the URL for the study location Do’s and Don’ts for Conducting Online Surveys

28. Secondary Analysis • Survey research involves 3 steps: questionnaire construction, sample selection, and data collection (interviewing/self administered questionnaires) • This is a form of research in which the data is collected and processed by one researcher are reanalyzed- often for a different purpose, by another • Example: General Social Survey (GSS) • Advantage: cheaper and faster than doing original surveys, benefit from the work of top flight professionals, enhance possibility of meta analysis (researcher brings together a body of past research on a topic) • Disadvantage: question of validity (have no assurance that data collected will be appropriate for you research interests)

29. Rigour in Survey Research • Internal Validity • To be internally valid, the conclusions of the research must be supported by the data. • Internal validity is judged according to the accuracy with which a description of particular events represents the data. • The essence of internal validity for survey research is complete confidence that your conclusions come from the data.

30. Reliability • Reliability • Reliability is generally concerned with replication: an account is considered to be reliable if the data are reproducible. • If the analytic strategy were repeated by the same or different investigator, then the results should be the same.

31. Another Type of Validity • External Validity • External validity or generalizability for survey research is captured by the question, “How can one determine the extent to which the findings of a particular inquiry have applicability in other contexts or with other subjects?” • Essentially, the extent to which the effect can be generalized to populations, settings, treatment variables, and measurement variables.

32. External Validity Continued… • This can also refer to “fit” or the degree to which the audience or reader of the report is able to transfer the research findings to contexts outside of the study situation to other settings. • The researcher must supply a substantial amount of clear and detailed information or thick description about the issue/phenomenon studied and the setting in which that issue/phenomenon was found. • The degree of transferability is a direct function of the similarity or “fittingness” between the two contexts.

33. Strategies for Ensuring Rigour • Verification Strategies • The most important way to ensure that research is rigourous is to focus on verification during the study. • Verification is the process of checking, confirming, making sure, being certain. • Doing this as the study is conducted is key as the researcher then can identify and correct threats to reliability and validity as they surface.

34. Initial Strategies Address: • Investigator Responsiveness • Methodological Coherence • Sampling • Data Analysis • Thinking Theoretically

35. Additional Strategies • There are other strategies that can be used during the research to contribute to rigour, namely: • Prolonged Engagement • Participant Checks • Journal Writing • Peer Review • Audit Trail.

36. Weaknesses for Survey Research • Lacks the context of social life • Appear superficial on coverage of complex topics • Subject to artificiality • Surveys can also be inflexible because the initial study design must remain the same • Therefore, surveys are weak on validity

37. Strengths for Survey Research • Describes the characteristics of a large population • Allows a large sample of respondents • Surveys are flexible, allows flexibility in your analyses • Surveys have a strength in measurement • Helps determine unemployment rates, and voting intensions • Helps examine official documents such as marriage, birth, or death records • Therefore, surveys are strong on reliability

38. Survey Design • 1. Purpose • 2. Respondent Group • 3. Questionnaire Construction • 4. Administration of Questionnaire • 5. Analysis • Example: Sample Questionnaire (Handout) Student Opinion Survey on Group Assignments in University Courses

39. Format Guidelines • Introduction • Spacing • Explanations • Order of items • Pre-testing

40. Additional Resources • Presentation Website: http://members.shaw.ca/kristopher.skinner/soc315 • Group Project E-mail: soc315project@hotmail.com