Survey Research

1. Survey Research 9810001M Eve 9810008M Leslie

2. Survey Research • What is a “Survey Research”? • Data Collection Choices • Design Phase • Sampling Plan • Designing the Questionnaire • Formatting the Questionnaire • Pilot Testing the Questionnaire • Language Difference Between the Researcher and the Respondent • Special Types of Questions

3. Survey Research • Letter of Transmittal • Conducting the Survey • Response Rate Considerations • Notes Specific to Phone Interviews • Notes Specific to Personal Interviews • Data Analysis With Survey Research

4. What is a “Survey Research”? As methods for descriptive research Or as data collection methods within other research designs

5. Data Collection Choices(p.172) • Ways of data collection: mail, telephone, personal interviews, e- mail, Web-based surveys, or combination of these methods • Consideration of choosing methods: • Purpose • Cost factors • Size & characteristics of the sample

6. Comparison

7. Design Phase • The process of design phase: • Articulate the purpose (25 words) • State specific objectives • Consider information which is needed • Evaluate design options

8. Design Considerations (p.171) Ex) Senior students’ research paper writing difficulties in the Department of Applied English at I-Shou University in 2009 Ex) 1-year, 2-year-, and 3-year graduates’ perceptions of RP course in the AE Department at ISU in 2006-2008 Ex) Different grade level students’ writing difficulties in the AE Department at ISU in 2009

9. Best Practices for Survey & Public Opinion Research(p.169) • Have specific goals • Consider alternatives • Select samples • Use designs which balance with costs • Match question wording to the concepts • Pretest questionnaire & procedures • Train interviewers • Construct quality checks

10. Best Practices for Survey & Public Opinion Research 9. Maximize cooperation or response rates in the limitation of subjects 10. Use appropriate techniques 11. Give respondents pledges of confidentiality 12. Disclose methods

11. Sampling Plan Identification of the Respondents • How to choose the source of information: • Who has access to the information • The characteristics in terms of age or disabling conditions • The type of information needed can decide the best source of that information

12. Sampling Plan • Three possible choices of longitudinal study: • Trend analysis- the same general population, but different individuals • Cohort analysis- the same specific population, but not the exact same individuals • Panel analysis- the same individuals over the time period

13. Sampling Plan Population Definition • Identify general nature of the respondents Become specific about information sources Translate conceptual definition of the population into operational terms Sampling Decisions • Two samplings :

14. Sampling Plan • Three errors in sampling: • Coverage error 1. should be in the sampling frame, but not there 2. people who are ineligible b. Nonresponse error 1. refuse to be interviewed 2. reject to finish the questionnaire 3. cannot be reached • Sampling error Use different (wrong) sampling strategies drawn from a population

15. Sampling Plan Sampling Minority Populations • Limitations of minority populations • Results based on the subset will be less reliable. • The small subset could not be broken down further into other subgroups for analysis. The researcher should … Plan to oversample minority populations

16. Designing the Questionnaire Two ways of designing the questionnaire • Before designing the questionnaire… Review the literature to see which part you might be able to borrow from • Develop your own questionnaire : • Outline the various topics you want to include in your survey. ( the Delphi Technique) • Explain to yourself why you are asking each question. 3. Decide on the degree of structure that is most appropriate. (closed / open formats)

17. Steps in Using the Delphi Technique • Select an expert panel • Design an open-ended questionnaire • Gather the responses & eliminate overlap or redundancies; reformate the topics • Return the reformatted list to each participant for review • Calculate the mean & standard deviation (SD); report summary statistics back to the respondents

18. Designing the Questionnaire 4. Ask questions to the pilot group first to develop the response options. 5. Avoid psychologically threatening questions. 6. Clarity is important. 7. Prefer to use short items. 8. Avoided to use negative wording. 9. Ask only one idea in each question. 10. Avoid jargon & big words; spell out acronyms. 11. Avoid biased or leading questions. 12. Use italics, underlining, or bold letters to emphasize critical words.

19. Formatting the Questionnaire • Hints for the appearance of questionnaire: • Make it attractive. • Organize & lay out the questions. • Number the items & the pages. • Put name & address. • Include brief & clear instructions. • Use examples if the item is confusing to respondents.

20. Formatting the Questionnaire 7.Organize questions in a logical sequence. 8. Begin with interesting & nonthreatening items. 9. Don not put the most important items at the end of a long questionnaire. 10. Avoid using “questionnaire” or “checklist” on the form. ( “Response form” replaced)

21. Pilot Testing the Questionnaire • Before testing the questionnaire, test a small group similar to the intended groups. • Steps of pilot test: • Select a sample similar to your population. • Allow room for modifying if necessary. • Encourage pilot participants to give suggestions • Follow the procedures for administration. • Add, change, or delete any questions if necessary

22. Language Difference Between the Researcher and the Respondent • Rely on collaboration with native- speaking colleagues; “back translation” is usually used • Disadvantage of “Back translation” : Result in unnatural, awkwardly words, or incomprehensible meanings • Strategy of helping “back translation” : Decentering- modify wording to the source document to accommodate concepts that are not directly translatable

23. Special Types of Questions 1. Demographic • “Background Information”; personal characteristics • Demographic data about disabilities can be added by some concerning categories 2. Nonthreatening Behavioral Questions • Ask people questions about usual behavior

24. Special Types of Questions • Suggestions for keeping in mind: • Aided recall may help. • Put a time period in the question. (bounded recall) • Use diary approach for detailed information about low-salience behavior • Specify whose behavior you are interested in.

25. Special Types of Questions • Two examples of surveying people who with disabilities: • Blame-the-victim theory (Are your difficulties in understanding people mainly due to a hearing problem?) • Transformative theory (Are your difficulties in understanding people mainly due to their inabilities to communicate with you?)

26. Special Types of Questions 3. Threatening Behavioral Questions Any questions elicit a defensive reaction in the participant. It’s better to use : Open-ended questions Longer questions with explanations Familiar words Put answers into sealed envelops. Use appropriate wording. (e.g., “Such behaviors occur with different frequencies…”, “Did you ever happen to…”) Use multiple measures until getting the truth.

27. Special Types of Questions 4. Knowledge Questions • Ask question to test participants’ knowledge of a subject in school. • It’s better to use: • The knowledge questions before asking attitude questions. • An appropriate level of difficulty. • Avoid asking ”do you happen to know….? ” • Avoid using mail survey. • Use the name recognition list.

28. Special Types of Questions 5. Attitude Questions • In attitude questions, make sure to give a definition of the term or concept in advance. • It’s better to ask three kinds of attitudes: • Affective –like vs. dislike: How does the person feel about this? • Cognitive- knowledge: What does that person know about this? • Action- What is the person willing to do about this? • Assess attitude strength – How much do you like or dislike…?

29. Special Types of Questions e. Avoid questions that include more than one concept. – Would you vote for me and peace or my opponent and war? f. Use bipolar questions rather than unipolar ones. - Are you satisfied or dissatisfied with….? g. Move on to measure degree of positive or negative attitude.- Are you strongly satisfied or moderately satisfied with….? h. Use five points/ alternatives on the rating scale.

30. Letter of Transmittal • Letter of Transmittal • A cover letter to specify the purpose of the survey. • Here are some hints to motivate the returned a questionnaire: • Appeal to authority: Need the well-known person’s signature. • Appeal to self-interest: “ You are one of the few people with the intelligence to be able to help us with this problem.” • Appeal to professional interests: “This is a very important question in our profession.”

31. Letter of Transmittal • Appeal to altruism: “The results of this survey will be used to solve one of our nation’s most pressing problems.” • Appeal to curiosity: Offer to send a copy of the results. • Appeal to greed: Offer to send a monetary incentive. • Appeal to a sense of connection with you : Enclose a pencil to make it easier to sit down and start writing.

32. Conducting the Survey The process of conducting the survey: • Be sure to send out an advance letter. • Enclose the questionnaire with the transmittal letter for mail surveys. • Supervise the data collection. • Send a follow-up to nonrespondetns. • Send again • Send a postcard reminder • Make a call

33. Conducting the Survey • Control processing errors. • Enter the data into the database of your choice. • Clean up the data before you begin analysis.

34. Response Rate Considerations • The American Association for Public Opinion Research offers a definition of response rate and various methods to calculate them at their Website. • Response rates- The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample. Response Rates : I • RR1=----------------------------------------- (I+P)+(R+NC+O)+(UH+UO)

35. Response Rate Considerations • Choose the follow-up mailing. • Cook, Heath, and Thompson(2000) conducted a meta-analysis of factors influencing response rates in Internet-based surveys. • Three factors that increased response rates. • Follow-up contacts with nonrespondetns • Personalized contacts • Contacting sampled people prior to sending out the survey

36. Response Rate Considerations • Other suggestions for increasing response rates include these: • Questionnaire length sometimes has an effect on response rates. • Good timing is very important. Avoid sending the questionnaires at the beginning or end of a school year or around a holiday period. • Be sure to make it easy to return the questionnaire by providing a stamped, self-addressed envelope. • Hand written envelopes seem to attract more attention now than computer-generated address labels.

37. Response Rate Considerations • Nonresponse is a serious issue for survey researchers. • Jones(1995) used a computer simulation to test the effect of response rate and the assumed differences between respondents and nonrespondents.

38. Response Rate Considerations • If the assumption holds that the respondents and nonrespondents are similar, even a 50% return rate would be acceptable. • The reduced response levels resulted in increasingly biased estimates. • When the effect size was moderate, a response rate of at least 90% was needed.

39. Notes Specific to Phone Interviews • It is a good idea to send an advance letter • Provide a brief explanation of your purpose, who you are, and what your expectations are. • Make sure that you are talking to the right person.

40. Notes Specific to Phone Interviews • Make sure that this is a good time to talk. • Try to keep the phone time to a short duration. • On the telephone, it is best to use somewhat structured questions.

41. Notes Specific to Phone Interviews • Establish rapport and move quickly. • Make your first questions fairly simple and nonthreatening. • Use an appropriate tone of voice.

42. Notes Specific to Phone Interviews • Speak at an appropriate speed . • Before conducting the survey, be sure to rehearse. • You can tape-record a phone call, but you must inform the respondent that you are doing so.

43. Notes Specific to Personal Interviews • Personal interviews are often associated with qualitative research. • Preparing for the Interview: • Hold an introductory meeting to share the purpose. • Schedule the interview time. • Make a flexible interview. • Don’t use yes-or-no questions. And plan to ask open-ended questions. (e.g., How do you feel about….? Or What do you think about….?) • Plan to conclude with open-ended questions. (e.g., Is there anything that I didn’t ask about…? Or Is there anything that you wanted to tell me…? ) • Pretest your interview procedures.

44. Starting and Conducting the Interview • Briefly review the purpose of the interview and information needed. • Focus on what the participant is saying. • Sequence the questions from general to specific. • Ask for specific examples. • Avoid asking “why” questions. It’s better to use ”how come.” • Ask a variety of different kinds of questions. • Use role play or simulation questions (e.g., If I were you….) • Record the interview and take notes.

45. Data Analysis With Survey Research • If you have conducted a follow-up of nonrespondents, you can do a comparison between respondents and nonrespondents. • You can do comparisons between those who responded on the first wave, second wave, and so on. • You do need to make decisions about how to handle missing data. • Display the data using tables and graphs.

46. Thank you for your listening.