1 / 62

Dr Pamela Campanelli Survey Methods Consultant Chartered Statistician Chartered Scientist

Dr Pamela Campanelli Survey Methods Consultant Chartered Statistician Chartered Scientist. The Questionnaire Design Pitfalls of Multiple Modes. Acknowledgements. Other main members of UK “Mixed Modes and Measurement Error” grant team: Gerry Nicolaas Ipsos MORI

feryal
Download Presentation

Dr Pamela Campanelli Survey Methods Consultant Chartered Statistician Chartered Scientist

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Dr Pamela CampanelliSurvey Methods ConsultantChartered StatisticianChartered Scientist The Questionnaire Design Pitfalls of Multiple Modes

  2. Acknowledgements Other main members of UK “Mixed Modes and Measurement Error” grant team: Gerry Nicolaas Ipsos MORI Peter Lynn University of Essex Annette Jäckle University of Essex Steven Hope University College London Grant funding from: UK Economic and Social Research Council (Award RES-175-25-0007)

  3. Larger Project Looked for Evidence of Mode Differences by Today a few highlights • Question Content • Question Format • Type of task • Characteristics of the task • Implementation of the task • Made recommendations

  4. Sensitive Questions • Mixed Mode Context • Very well-known • Sensitive questions prone to social desirability effects in interviewer modes (see Tourangeau and Yan, 2007; Kreuter, Presser and Tourangeau, 2008) • But not all questions (Fowler, Roman, and Di, 1998) • Difference by time frame (Fowler, Roman, and Di, 1998)

  5. Non-Sensitive: Factual Versus Subjective (1) • Mixed Mode Context • Subjective questions more prone to mode effects than factual questions (see LozarManfreda and Vehovar, 2002; Schonlau et al, 2003) • But factual questions also susceptible (Campanelli, 2010) • Subjective scalar questions can be prone to TEL positivity bias

  6. TEL (and F2F) Positivity Bias • Dillman et al (2009) - aural versus visual effect • TEL Rs giving more extreme positive answers • Ye et al (2011) - TEL Rs giving more extreme positive answers • But found that F2F was like TEL • Concluded caused by a MUM effect • Hope et al (2011) – TEL Rs giving more extreme positive answers • But no trace of this in F2F (with a showcard and without a show card) • Thus, actual cause for the TEL positivity bias is still unclear

  7. Non-Sensitive: Factual Versus Subjective (2)

  8. Inherently Difficult Questions (1) • General Questionnaire Design Context • Inherent difficulty: Question is difficult due to conceptual, comprehension and/or recall issues • Survey satisficing should be greater for inherently difficult questions (Krosnick, 1991) • But this is not true for all inherently difficult questions (Hunt et al, 1982; Sangster and Fox, 2000; Nicolaas et al, 2011)

  9. Inherently Difficult Questions (2) EXAMPLE: Nicolaas et al (2011)

  10. Inherently Difficult Questions (3)

  11. Mark All That Apply vs. Yes/No for Each (1) Mark at that apply Yes/No for each

  12. Mark All That Apply vs. Yes/No for Each (2) • General Questionnaire Design Context • ‘Mark all that apply’ is problematic • Sudman and Bradburn (1982) • Rasinski et al (1994), Smyth et al (2006) and Thomas and Klein (2006) • Thomas and Klein (2006) • Smyth et al (2006) • Nicolaas et al (2011)

  13. Mark All That Apply vs. Yes/No for Each (3) • Mixed Mode Context • Smyth et al (2008) - student sample • Nicolaas et al (2011) - probability sample of the adult population • More research needed

  14. Mark All That Apply vs. Yes/No for Each (4) Mark all that apply Yes/No for each

  15. Ranking versus Rating (1) Ranking Battery of Rating Questions

  16. Ranking versus Rating (2) • General Questionnaire Design Context • Ranking • Is difficult (Fowler, 1995) • Primacy effects (see Stern, Dillman & Smyth, 2007) • Better quality (see Alwin and Krosnick, 1985; Krosnick, 1999; Krosnick, 2000).

  17. Ranking versus Rating (3) • Mixed Modes Context • Rating more susceptible to non-differentiation in Web than TEL (Fricker et al, 2005) • Similarly, rating sometimes more susceptible to non-differentiation in Web or TEL than F2F (Hope et al, 2011) • Ranking more susceptible to non-differentiation in Web than F2F (TEL not tested) (Hope et al, 2011)

  18. Ranking versus Rating (4) Ranking Rating

  19. Agree/Disagree Questions • General Questionnaire Design Context • Agree/Disagree questions are a problematic format in all modes • They create a cognitively complex task • Are susceptible to acquiescence bias • For additional problems see Fowler (1995), Converse and Presser (1986), Saris et al (2010) and recent Holbrook AAPOR Webinar • Mixed Modes Context • Differences across modes were found with more acquiescence bias in the interview modes and curiously, more middle category selection in SA (Hope et al, 2011)

  20. Use of Middle Category (1)

  21. Use of Middle Category (2) • General Questionnaire Design Context • Kalton et al (1980) • Krosnick (1991) and Krosnick and Fabrigar (1997) • Schuman and Presser (1981) • Krosnick and Presser (2010) • Krosnick and Fabrigar (1997) • O’Muircheartaigh, Krosnick and Helic (1999) • Hope et al (2011)

  22. Use of Middle Category (3) • Mixed modes context • More use of the middle category in visual (as opposed to aural) mode (Tarnai and Dillman, 1992) • More selection of middle categories on end-labelled scales than fully labelled scales, but less so for TEL (Hope et al 2011) • More use of the middle category in Web as opposed to F2F or TEL (Hope et al 2011)

  23. Use of Middle Category (4)

  24. OverallTypology of Questions

  25. A classification of question characteristics relevant to measurement error

  26. In Summary Mode is a characteristic of a question Good questionnaire design is key to minimising many measurement differences But we are unlikely to eliminate all differences as there are different types of satisficing in different modes We need to do more to assess any remaining differences and find ways to adjust for these (more on this in the next few slides)

  27. Assessing Mixed Mode Measurement Error (1) • Quality indicators • For example: • Mean item nonresponse rate • Mean length of responses to open question • Mean number of responses in mark all that apply • Psychometric scaling properties • Comparison of survey estimates to a ‘gold’ standard (de Leeuw 2005; Kreuter et al, 2008; Voogt and Saris, 2005) • Although validation data often hard or impossible to obtain • Etc.

  28. Assessing Mixed Mode Measurement Error (2) • How was the mixed mode data collected? What are the confounding factors or limitations? • Random assignment • R’s randomly assigned to mode (Nicolaas et al, 2011): But this is not always possible • Random group changes mode during the interview (Heerwegh, 2009) • In both cases non-compatibility can occur due to differential nonresponse bias • R choses mode of data collection • May reduce nonresponse, but selection and measurement error effects are confounded (Vannieuwenhuyze et al, 2010)

  29. Assessing Mixed Mode Measurement Error (3) • Ways to separate sample composition from mode effects • Compare mixed mode data to that of a comparable single-mode survey (Vannieuwenhuyze et al, 2010) • Statistical modelling: • Weighting (Lee, 2006) • Multivariate model (Dillman et al, 2009) • Latent variable models (Biemer, 2001) • Propensity score matching (Lugtig et al, 2011) • Matching Rs from two survey modes which share the same background characteristics • Identify Rs who are unique to a specific survey mode and those who are found in both modes • May be a useful technique

  30. Assessing Mixed Mode Measurement Error (4) • The size of effects between modes • Depends on the type of analyses, which • Depends on the type of reporting needed • For example: • Reporting of • Means • Percentages for extreme categories • Percentages for all categories

  31. We hope that today’s talk has given you. . . • More understanding of the theoretical and practical differences in how Rs react to different modes of data collection • More awareness of specific question attributes that make certain questions less portable across modes • More knowledge and confidence in executing your own mixed modes questionnaires

  32. Thank you all for listening dr.pamela.campanelli@thesurveycoach.com Complete table of results and recommendations available upon request

  33. Appendix

  34. Open Questions (1) • General Questionnaire Design Context - SA • Lines in text boxes versus an open box • Christian and Dillman (2004) • But Ciochetto et al (2006) • Slightly larger answer spaces (Christian and Dillman, 2004)

  35. Open Questions (2) • Option 1: Fully open questions (continued) • Mixed Mode Context • TEL Rs give less detailed answers to open-ended questions than F2F Rs (Groves and Kahn, 1979; Sykes & Collins, 1988; de Leeuw and van der Zouwen, 1988) • Paper SA Rs give less complete answers to open-ended questions than F2F or TEL Rs (Dillman, 2007; de Leeuw,1992, Groves and Kahn, 1979) • Web Rs provide 30 more words on average than paper SA Rs (Schaeffer and Dillman, 1998) • Positive effects of larger answer spaces may also apply to interview surveys (Smith, 1993; 1995)

  36. Open Questions (3) Option 1: Fully open questions (continued)

  37. Open Questions (4) • General Questionnaire Design Context - SA • Small changes in visual design can have large impact on measurement • Examples  • Couper, Traugott and Lamias (2001) • Smith (1993; 1995) • Dillman et al (2004) • Martin et al (2007)

  38. Open Questions (5) Option 2: Short number, date or textual/verbal response (continued) Mixed Modes Context

  39. End-labelled versus Fully-labelled (1) • General Questionnaire Design Context • Krosnick and Fabrigar (1997) suggest that fully-labelled scales are • Easier to answer • More reliable and valid • Two formats are not equivalent • Fully-labelled scales produce more positive responses (Dillman and Christian, 2005; Campanelli et al, 2012) • End-labelled scales have a higher percent of Rs in the middle category (Campanelli et al, 2012; not discussed in text but in tables of Dillman and Christian, 2005)

  40. End-labelled versus Fully-labelled (2) • Mixed Modes Context • Although higher endorsement of middle categories on end-labelled scales • Less true for TEL Rs (Campanelli et al, 2012)

  41. Branching versus No Branching (1) • General Questionnaire Design Context • In TEL surveys, ordinal scales are often changed into a sequence of two or more branching questions in order to reduce the cognitive burden • Krosnick and Berent (1993) • Malhotraet al (2009) • Hunter (2005) • Nicolaas et al (2011)

  42. Branching versus No Branching (2) • Mixed Modes Context • Nicolaas et al (2000) found more extreme responses to attitude questions in the branched format in TEL mode (but unclear whether more valid) • Nicolaas et al (2011) found • Mode differences between F2F, TEL and Web, but with but with no clear patterns • No mode difference for the non-branching format • More research needed

  43. Branching versus No Branching (3)

  44. Implementation of task

  45. Use of instructions, probes, clarifications, etc. (1)

  46. Use of instructions, probes, clarifications, etc. (2) • It is common practice to provide interviewers with additional information that can be used if necessary to improve the quality of information from Rs • Although not yet studied in mixed modes, it is likely that this may result in differences across modes in a study that uses SA alongside interviewer modes

  47. Use of instructions, probes, clarifications, etc. (3)

  48. Don’t Know (1) • General Questionnaire Design Context • Offering explicit ‘don’t know’ response greatly increases cases in this category • Particularly true for R’s with • lower educational attainment • (see Schuman and Presser, 1981; Krosnick et al, 2002) • Common practice not to provide an explicit ‘don’t know’ in TEL and F2F • In SA modes, the ‘don’t know’ option tends to be either an explicit response option or it is omitted altogether

  49. Don’t Know (2) • Mixed Mode Context • Treating ‘don’t know’ differently in different modes may result in different rates of ‘don’t know’ across the modes • Fricker et al (2005) • Dennis and Li (2007) • Bishop et al (1980) • Vis-Visschers (2009)

  50. Don’t Know (3)

More Related