Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim

1. Using Probing Techniques to Assess Intercultural Validity The Case of Attitudes Towards Immigrants Michael Braun & Dorothée Behr GESIS – Leibniz Institute for the Social Sciences, Mannheim

2. Overview • Establishing comparability across countries • Conventional use of cognitive techniques • Problems and alternatives • Methods and data • Results • Conclusions

3. Establishing comparability across countries

4. Establishing comparability across countries • In intercultural research, measurement invariance has to be established across countries • Application of data-analytic procedures does not get – or at least not always – at causes of incomparability (possible exception: ML-SEM) • Possible solution: cognitive techniques (category-selection probing, comprehension probing, specific probes)

5. Conventional use of cognitive techniques

6. Conventional use of cognitive techniques • as a pretesting device to detect bad items and improve a questionnaire • mainly in a cognitive laboratory, i.e. not under regular field conditions • respondents are briefed about the setting as being a pretest and their task being to help find (comprehension) problems • often “professional” pretest respondents are sought who might develop lay theories

7. Conventional use of cognitive techniques • normally conducted by interviewers proactively searching for hidden comprehension problems • mostly applied to behavioral questions • traditionally based on small quota samples (often not more than 20 interviews) • low number of cases does not permit analysis of diverging argumentation patterns, shared only by a small proportion in the population, and quantification of results is impossible

8. Conventional use in intercultural studies • mainly aiming at the improvement of translations • often application to ethnic groups in only one country • cognitive interviewing across countries very difficult to organize (availability of cognitive laboratories; standardization of procedures) • different house styles in recruiting respondents and guidelines specifying interviewer behavior

9. Problems and alternatives

10. Problems and an alternative • personal interviews even counterproductive (standardization of interviewer qualification and behavior hard to achieve) • time requirements of implementation by face-to-face interviews in the laboratory • no compelling reason (other than costs under traditional implementation) for restrictions • application of cognitive interviewing in the field as independent part of the research process

11. Advantages of using internet surveys • allow for a higher degree of anonymity, improving data quality • reduction of respondent burden • online access panels as cost-efficient means to increase sample size and to investigate small subgroups with diverging response behavior

12. Our project “Enhancing the Validity of Intercultural Comparative Surveys. The Use of Supplemental Probing Techniques in Internet Surveys” Funded by the German Science Foundation (DFG)applicants: Michael Braun, Wolfgang Bandilla & Lars Kaczmirek

13. Project goal • Detection of interpretation differences in international surveys by cognitive methods (category-selection, comprehension and specific probing). • Demonstrating the feasibility to implement cognitive methods in non-probability online access panels. • Testing the approach with substantive survey topics, in the present case immigration items from the International Social Survey Program (ISSP) 2003 National Identity module.

14. Methods and data

15. There are different opinions about immigrants from other countries living in [country]. (By “immigrants” we mean people who come to settle in [country]) How much do you agree or disagree with each of the following statements? a. (crime) Immigrants increase crime rates. b. (economy) Immigrants are generally good for [country’s] economy. c. (jobs) Immigrants take jobs away from people who were born in [country]. d. (culture) Immigrants improve society by bringing in new ideas and cultures. [5-point Likert scale from agree strongly to disagree strongly; crime and jobs reverse coded, such that high values = negative attitudes.]

16. Study design and implementation • Non-probability online access panels in 7 countries/regions (ca. 500 cases each). • Canada, Denmark, United States, (western and eastern) Germany, Hungary, Spain. • Target population defined as country nationals aged btw. 18-65. • Quotas set according to gender, education, age. • Data collection period January 2011.

17. Study design and implementation • Specific probing: “Which type of immigrants were you thinking of when you answered the question? The previous statement was: #.” • 4 immigrant items were rotated using 4 splits, only the first item was probed

18. Challenges • Can we reproduce the answer patterns found in the ISSP data for the different countries? • Do answers to the specific probes reveal different patterns across countries?

19. Results

20. Attitudes towards immigrants

21. Attitudes towards immigrants

22. Attitudes towards immigrants

23. Attitudes towards immigrants

24. Attitudes towards immigrants

25. CICOM compared to ISSP • ISSP data are neatly reproduced in the CICOM data in most cases. • As for deviations, it is not clear whether our sample or social change is the reason (ISSP data were collected in 2003). • It is particularly the Hungarians which seem to be less xenophobic in CICOM than ISSP

26. Answer patterns across countries • Probing question not answered • Answers to probing not pertinent to question, not categorizable answers (“other” category) • Reference to immigrants in general, no specific group • Reference to specific ethnicities, and to which • Reference to immigrant groups defined by either the positive or negative behavior referred to in the item • Reference to the legal/illegal immigrant distinction

27. Results from specific probing: no answer and “other” “other“: partly interpretation as category-selection probing

28. Results from specific probing:Percentages for major answer types

29. Results from specific probing:percentages of major specific ethnicities

30. Probing and reality • This reflects nicely immigration reality in most of these countries: • Canada: really multi-ethnic, many Asians … • Denmark: who knows this country??? • United States: though also multi-ethnic, Latino migration dominant • West (and East) Germany: Turks first, Russians (but perhaps partly ethnic Germans referred to as Russians), Latin Americans unfortunately missing

31. Probing and reality • Spain: Romanians as biggest single migrant group, followed by Marroccans; Latin Americans lumped together is biggest group (but in many cases not distinguishable from natives – or even originally Spanish remigrants from southern America, e.g. from Argentina) • But: What the hell have the Chinese to do with Hungary? Where are they?

32. Magyar Chinatown in Budapest

33. Further results • Immigrants from EU-15 and from Subsaharan Africa are also mentioned, but not as frequently. • Sinti and Roma are rarely mentioned, and mostly restricted to Hungary. • Other groups are still less often mentioned and are put in the “other“ category in our coding schema.

34. Further results • Very small country differences in xenophobia. • After inclusion of the probing codes no country differences left. • Unsurprisingly, thinking of immigrants defined by the positive pole reduces and by the negative pole increases xenophobia.

35. Results from specific probing:general attitudes towards immigrants dependent on thinking of the major specific ethnicities Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

36. Results from specific probing:general attitudes towards immigrants dependent on thinking of the major specific ethnicities Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

37. Results from specific probing:general attitudes towards immigrants dependent on thinking of the major specific ethnicities Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

38. Results from specific probing:general attitudes towards immigrants dependent on thinking of the major specific ethnicities Minimum of 10 respondents mentioned ethnic group; a = ethnic group mentioned by 2 respondents or less.

39. Further results • In general: • Thinking of immigrants in general is connected to less xenophobia. • Thinking of immigrants from Islamic and Eastern European countries is connected to higher xenophobia. • Thinking of immigrants from Asia is related to less xenophobia. • Thinking of Latin Americans does not show a clear tendency (in the US: negative).

40. Further results • Differences between issues and specific immigrant groups mentioned are not really dramatic • Islamic countries are mentioned most frequently with crime item (15.4%) and least with economy item (9%) • The same for eastern Europeans: 14.3% in connection with crime, 4.8% in connection with the economy • There are no relevant issue-specific differences for Asians and Latin Americans

41. Conclusions

42. Some Conclusions • The reasoning in the seven countries/regions is highly similar on an abstract level, though the concrete groups of immigrants mentioned differ. • Respondents with positive beliefs tend to use inclusive statements (e.g. they refer to all immigrants), respondents with negative focus more on selective groups. • In the American context, the legal status of immigrants is mentioned more often than in the European context. • Latin Americans are clearly not equivalent in the Spanish and US-American contexts.

43. Thank you!