Substantive vs. Quantitative Determination of Item Comparability in an Organizational Culture Survey

1. Substantive vs. Quantitative Determination of Item Comparability in an Organizational Culture Survey Nathan T. Carter Bowling Green State University Lindsey M. Kotrba Michael Gillespie Denison Consulting Michael J. Zickar Dalia Diab Bing Lin Shuang Pui Bowling Green State University

2. Introduction Rapid globalization ? New challenges National/Regional Cultural Issues Language Issues Increased Value of Org. Surveys Central decision making Organizational intervention planning Challenge: Comparing item- and scale-scores across languages and National culture

3. Background Large multinational client of Denison Consulting Problem: “Can we compare scores on DOCS across language adaptations?”

4. Background BGSU’s Institute for Psychological Research and Application (IPRA) contracted Item Response Theory Analyses Differential Item Functioning Analyses At the same time… Client conducted focus groups to identify items that were perceived as biased in reference to Reference group (English US)

5. A Brief Introduction to Item Response Theory Why Item Response Theory? Foundation of contemporary differential item functioning (DIF). Utilized for examining item-level differences. Items differ in: Item Discrimination Strength of relationship between the trait and responses to the inventory Option Endorsement Rates The frequency at which persons endorse a particular item option.

6. The Basic Concept of Differential Item Functioning (DIF) Enables in-depth multinational comparison of self-report inventories at the item level. Examines score differences between two hypothetical persons who: 1) Took Different Forms of the Inventory 2) Have the Same True Score Differential Functioning Has Occurred When: Those with the same True Score have different probabilities in endorsing the item in a particular direction as a function of the form taken. What Does DIF Imply? Expected Scores are a function of the form taken. Scores on the inventory are not directly comparable.

7. Purpose of this Study To examine different trends in the rate of identification of “biased” items Quantitative Approach (DFIT Study) Substantive Approach (Focus Groups) Provide a more thorough understanding of complimentary approaches E.g. Are they redundant?

8. Results DOCS is generally comparable across language translations Identification rates significantly higher for Substantive Method (19.2% of 840 item comparisons) than the Quantitative (12.5% of 840 item comparisons) McNemar c2=15.15, p<.001 Eastern vs. Western languages Substantive: Eastern: 31.3% Western: 14.3% Kilmogorov-Smirnof z = 2.22, p<.001 DFIT: Eastern: 15.8% Western: 11% K-S z = .63, p = .82 Very little overlap in which items were identified by the methods Overall – Spearman’s Rho = .08 Eastern – Spearman’s Rho = .05 Western – Spearman’s Rho = .08

9. Results Translations with most violations (in bold) for quantitative vs. substantive approaches: None were the same Decisions would be highly different Corrections based on Substantive would be far more intensive

10. Results DFIT: Appears to be more interaction between lingual culture and the construct measured.

11. Results SUBSTANTIVE: Large differences in identification rates across Regions. Sizable differences between Regions for all subscales of Organizational Culture

12. Conclusions Unanswered Questions: No “gold standard” for which is “correct” Two ways to view this: 1) The Substantive Approach offers unique information 2) The Substantive Approach is biased (Ross & Okabe, 2006) DFIT procedure Detection rates less influenced by Region More “efficient” detection of item bias Less costly method of item bias detection

13. Recommendations Focus on scores as the primary concern Decisions made on scores Quantitative approaches will be more useful in this sense. Hambleton & de Jong (2006): “The goal should… be to minimize differences to acceptable semantic, psychometric, linguistic, and psychological levels” (p. 128) This suggests that meaningful information may be gleaned from both DFIT clearly gives guidance on psychometric levels However: Does the substantive approach meaningfully address the semantic, linguistic, and/or psychological levels? What is known: Differences do exist in the two approaches Both have unique identification patterns More research needed to know the exact benefits and limitations of both approaches Is one approach “catching” something the other is not? Are they complimentary? Is one superior to the other?