Response Biases in Survey Research. Hans Baumgartner Smeal Professor of Marketing Smeal College of Business, Penn State University. Response biases. when a researcher conducts a survey, the expectation is that the information collected will be a faithful representation of reality;
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Smeal Professor of Marketing
Smeal College of Business, Penn State University
when a researcher conducts a survey, the expectation is that the information collected will be a faithful representation of reality;
unfortunately, this is often not the case, and survey researchers have identified many different sources of error in surveys;
these errors may contaminate the research results and limit the managerial usefulness of the findings;
if the response provided by a respondent does not fully reflect the “true” response, a response (or measurement) error occurs (random or systematic);
response biases (systematic response errors) can happen at any of the four stages of the response process, are elicited by different aspects of the survey, and are due to a variety of psychological mechanisms;
The relationship between observed measurements and constructs of interest
Should reverse-keyed items (also called oppositely-keyed, reversed-polarity, reverse-worded, negatively worded, negatively-keyed, keyed-false, or simply reversed items) be included in multi-item summative scales?
If reversed items are to be used, does it matter whether the reversal is achieved through negation or through other means?
What’s the link between reversal and negation, what types of MR result, what psychological mechanisms are involved, and how can MR be avoided?
MR → within-participant inconsistency in response to multiple items intended to measure the same construct;
although responding to reversed items is error prone, wording all questions in one direction does not solve the problem;
negations should be employed sparingly, esp. if they do not result in an item reversal (note: negations come in many guises);
polar opposite reversals can be beneficial (esp. at the retrieval stage), but they have to be used with care;
Attending to and interpreting survey questions (careless responding)
Generating a retrieval strategy and retrieving relevant beliefs from memory (confirmation bias)
Integrating the information into a judgment
Mapping the judgment onto the response scale and answering the question(acquiescence)
both NARS (gNARS = .33, p < .001) and IMC (gIMC = .31, p < .001) were highly significant determinants of inconsistency bias;
the effect of NARS on inconsistency bias was invariant across item arrangement conditions, as expected;
the effect of IMC did not differ by item arrangement condition;
the manipulation of whether or not the first target item was reversed (FIR) did not affect responses (although in the first study the effect was significantly negative);
the effect of FIR did not differ by item arrangement condition;
eye-tracking data may provide more detailed insights into how respondents process survey questions and arrive at an answer;
eye movements can be recorded unobtrusively, and eye fixations show what respondents attend to while completing a survey;
AOI1a to AOI1e
Note: These results are based on a mixed model with respondent and construct as random effects.
Note: These results are based on a mixed model with dist=gamma and construct as a random effect.
when respondents minimize the amount of effort they invest in formulating responses to questionnaire items by selecting the first response that is deemed good enough, they are said to be satisficing; when respondents put in the cognitive resources required to arrive at an optimal response, they are optimizing (Krosnick1991);
the effectiveness of procedural remedies to prevent or at least reduce satisficing (MacKenzie & Podsakoff2012) is limited;
post hoc indices designed to identify satisficersoften exhibit limited convergent validity and unambiguous cutoff values are often unavailable;
online surveys are likely to contain data from respondents who are satisficing, but what will be the consequences?
we review satisficing and related measures that have been proposed in the literature and propose a new measure called OPTIM;
we investigate the effect of satisficing on two stylistic response tendencies (ERS and MRS) and we demonstrate that the direction of the relationship varies across individuals;
the notion of satisficing is consistent with the view of people as cognitive misers (Fiske and Taylor 1991);
satisficing is conceptually similar to carelessness, inattentiveness, insufficient effort responding, and content-nonresponsive, content-independent, noncontingent, inconsistent, variable or random responding;
Krosnick (1991) argues that in weak forms of satisficing each of the four steps of the response process (comprehension, retrieval, judgment, response) might be compromised to some extent, whereas in strong forms of satisficing the second and third steps might be skipped entirely;
a single-category measure is unlikely to assess satisficing adequately;
direct measures of satisficing are desirable (esp. response time measures);
bogus items and IMC’s have limitations;
response differentiation for unrelated items might be a good outcome-based measure;
two online studies with Belgian (n=320) and Dutch (n=401) respondents;
in dataset A 10 heterogeneous attitudinal items and in dataset B Greenleaf’s (1992) ERS scale;
these items were used to construct the ERS (number of extreme responses), MRS (number of midpoint responses) and DIFF measures; survey duration was measured unobtrusively;
use of a multivariate Poisson regression mixture model of ERS and MRS on OPTIM;
how do stylistic response tendencies evolve over the course of a questionnaire?
prior research has only considered the effect of stylistic responding on the covariance structure of items or sets of items and has ignored the mean structure;
are there individual differences in both the extent to which stylistic response tendencies occur across respondents and the manner in which stylistic response tendencies evolve over the course of a survey?
prior research has not emphasized heterogeneity in stylistic response tendencies across people;
data from 523 online respondents;
each participant responded to a random selection of eight out of 16 possible four-item scales shown on eight consecutive screens in random order;
eight separate response style indices were computed for both (net) acquiescence response style or NARS (i.e., respondents’ tendency to express more agreement than disagreement) and extreme response style or ERS (i.e., respondents’ disproportionate use of more extreme response options);
the design guarantees that there is no systematic similarity in substantive content over the sequence of eight scales across respondents;
yijt = ijt + ijt jt + ijt + ijt
yijt a person’s observed score on the ith measure of construct j at time t
jt a person’s unobserved score for construct j at time t
ijt systematic error score
ijt random error score
ijt coefficient (factor loading) relating yijt to jt
ijt intercept term (additive bias)
we analyzed items from volumes 1 through 36 of JCR (1974 till the end of 2009) and volumes 1 through 46 of JMR (1964 to 2009);
we included all Likert-type scales for which the items making up the scale were reproduced in the article and factor loadings or item-total correlations were reported;
total of 66 articles in which information about 1330 items measuring 314 factors was provided;
of the 1330 distinct items in the data set, 608 came from JCR and 722 from JMR;
in our data set of 1330 items, between 83 and 86 percent of items were nonreversed (depending on the definition of reversal);
the proportion of factors (or subfactors in the case of multi-factor constructs) that do not contain reversed items was 70 percent;
only 8 percent of factors (out of 314 factors) were composed of an equal number of reversed and nonreversed items (i.e., the scale was balanced);
Theoretical explanations of MR:Reversal ambiguity and comprehension
Top management in my company has let it be known in no uncertain terms that unethical behaviors will not be tolerated.
In uncertain times, I usually expect the best.
I’m always optimistic about my future .
Overall, I expect more good things to happen to me than bad.
If something can go wrong for me, it will.
I hardly ever expect things to go my way.
I rarely count on good things happening to me.
models in which method effects are included generally yield a much better fit to the data than models in which only substantive factors are included;
it is often difficult to clearly distinguish between different method effect specifications on the basis of statistical criteria;
the psychological processes causing method effects are frequently left unspecified;
although method factors have been related to a variety of other psychological constructs, the choice of these other constructs often seems ad hoc;