Do Attempts to Improve Respondent Attention Increase Social Desirability Bias?
Download data and study materials from OSF
Stony Brook University
Sample size: 1200
Field period: 06/30/2012-12/10/2012
Satisficing refers to the tendency of people to devote insufficient effort while answering questionnaires. We investigate the effectiveness of warnings as a method for decreasing satisficing by boosting respondent motivation. Four different styles of warning messages are examined in data from a randomized survey experiment conducted on the Internet. The analysis shows that three of the four warnings significantly improve respondent engagement. There is some evidence, however, that warning messages increase socially desirable responding (SDR) for certain types of people. We conclude that warnings can be a useful method for increasing attention, but that researchers should first establish that these protocol do not have unintended consequences, either for the entire sample or for theoretically relevant subgroups.
H1: Warning messages will increase respondent engagement
H2: Warning messages will increase the likelihood of socially desirable responding (SDR)
H3: The effect of warning messages on SDR will be greatest for respondents with high levels of education.
The study was a between subjects design with four treatment groups and an untreated control group. The experimental manipulations consisted of brief warning messages that respondents read at the beginning of the questionnaire. See the paper's appendix for the exact text of each warning message.
Measure of attention: two Instructional Manipulation Checks (IMCs). We also examined item non-response and the length of time spent answering questions as indicators of satisficing.
There were seven items that served as indicators of socially desirable responding. See the paper's appendix for the exact text of each warning message.
Summary of Results
The results show that warning messages increase attention as intended, though there is variation in the effectiveness of different warning messages. There is no evidence that warning messages increase aggregate levels of socially desirable responding, but they did increase socially desirable responding among more educated respondents.
Clifford, Scott, and Jennifer Jerit. 2014. Do Attempts to Improve Respondent Attention Increase Social Desirability Bias? Unpublished paper, Stony Brook University.