How Should We Measure the Perceived Ideological Position of the US Supreme Court?
Download data and study materials from OSF
Pennsylvania State University
Sample size: 2106
Field period: 07/01/2014-10/21/2014
Scholarly interest in the legitimacy of the U.S. Supreme Court has exploded in recent years, with a major dispute emerging in the literature: whether a citizen's perceived level of ideological disagreement with the Court's policy outputs diffuse support evaluations. The conclusions drawn by scholars are also associated with the survey question they use to judge the ideological placement of the Court. Bartels and Johnston (2013), who find that ideology affects diffuse support, rely on the following question: "Judging by its recent decisions, do you think the Supreme Court is generally liberal, generally conservative, or is it making decisions more on a case-by-case basis?" Gibson and Nelson (2015) criticize this question, arguing that the measure creates a heterogeneous middle category. Gibson and Nelson (2015), who find that ideology plays only a small role in diffuse support evaluations, use the following question: "Thinking about the United States Supreme Court in Washington and the decisions that it has been making lately, would you say that the Supreme Court is a very liberal court, a somewhat liberal court, a somewhat conservative court, or a very conservative court." This question has also come under fire for its omission of a middle category, the lack of which fails to provide respondents a stated option for "moderate" policymaking. This experiment tests the consequences of question wording on respondents' subjective ideological disagreement with the U.S. Supreme Court, investigating both the original question wordings as well as revised wordings that account for the criticisms of each wording.
To what extend does question wording affect subjective ideological disagreement with the U.S. Supreme Court?
In this question wording experiment, respondents are divided into four groups. Condition 1 is the Bartels and Johnston question: "Judging by its recent decisions, do you think the Supreme Court is generally liberal, generally conservative, or is it making decisions more on a case-by-case basis?" Condition 2 is the Bartels and Johnston question, replacing "case-by-case" with "generally moderate." Condition 3 is the Gibson and Nelson question: Thinking about the United States Supreme Court in Washington and the decisions that it has been making lately, would you say that the Supreme Court is a very liberal court, a somewhat liberal court, a somewhat conservative court, or a very conservative court?" Condition 4 adds a middle category ("a moderate court" to the Gibson and Nelson question.
Since we are interested in the relationship between subjective ideological placement and institutional support, we also include two criterion variables, one relating to the Court's legitimacy and one relating to respondent's overall level of performance satisfaction with the Court. While most extant research on the Court's diffuse support relies upon a scaled measure of legitimacy, the strict question limits inherent in a SSP proposal prohibit the use of the full scale; thus, we include a widely-used indicator of diffuse support ("If the U.S. Supreme Court started making a lot of decisions that most people disagree with, it might be better to do away with the Supreme Court altogether."). The performance satisfaction item we include ("How well do you think the U.S. Supreme Court does its main job in government? Would you say it does a great job, pretty good job, not very good job, or a poor job") has been widely used to measure the Court's specific support.
Summary of Results
The results indicate that both critiques lead to univariate frequencies that differ substantially. About 40% of respondents select the middle category to the Gibson and Nelson question when it is an option. Similarly, whereas a majority of respondents believe that the Court decides cases on a "case-by-case" basis when asked the Bartels and Johnston question, that number declines substantially when that phrase is changed to "generally moderate."
Gibson, James L. and Michael J. Nelson. 2015a. "Measuring the Perceived Ideological Position of the U.S. Supreme Court: Alternative Approaches, Different Conclusions?" Working Paper.
Gibson, James L. and Michael J. Nelson. 2015b. "Too Liberal, Too Conservative, or About Right? The Implications of Ideological Dissatisfaction for Supreme Court Legitimacy" Working Paper.