Who's prejudiced? Who's a conspiracy theorist? Sampling bias in social science

Probably an inevitable result of the political bias of social science

You have no doubt heard that conservatives are more prejudiced and just generally stupid than leftists. For instance, LiveScience in 2012:

And in 2017 Vox talks openly about how all sorts of prejudices are acceptable with conservatives:

And in 2020 New York Times asks about how we can do something about the bigotry that Trump has unleashed:

When's the last time you saw major media or organizations talk about bigotry on the left?

One can easily find academic articles that such articles relied upon. Take this summary from 2013:

Past research closely associates political conservatism with prejudice. Studies have found, for example, that conservatives report less favorable attitudes than liberals toward minorities, including African Americans (Sears & Henry, 2003), Arabs (Echebarria-Echabe & Guede, 2007), homosexuals (Terrizzi, Shook, & Ventis, 2010), and immigrants (Meertens & Pettigrew, 1997). Thus, there is ample evidence for a prejudice gap between liberals and conservatives, with conservatives appearing less egalitarian and tolerant. Current theories of this prejudice gap suggest that conservatives engage in motivated thought processes that legitimize prejudice. System-justification theory (Kay & Jost, 2003), for example, posits that conservatives in particular are motivated to defend the status quo, and that groups (e.g., African Americans) seeking social change and reform are therefore regarded as threatening. According to symbolic-racism theory (Sears & Henry, 2003), anti-Black prejudice stems from a commitment to traditional American values of individualism and self-reliance, and because conservatives endorse those values more strongly than liberals, they are more willing to deny that racial prejudice exists and oppose programs aimed at achieving equality. Similarly, social-dominance theory (Sidanius & Pratto, 2001) contends that there is a basic predisposition to form group-based hierarchies, and that groups in power (e.g., wealthier individuals, who tend to be more ideologically conservative) are motivated to justify their advantaged position over less powerful social groups. Implicit in each of these theories is the notion that political conservatives are more motivated than liberals to rationalize and defend the position of higher-status groups over lower-status groups (often racial minorities), and hold system-justifying beliefs that sustain and legitimize prejudice.

Sounds pretty clear cut, right? There is just one problem. Every group mentioned above is left-wing:

The problem with past studies examining the prejudice gap is that they have confounded the target group’s race or minority status with its stance on core conservative and liberal values. The targets in these studies have tended to be racial, ethnic, or religious minority groups—often groups who are known or assumed to oppose core conservative values (e.g., African Americans generally support affirmative action and social welfare programs, which conservatives oppose). Thus, it is difficult to tell whether these studies’ findings of greater prejudice among conservatives are due to the race or minority status of those groups or due to the groups’ opposition to core conservative values. At the same time, many studies have failed to include as targets relevant comparison or control groups, such as religious fundamentalists, entrepreneurs, or financially well-off individuals—groups that embrace core conservative values and toward whom political liberals might express greater prejudice. Conflating the target group’s race or minority status with its perceived position on core values limits conclusions that can be drawn about general ideological differences in prejudice. Building on research in political science showing that people tend to be more intolerant of groups with ideologies dissimilar to their own (Sullivan, Piereson, & Marcus, 1979), we examined the ideology-prejudice confound by manipulating targets’ race and political attitudes orthogonally and testing whether these factors have separate (and possibly interactive) effects on liking.

You can now see the idea. Suppose you want to prove that conservatives really are as prejudiced and bigoted as the talking heads on MSNPC tells you, how would you study this? Well, just follow the normal procedure. Think about some list of protected groups that are being disliked by conservatives. Then have students rate these groups on a scale of how much they like them 0-100, the so called feeling thermometer. Then you use slight of hand and change the meaning of prejudice from something like "preconceived opinion that is not based on reason or actual experience." into "rating a group lower on a feeling thermometer". Viola, you can now prove in every study that conservatives are prejudiced against your favorite protected classes. Then you do the final thing where you conveniently leave out that these groups were selected for the express purpose of conservatives disliking them.

To put it pointedly: making conservatives look bad by asking them whether they dislike, or like less, groups that are well known to be leftist. Then relabeling this as prejudice. Then pretending this generalizes to prejudice in general.

The same 2013 study then decided to ask people about an actual representative list of groups, and a 2014 study presented this plot:

• Brandt, M. J., Reyna, C., Chambers, J. R., Crawford, J. T., & Wetherell, G. (2014). The ideological-conflict hypothesis: Intolerance among both liberals and conservatives. Current Directions in Psychological Science, 23(1), 27-34.

It's almost yawn tier when laid out clearly like this. The whole enterprise of proving conservatives prejudiced this way was dumb to begin with.

Furthermore, there is no serious attempt at dealing with the rationality of disliking groups. From the perspective of ordinary citizens, criminals, drug abusers, welfare hustlers, etc. are all essentially net negative members of society, so it makes sense to dislike them. They aren't making your life better, and they aren't helping civilization, and they cost tax money. Civil rights leaders, they sound good in theory, but in practice these are just communists with law degrees, so it is not so surprising that these are strongly disliked. Similarly, if one buys into socialism, it makes sense to dislike the enemies of the revolution (the reactionaries, the counter-revolutionaries). These are the people who are privileged by birth who don't deserve it (because meritocracy is a lie right). The bourgeoisie parasites. And the various groups that are known to have a lot of these people, say, business people, Christian fundamentalists, and the military, well, you dislike them too.

Conspiracy theories

What about conspiracy theories? Again, there's no shortage media and institutions talking about conservatives believing in these. The EU parliament tells us about right-wing extremism and conspiracy theories:

And the National Public Radio, NPR, ostensibly a public service non-profit:

And the Canadian Broadcast Corporation (CBC) tells us:

When's the last time you saw major media or organizations talk about dangerous conspiracy theories on the left?

Fortunately, there's a new paper out making the rounds about this issue:

• Enders, A., Farhart, C., Miller, J., Uscinski, J., Saunders, K., & Drochon, H. (2022). Are Republicans and Conservatives More Likely to Believe Conspiracy Theories?. Political behavior, 1-24.

A sizable literature tracing back to Richard Hofstadter’s The Paranoid Style (1964) argues that Republicans and conservatives are more likely to believe conspiracy theories than Democrats and liberals. However, the evidence for this proposition is mixed. Since conspiracy theory beliefs are associated with dangerous orientations and behaviors, it is imperative that social scientists better understand the connection between conspiracy theories and political orientations. Employing 20 surveys of Americans from 2012 to 2021 (total n = 37,776), as well as surveys of 20 additional countries spanning six continents (total n = 26,416), we undertake an expansive investigation of the asymmetry thesis. First, we examine the relationship between beliefs in 52 conspiracy theories and both partisanship and ideology in the U.S.; this analysis is buttressed by an examination of beliefs in 11 conspiracy theories across 20 more countries. In our second test, we hold constant the content of the conspiracy theories investigated—manipulating only the partisanship of the theorized villains—to decipher whether those on the left or right are more likely to accuse political out-groups of conspiring. Finally, we inspect correlations between political orientations and the general predisposition to believe in conspiracy theories over the span of a decade. In no instance do we observe systematic evidence of a political asymmetry. Instead, the strength and direction of the relationship between political orientations and conspiricism is dependent on the characteristics of the specific conspiracy beliefs employed by researchers and the socio-political context in which those ideas are considered.

It's the same issue as before: the selection of conspiracy theories was made so to make conservatives look bad. When one picks a larger set of such theories, the picture is pretty mixed. Their main plot:

I generally understand a conspiracy theory to be a model about how some people are planning and or already executing a plan to accomplish something that is against the law or at least wishes of the population. Most of the ones they include are nice examples. I've never heard of the theory that Trump faked COVID, but sure it's possible, even somewhat plausible, that Trump's inner circle decided that this might gain him some sympathy and that might help the election. But, as Noah Carl has discussed in his own post on this study, there are some conspicuous absences:

While the study is interesting and well-done, one thought I had upon seeing the chart above is that one prominent conspiracy theory is missing. What about the belief that there’s “systemic police racism” in the United States? You might say that’s not a conspiracy theory, regardless of whether it’s true. But I would argue it’s at least as much of a conspiracy theory as the idea that the threat of Covid is exaggerated.

The researchers define conspiracy theory as “an explanation of an event or circumstance that accuses powerful actors of working in secret for their own benefit, against the common good, and in a way that undermines bedrock societal norms, rules, or laws”. However, this is not the only definition available.

The political scientist Michael Barkun distinguishes three types of conspiracy theory. The first is event conspiracies, where the conspiracy concerns limited and well-defined events (e.g., the assassination of President Kennedy). The second is systemic conspiracies, where the conspiracy concerns ongoing machinations by a group of people (e.g., Jews, Freemasons, the Illuminati). The third is superconspiracies, where multiple conspiracy theories are linked together hierarchically (e.g., the work of David Icke).

There’s a case to be made that “systemic police racism” is of the second type, with the group in question being police officers, and the ongoing machinations being racially motivated shootings of black men. According to this conspiracy theory, a disproportionate number of black men are shot by police because of “systemic” racism within law enforcement, rather than for any more mundane reason.

One can also look beyond America:

Pearson’s correlations between beliefs in conspiracy theories and left–right ideological self-identification across 20 countries. Horizontal bands represent 95% confidence intervals, two-tailed tests

For there's a smaller sample of theories here, but we see that the results are all over the place. In Mexico, Nigeria, Egypt, it looks like everything is associated with leftism, and in Italy and France, everything is associated with conservatism. Most countries show some mix. Overall, the association across theories is null:

Remarkably, not a single conspiracy theory investigated in Fig. 2 exhibits correlations with consistently positive or negative signs (regardless of statistical significance) across all 20 countries. For example, the 9/11 conspiracy theory finds significantly more support among those on the left in 8 of the 20 countries and is not significantly related to ideology in the remaining 12. Even the Global Warming Hoax belief, a feature of the political right in the U.S., finds significantly more support among the left in 5 countries (Egypt, Mexico, Nigeria, Saudi Arabia, and Turkey) and exhibits no correlation with ideology in 2 others (Hungary and South Africa). Despite some ideological “balance” across Fig. 2, most correlations are quite weak, suggesting a lack of ideological discrepancy altogether. Indeed, the average correlation––across all conspiracy theory beliefs and countries––is a meager 0.03 (ranging from − 0.05 for 9/11 Truther to 0.10 for Global Warming Hoax).

Sampling bias in theory making

These two case studies nicely illustrate the issue of sampling bias. In statistics, sampling is when you take some subset of a population. A population here usually means a national population, or the human population, or conservatives, or White people, or whichever. But a population can also be the set of all conspiracy theories, or groups one can be prejudiced against, or countries in the world, or whatever else. For the statistics to work correctly, these samples should be random, which is to say, it should be equally likely for each member of the population to end up in the sample. When this is not the case, you have what is called sampling bias. If you are doing a traditional survey via the phone, and you find that unemployed people rarely participate, you have discovered that you have sampling bias. Researchers may then take some extra steps towards solving this. Maybe recruit some people to drive out to unemployed people to talk to them face-to-face and convince them to participate in the survey. Or at least assign more weight in the calculations to the unemployed people one managed to include.

In the research above, however, we see almost deliberate sampling bias. Instead of thinking about the complete set of all groups someone might dislike, a distinct subset was chosen that was designed to produce the results they desired. The same way with the conspiracy theories (and with pseudoscience). There's 100s of conspiracy theories around. One could have started with a survey of ordinary people asking them to name a bunch of conspiracy theories they have heard of. Based on this set, one could present people with a list of the 100 most common ones for their assessment. After this, one could have a reasonable claim to saying that conservatives, or leftists, or women etc. are higher in conspiratorial belief. In fact, if I had to guess, I would guess there's a small association with political leftism due to the mental illness association. Schizophrenics are famously conspiratorial, and these vote mainly left-wing. So overall, we would expect conspiratorial schizophrenia-like thinking to have some left-wing association, probably small.

One way to get around sampling bias is to use fictional, generic theories that are mirrored at random. The authors of the above study did that as well:

Do you think that Democratic [Republican] political elites are…

definitely secretly plotting with large banks to lie about the health of the economy to gain support for their economic policy proposals (coded 4)

probably secretly plotting with large banks to lie about the health of the economy to gain support for their economic policy proposals (3)

probably not secretly plotting with large banks to lie about the health of the economy to gain support for their economic policy proposals (2)

definitely not secretly plotting with large banks to lie about the health of the economy to gain support for their economic policy proposals (1)

Full question wordings for the election fraud, political extremism, economy, health policy, and crime questions appear in the online appendix. Partisanship was measured via the standard seven-point branched question, and ranges from “strong Democrat” to “strong Republican.” Symbolic ideology was measured via a seven-point scale ranging from “extremely liberal” to “extremely conservative.”

And the results:

The authors then undersell their results:

While we observe evidence of partisan motivated conspiracy theory endorsement in all instances, we do not observe a tendency for Republicans or conservatives to engage in such reasoning to a greater extent than Democrats and liberals. Regarding partisanship, in five of the 10 tests, the correlations are not significantly different from one another, indicating symmetry (i.e., p > 0.05). In the other five, Democrats evidence a greater propensity for partisan motivated conspiracy endorsement than Republicans. A similar pattern emerges for ideology. In four of the 10 tests, the correlations are not statistically significantly different from one another. In the remaining six, liberals engage in greater motivated conspiracy theory endorsement than conservatives.

In sum, we find no support for the hypothesis that those on the right are more likely to endorse conspiracy theories that impugn liberals than liberals are to endorse the exact same conspiracy theories when they impugn conservatives. This finding has an important implication for the asymmetry thesis: many individuals accept the opportunity to derogate the opposing political side, regardless of the specific factual details of the charge and even if the charge alludes to a conspiracy. These results explain the patterns in Fig. 1 and lead to our final analysis: an examination of potential partisan/ideological asymmetries in conspiracy thinking, the general predisposition. Despite our findings thus far, this predisposition may be asymmetric across partisan/ideological lines.

I mean, they are certainly right in that particular wording: they don't find that the right is more likely to endorse these theories. They find that the left is more likely. Clearly, a meta-analysis of these results would show an overall effect in the direction of leftism being more with such conspiratorial thinking.