Aversive statistical methods explain differences in “dark” publication in PNAS across subject areas

Bill B. points to this recently published article, Aversive societal conditions explain differences in “dark” personality across countries and US states, which begins:

Humans differ in their levels of aversive (“dark”) personality traits such as egoism or psychopathy. Building upon theories suggesting that socioecological factors coshape the development of personality traits, it can be predicted that prior aversive societal conditions (ASC) (herein assessed via corruption, inequality, poverty, and violence) explain individuals’ levels of aversive personality (assessed via the Dark Factor of Personality, the common core underlying all aversive traits). Results considering individuals from 183 countries (N = 1,791,542) and 50 US states (N = 144,576) support the idea that ASC coshape individuals’ levels of aversive personality.

He writes:

I read the stuff on the link and I have to say that this requires a lot more thought and investigation to figure out if the model makes any sense.

That said, I think they got their data from website questionnaires to cross with ASC index matching to responses. Is this a legit way to do things?

This is way beyond my understanding about how to do research and build models. I think they equate the validity on developing a D statistic with that in developing a g statistic.

Don’t get me wrong, reading the abstract and contents felt pretty good, all sounded good, but I don’t have time to go down the rabbit hole of their citations. (Meager content though). But it looks like bullshit to me…

From the paper:

Recent advances in personality research have provided strong evidence for the existence of a single disposition underlying all aversive traits. The Dark Factor of Personality (D) offers a clear conceptualization of this disposition, defined as “the general tendency to maximize one’s individual utility—disregarding, accepting, or malevolently provoking disutility for others—accompanied by beliefs that serve as justifications”. Much like the g factor of intelligence, D represents the “aversive essence of personality,” from which all aversive traits arise as specific manifestations.

They give some references, and I wouldn’t call it “bullshit” exactly, but I’m skeptical. Over the years I’ve dealt with a lot of assholes (I think that’s the colloquial term for high-D people), and asshole behavior seems very context-dependent. People are assholes in some settings and not in others (as discussed in detail here), so it’s hard for me to think of D as a universal trait. I mean, sure, you can measure it–their paper links to a survey form with options from Strongly Disagree to Strongly Agree for questions such as, “It is hard for me to see someone suffering” and “All in all, it is better to be humble and honest than important and dishonest”–but, even setting aside the issue of insincere responses (something that could be an issue with assholes, no?), I just don’t know how much this construct would predict individual attitudes and behavior.

There seems to be some literature on the topic; from the above-linked paper:

Put simply, D is the underlying trait that predisposes individuals to engage in all kinds of aversive behaviors, as dictated by the idea of a reflective construct. This conceptualization has been supported by several findings, including D predicting behavioral outcomes such as dishonesty, selfishness, or outgroup harm, with specific aversive traits typically not yielding any incremental validity beyond D for self-serving behavior at the cost of others.

OK, maybe. I don’t know. I haven’t looked into this; I’ll have to ask some psychometrician colleagues. As I said, I’m skeptical but I’m open to being convinced.

But the point of the paper is not to talk about D; there’s already a literature on it. Rather, their contribution is to correlate the national average D (as collected from an online survey in which “the country in which participants were located was assessed via geo-locating their IP addresses”) with a country-level measure of aversive societal conditions (corruption, inequality, poverty rate, and homicide rate).

So their basic story is: 1. People in poor countries are assholes. 2. It’s because they’re living in bad, low-trust environments.

They also do a statewide analysis in the U.S. Amusingly enough, Louisiana and Mississippi have the highest level of aversive social conditions, but the states with the most assholes per capita are Nevada, New York, and Texas (also South Dakota, but we can’t do much with that, so we’ll chalk that one to small sample size). The nicest people are in Vermont and Utah, which sounds about right. So at least the measure has face validity.

Setting aside any data issues, there’s a big big problem with this paper: it doesn’t have any causal identification at all! They don’t even attempt causal identification. It’s entirely a correlational story–and the correlations are not so high.

I’ll tell you one thing. There’s absolutely zero chance this paper could ever have been published in an econ journal. Say what you want about economists (and I’ll say a lot), they’ll at least try to identify their causal claims. This one is just . . . here are some correlations and we’re gonna tell a causal story. All righty.

I also have some statistical concerns with the analysis. They adjust for age and sex of respondents, but this seems to occur only in robustness checks, not in the main analysis. That doesn’t make sense to me: you’d want to adjust for these things in your primary analysis, no? This is not a big deal given the larger issues of measurement, sampling and causality, but since I’m here, I’ll mention it.

In summary, it all seems plausible to me that there are such correlations between states and countries. But to say the societal conditions “explain” the differences across personalities doesn’t seem right.

Not at all! Not one bit! It’s not even like they did some causal inference and they have some shaky assumptions. They didn’t even try to do causal inference, and their causal story just seems entirely made up. Nor for that matter does it jibe with my own anecdotal experiences with assholes. I can accept that different countries and different states have different “cultures”–for that matter, different groups within a state or country will have different rates of asshole behavior–; the problem is in trying to pin it to this variable that happens to be very weakly correlated at an aggregate level. This ain’t it, chief.

But, it got published in PNAS, so it got some press, including this ludicrous headline from the (London) Independent: These are the states where psychopaths are most likely to live.

That’s what I call bad reporting. There was nothing in the study about psychopaths. It was a survey form, for Christ’s sake.

Oh well, as long as this sort of research gets shoved into the world’s most prestigious scientific journals (along with air rage, ages ending in 9, himmicanes, and other PNAS classics), and as long as this stuff gets uncritical press coverage, we’ll keep seeing more of it.

It’s almost like the culture of certain scientific subfields enables and encourages certain dark characteristics in their research.

Just to be clear, I’m not trying to say that the authors of this paper are bad people or are trying to do bad research. I assume they don’t know any better: they’re living within an environment in which this sort of behavior is celebrated, and so it makes sense that they do more of it.

And we can hardly blame the staff of the Independent for that stupid article and its incredibly inaccurate headline: journalism today is at a zero budget, and this was probably one of 100 articles and headlines that someone was tasked to write in one day, or maybe they had a chatbot do it.

But . . . maybe we could blame the National Academy of Sciences, or whoever reviewed and edited this paper. Somebody should’ve seen the problem, no? I’d love to see the referee reports on this one.

P.S. On the plus side, they make use of my statistical method:

The multilevel Bayesian model with random loadings and intercepts was estimated in Mplus 8.3 (16) using Gibbs sampling with two MCMC chains over a minimum of 5,000 iterations each, using a potential scale reduction criterion of a maximum of 1.01 to indicate convergence.

How cool is that? This little thing I figured out one day sitting at my desk 35 years ago in Murray Hill, New Jersey, and it’s still being used! A little bit of immortality.