Research connects overpublication during national sporting events to science-journalism problems

Ivan Oransky pointed me to a delightful science-based press release, “One’s ability to make money develops before birth”:

Researchers from the Higher School of Economics have shown how the level of perinatal testosterone, the sex hormone, impacts a person’s earnings in life. Prior research confirms that many skills and successes are linked to the widely known 2D:4D ratio, also knows as the digit ratio. This is the ratio of the index and ring fingers . . . research conducted by a team of scientists from HSE’s Centre for Institutional Studies (John Nye, Maria Yudkevich, Maxym Bruhanov, Ekaterina Kochergina, Ekaterina Orel, and Sergei Polyachenko) became the first study to use Russian data to show the link between the 2D:4D ratio and a person’s income. The study was published in the journal Economics and Human Biology. . . . The number of observations in the base regressions totalled nearly 700 for men and 900 for women, and the age of the subjects varied between 25 and 60. A 2D:4D ratio was made for each participant using a specialised apparatus. In addition, the respondents, whose identities remained anonymous, were asked a number of questions concerning income and salaries.

The results of the regression analysis showed a negative correlation between the income and 2D:4D ratios of women. In other words, the higher the salary, the lower the ratio. The effect was negative even when taking into account salary predictors such as gender, age, education level, job position, and the position’s economic sector. What is interesting is that this quantitative association is seen in men as well, though only after taking into account respondents’ level of education.

Savvy researchers will (a) note the challenge of taking gender into account when the analysis was performed separately for each sex, and (b) the forking-paths and difference-betweeen-significant-and-not-significant aspects of the last sentence above. Other fun things that you can see by following the link to the original paper is that the researchers looked at three different outcome measures and also tried everything with left and right hands. Also, despite the first sentence of the press release, and despite the title of the paper (“The effects of prenatal testosterone . . .”), there are actually no measurements of prenatal testosterone involved in this research (let alone any causal identification).

Bad form to put something in the title of the paper that’s not actually being measured.

I should perhaps emphasize that I have no objection to people researching such things and publishing their results; it’s just that it’s an absolute disaster to rummage around in a pile of data, pulling out things based on their statistical significance level. To put it another way: lots of their apparently statistically significant comparisons will be noise, and lots of the things that they dismiss as not statistically significant can actually correspond to real correlations. All this is made worse by rampant selection (for example, on the three different outcome measures) and the casual slipping from finger length to (unmeasured) hormone levels to (nonidentified) causal effects. What they have is some data, and that’s fine, and I think they’d be better off just publishing their dataset and being more aware of how little they can learn from it.

And of course you don’t have to be John D. Rockefeller IV to realize that “One’s ability to make money develops before birth.” No finger measurements are necessary to discover this obvious social fact.

P.S. I also love the very last paragraph of the press release:

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.

All right, then.

I went to the main Eureka Alert site and found this gem: “Research connects overeating during national sporting events to medical problems,” which begins:

People who overeat during national holidays and national sporting events – like this weekend’s Super Bowl – are 10 times more likely to need emergency medical attention for food obstruction than any at other time of the year, according to a new study led by a University of Florida researcher.

Following along, I see this:

Most of the problems affected men, and most of the cases came during or just after the Thanksgiving holiday. . . . Over the study period, from 2001 to 2012, 38 people underwent an emergency procedure on the esophagus during or just after the holiday or sporting event time period (within three days of the event). Nearly 37 percent of those were due to a food impaction. Comparatively, of the 81 who had the same procedure two weeks before and two weeks after the event during the “control period,” just under 4 percent were due to food impaction. During holidays and national sporting events, the most common impacted food was turkey (50 percent), followed by chicken (29 percent) and beef (21 percent).

I just loooove how “people stuff their faces on Thanksgiving” transmutes to “overeating during national sporting events.” Tie-in to the Super Bowl, perhaps?

This last one indeed got picked up by some bottom-feeding news organizations, for example NBC Chicago which took the bait and led with the headline,

You’re Up to 10 Times More Likely to Choke on Snacks During Super Bowl, Health Officials Say

Suckers! Hey, NBC Chicago: you got played! Pwned by the University of Florida public relations department. Pretty embarrassing, huh?

But it’s all OK, because:

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.

On the plus side, NPR doesn’t seem to have fallen for either of these stories. So there’s some hope yet.

Why bother?

As always, the question arises, why bother even drawing attention to this sort of sloppiness? The quick answer is that someone sent me a link and I found it amusing. The long answer is that that the same research and reporting problems discussed above, also arise in more consequential areas. Remember that dude who was crunching numbers and said Hillary Clinton had a 98% chance of winning the election? Or those guys who said that early childhood intervention increased future earnings by 40%? Selection bias, confusions about causality, leaps that ignore differences between data and the underlying constructs of interest, and credulous reporting that treats every published paper as a Eureka discovery: These occur in problems big and small, and when studying statistical practice, statistical reasoning, and statistical communication, it can be helpful to study the many small cases as well as the few large ones.

The post Research connects overpublication during national sporting events to science-journalism problems appeared first on Statistical Modeling, Causal Inference, and Social Science.