Science communication is not a one-shot game

In our recent discussion of Ted doubling down on power pose, commenter Michael raised an interesting question:

I think the general attitude of most people who work on communicating science to the public is that their responsibility is only to make sure that any information they present has a source with the proper credentials (published in a peer-reviewed journal, endorsed by PhD experts in the relevant disciplines at universities). Since they are not themselves PhD experts, the feeling is that “Who am I to challenge this expert? I am just telling you what my expert says, it’s not my job to get involved in these obscure internal arguments”. . . . If Slate can let Andrew Gelman write an article, or Retraction Watch can publish an interview with him expressing his position without publishing comments from experts with objectively equal qualifications who disagree, why can’t TED let Amy Cuddy put out her ideas? How should someone outside of the relevant disciplines be expected to know when what an expert is saying needs to be challenged? I can’t think of a good solution.

I replied as follows:

One difference between Cuddy’s Ted talk and my Slate articles is that I take the other side of the argument seriously, even if I express disagreement.

For example, today in Slate I looked into Jon Krosnick’s claim that the outcome of the 2016 election was determined by Trump being listed first on the ballot in many swing states. I concluded that it was possible but that I was skeptical that the effects would’ve been large. True, Slate did not invite Krosnick to respond. But in my article I linked to Krosnick’s statement, I clearly stated my sources of evidence, I linked and took seriously a research article by Krosnick and others on the topic . . . I did my due diligence.

In contrast, the Ted team avoids linking to criticisms of Cuddy’s work, and I do not consider her statements to be in the full spirit of scientific inquiry. It seems like a damage control operation more than anything else. As to the original Carney, Cuddy, and Yap article: as I noted above, it makes a claim in the abstract that is not supported by anything in the paper. And more recently Carney gave a long list of problems with the paper, which again Cuddy is not seriously addressing.

This response is fine as far as it goes, but I realized something else is going on, which is that Slate and Ted and other media outlets get multiple chances. If Jon Krosnik can make a strong case that my skepticism in his theory is misplaced, he can write about it—and even if Slate doesn’t run Krosnick’s (hypothetical) right away, they’d certainly be aware of it if they were to do more reporting on ballot-order effects. What about Ted? It’s hard to fault them for greenlighting Cuddy’s talk: at the time, the Carney/Cuddy/Yap paper had not been widely criticized, the failed replications were still in the future, and Cuddy had that Harvard credential. So, fine. My problem with Ted comes later, when they continue to endorse the work—and, more recently, to go all-in on it—in a way that dodges all criticism. I can’t expect typical journalistic outlets to discover flaws in research claims that already managed to get past peer review. But I can fault them for not updating their priors in light of new information.

Journalism, like science and bookmaking, is a multiple-round game, and the big fails can come from not knowing when to cut your losses.

The post Science communication is not a one-shot game appeared first on Statistical Modeling, Causal Inference, and Social Science.