Freakonomics does it again (not in a good way). Jeez, these guys are credulous:

From the team that brought you “good-looking parents are 36% more likely to have a baby daughter as their first child than a baby son” and “The PDO cool mode has replaced the warm mode in the Pacific Ocean, virtually assuring us of about 30 years of global cooling” (background here and here) comes a new nugget of 24-carat credulity:

I [economist and author Steven Levitt] cannot think of an academic whose research findings have more consistently surprised me than my guest today, Harvard psychologist Ellen Langer. She’s a scientist, but her results seriously challenge the beliefs of mainstream science.

If the findings consistently surprise you, and they seriously challenge the beliefs of mainstream science, then maybe you should more seriously consider the possibility that these findings are wrong! Langer’s much-publicized work has been questioned before (for example, here and here).

Levitt continues:

I’ve got a model in my head of how the world works — a broad framework for making sense of the world around me. I’m sure you’ve got one, too. My model is, I think, pretty typical of someone who puts faith in modern science. Perhaps with a little added cynicism about human nature. So when I hear about a new research study, I have a habit of asking myself, “Given my model of the world, what results would I expect the study to generate?” Usually I’m pretty good at guessing what the researchers actually find. But with Ellen Langer, over and over and over, she gets results that I would never predict. So here are my questions for you as you listen to this conversation: First, do you find her research results as stunning as I do? And the second question I’d like you to think about is when research findings surprise you, what’s the right reaction? How do you know whether you should believe surprising results?

Good question. Excellent question. Indeed, it’s a question that often comes up with research that’s been promoted on Freakonomics.

So let’s see how Levitt handles his own question. He’s interviewing Langer on his podcast:

LEVITT: I’ve read the work of many scholars and I can honestly say that you win the prize for the body of research that most consistently finds results that are completely the opposite of what I would have predicted. You and I have completely different models of how the world works. And the data keep supporting your model. . . .

That’s not true! The data don’t keep supporting Langer’s model. Just for starters, see the two links given above, or this discussion by linguist Mark Liberman from 2009). Or my recent paper with Nick Brown. Or this recent post at The Skeptic.

This is a big, big problem. If you come into the discussion with the question, “How do you know whether you should believe surprising results?”, then you can’t just say, “the data keep supporting your model.” By making this unsupported claim about the data, you’ve given up your investigation before you’ve even begun!

Let’s go back to the interview:

LEVITT: Now, my first reaction to hearing the findings of your counterclockwise study would’ve been, “Well, that result will never be replicated. It has to be a fluke.” But it actually has been replicated a number of times, right?

LANGER: Yeah, and in different ways. I mean, to me, the important thing was the test of the mind-body unity idea. The next study in that series was a study Alia Crum and I did where we took chambermaids. And first thing, we just ask them how much exercise they get. And surprisingly, they don’t think they get any exercise because they think exercise is what you’re supposed to do after work. And after work, they’re just too tired. So for the study, what we did was very simple: we just taught half of them that their work was exercise. Different things that they’re doing — making beds, cleaning the windows, and what have you — were likened to working in different machines at the gym. And so at the end of this we found that this group wasn’t working any harder, eating any differently. Everything was basically the same as the group that wasn’t taught this change in mindset.

LEVITT: So there’s no intervention other than teaching.

LANGER: Exactly.

LEVITT: Nothing can possibly happen.

This is wrong in several ways. First, Levitt starts out by accepting that a certain suspect claim “actually has been replicated a number of times.” Going with your interviewee can make sense in a podcast, but, again, it’s counter to Levitt’s earlier goal of asking, “How do you know whether you should believe surprising results?”

Second, Levitt accepts without question Langer’s claim that “at the end of this we found that this group wasn’t working any harder, eating any differently. Everything was basically the same as the group that wasn’t taught this change in mindset.” As Brown and I discuss in our above-linked paper, that claimn of no changes in behavior is not supported by the data. The paper in question does not report any direct measures of diet and physical activity at either the start or end of the study, just information from a retrospective questionnaire. It is problematic to take survey responses as measures of actual behavior, especially in the context of a study of an intervention specifically designed to alter perceptions of exercise. Beyond this, the data in the study actually do show a large increase in perceived amount of exercise (the average going from 3.8 to 5.7 on a 0–10 scale).

OK, we’re getting into the weeds here, and I can hardly expect for Levitt to have read an unpublished paper on my website in preparation for doing this podcast. The larger point here is that he should be skeptical. He’s interviewing someone who’s produced a steady flow of counterintuitive—many would say implausible—claims, that have been much disputed. So when Langer makes a statement about one of her studies, sure, let her say her piece, but then engage your skepticism. Don’t just take her words at face value.

When Levitt states, “Nothing can possibly happen,” he’s implicitly endorsing Langer’s unsupported claims. It would be better for him to say something like, “OK, if what you say is true, that there were no behavioral changes, then nothing should be happening.” Sure, it’s just a podcast, he’s reacting in real time, and everyone makes mistakes—but, again, if your goal is to ask, “How do you know whether you should believe surprising results?”, then at some point you have to put on your skeptical hat.

There’s a lot more interview, then near the end we finally get to some pushback:

LEVITT: There’s a concept of mindfulness and we’ve talked about that. . . . And it’s completely implausible that simply by thinking or believing something different, you could make pain go away. It’s interesting that for you, those two are the same thing, but for me, they’re completely different. . .

Langer gives a long response (you can read it, it’s all in the transcript), Levitt keeps the conversation going, and then we get this:

LEVITT: I’m skeptical, at some level, of many of the things that you profess.

LANGER: Don’t say that!

LEVITT: Although sympathetic and hopeful. But what’s interesting as I just reflect, the people I know who are in their 80s and their 90s, who have influenced me the most, it is my father, my grandfather, Danny Kahneman, a few others — they all exhibit the exact features of mindfulness and aliveness that you’re describing. And I’m definitely going to add you to that list because in many ways, the most convincing thing to me about this conversation is that you embody the ideas you talk about. You are obviously mindful, and as you talk, it is exciting to talk to you and that enthusiasm feeds into my own experience. I’ll walk out of this discussion and for at least a few hours, hopefully longer, I’m going to be more mindful.

What kind of skepticism is this, Steve? You say you’re skeptical and then you immediately fold!

Levitt then summarizes:

How do you know whether to believe surprising results? I’m still grappling with that second question myself. I would love to live in the world suggested by Ellen Langer’s research. A world where I could will pain away simply through the power of thinking. A world where I could control aging gracefully. But it isn’t so easy to toss out a life’s worth of believing that those things just aren’t possible. So my own reaction to Ellen’s research is to be open minded to the possibility she’s right, and I’m actually going to spend some time practicing her approach to mindfulness and also going forward, I’m going to be very attentive to any evidence I see in my own life that supports her worldview. I haven’t really looked for that kind of evidence because I didn’t believe it was possible. But let me be honest with you, I’m a lot more open to Ellen’s research because I like the findings.

It’s good to be open. But it’s a big, big mistake to take suspect claims at face value. The moment Levitt said, “the data keep supporting your model,” he was already gone, game over.

Previously on Freakonomics

Nudges by Chopstick: From 2009, a completely uncritical plug of the work of Brian “Pizzagate” Wansink.

There’s No Such Thing as a Free Appetizer: From 2014, a completely uncritical plug of the work of Brian “Pizzagate” Wansink.

A bunch of completely uncritical plugs of Dan “Shreddergate” Ariely.

Are Cornell Students Psychic?: From 2010, a completely uncritical report of the notorious ESP experiments by Daryl Bem.

It makes me sad

I write some of this in a jocular tone, only because that’s one way for me to deal with it. I laugh because that’s better than crying. In all seriousness, I think that experimental science can improve our lives, and it frustrates me when bad science takes up the space that could be occupied by good science.

And Steven Levitt . . . he’s got the training and experience to evaluate scientific claims! He could read Langer’s papers, he could download and reanalyze what data are available, he could google search for replications and criticisms, he could read what Mark Liberman and others had to say, etc. He doesn’t have to do that work, but, if he’s gonna ask, “How do you know whether to believe surprising results?”, then he should. Otherwise, why bother? Langer’s been interviewed a million times already; what’s the point of one more puff piece? I just don’t get it. Levitt can play a useful role in the conversation here, and he chooses not to. Really frustrated.

Again, I think it’s just fine that Langer and her colleagues do their research—not that they need my approval or endorsement! Speculative studies are part of the research ecosystem. My problem is not with high-risk, potential high-return research; my problem is the misrepresentation of scientific evidence.