Shreddergate! A fascinating investigation into possible dishonesty in a psychology experiment

From A to Z

A couple years ago we discussed a post by Mark Zimbelman expressing skepticism regarding a claim by psychologist (also business school professor, Ted talk star, NPR favorite, Edge Foundation associate, retired Wall Street Journal columnist, Founding Partner of Irrational Capital, insurance agent, bestselling author, etc etc) Dan Ariely. Zimbelman wrote:

I have had some suspicions about some experiments that Professor Ariely ran using a shredder that was modified so it looked like it was shredding but really wasn’t . . . He claimed it was “quite simple” to convert a shredder by breaking the teeth in the middle with a screwdriver . . . We were unable to break any teeth on the shredders we purchased but ended up finding a way to remove some of the teeth in the center by taking the shredder apart. Unfortunately, when we did this the papers would no longer go through the shredder without getting turned to one side or another and they inevitably got stuck because the shredder no longer had enough teeth to pull them through. We concluded that it was impossible to modify any of the shredders we bought . . .

A couple weeks after his first post, Zimbelman followed up with further investigation:

I [Zimbelman] did an extensive literature search (involving several others who helped out) looking for the research that he claims was done with the modified shredder. The end result is that I can’t find any published paper that discusses using a modified shredder. I even called one of his co-authors and asked him if the experiment that they ran together used a modified shredder. He said the shredder in their study was not modified.

I did find a few papers that used a regular shredder but did not mention any modifications. I also found several statements (including this one and the one linked above) where he claims to use this mysterious modified shredder. Overall then, here’s where we are with the shredder:

1. Dr. Ariely has made numerous claims to use a modified shredder in his matrix experiments.

2. I am unable to find any published papers by Dr. Ariely that use a modified shredder.

3. Modifying a shredder to do what he has claimed appears to be very unlikely.

Zimbelman’s posts were from late 2021, and I reported on them in early 2022. That’s where things stood for me until an anonymous tipster pointed us to this video that Ariely posted to Youtube in September 2023, along with this note:

Over the years I ran many different versions of honesty/ dishonesty experiments. In one of them, I used a shredder. Here is a short piece from the Dishonesty movie in which this shredder is starring and can be seen in action.

Here are some screenshots:

The Dishonesty movie is from 2015. Here’s a press release from 2016 promoting the movie. The press release says that the experiment that used the shredder was performed in 2002.

I assume the scene in the movie showing the experiment was a reconstruction. It just seems doubtful that in 2002 they would’ve taken these videos of the participants in that way. There’s nothing dishonest about reconstructing a past scene—documentaries do that all the time!—I’m just trying to figure out exactly what happened here. Just to be sure, I watched the clip carefully for any clues about when it was shot . . . and I noticed this:

Let’s zoom in on that mobile phone:

I don’t know anything about mobile phones, so I asked people who did, and they assured me that the phones in 2002 didn’t look like that, which again suggests the scene in the documentary was a reconstruction. Which is still fine, no problems yet.

Home Depot . . . or Staples?

Zimbelman’s post links to a voicemail message from Ariely saying they bought the shredder from Home Depot. But in the video, the shredder is labeled Staples:

Wassup with that? In the label on the Youtube video, Ariely says that the shredder in the video is the same as the one they used. (“Over the years I ran many different versions of honesty/ dishonesty experiments. In one of them, I used a shredder. Here is a short piece from the Dishonesty movie in which this shredder is starring and can be seen in action.”) Which makes sense. It’s not like they’d have a whole bunch of modified shredders kicking around. But then where did they buy it? Back in 2002, was Home Depot carrying Staples brand products? I guess a real shredder-head could tell just by looking whether the shredder in the video is vintage 2002 or 2015. The shredder in the video doesn’t look 13 years old to me, but who am I to say? Unfortunately, the image of the label on the back of the machine isn’t readable:

You can make out the Staples brand name but that’s about it.

So, what happened?

There are several possibilities here:

1. The 2002 experiment happened as was described, they really modified that shredder as claimed, they kept the shredder around, it was still operating in 2015, they demonstrated it in the video, and it worked as advertised. Also, that shredder in 2002 had actually been bought at Staples, and Ariely just had a lapse in memory when he said they’d bought it at Home Depot.

2. The 2002 experiment happened as was described, they really bought that shredder at Home Depot and modified it as claimed, but then the shredder was lost, or discarded, or broke. When they made the movie, they went to Staples and bought a new shredder, modified it (using that approach which Ariely said is simple but which Zimbelman in the above-linked post said is actually difficult or impossible), and it worked just as planned. Under this scenario, Ariely was not telling the truth when he wrote that the shredder is the same as in the original experiment, but, hey, it’s just a movie, right? The Coen brothers had that title card saying that Fargo was based on a true story, and it wasn’t—that doesn’t mean they were “lying,” exactly!

3. The 2002 experiment happened as was described, they really bought that shredder at Home Depot and modified it as claimed, but then the shredder was lost, or discarded, or broke. When they made the movie, they went to Staples and bought a new shredder, but at this point they didn’t bother to try to modify it. It was just a dramatization, after all! Why ruin a perfectly good shredder from Staples? Instead they just filmed things just as they literally look in the video: they put paper in the shredder, then off screen they mangle the edges of some other sheets of paper, put them in the bin in the bottom of the shredder, and then turn on the video again and take the partially-mangled papers out. The way I wrote this, it seems kinda complicated, but if you think about it from the standpoint of someone making a video, this is much easier than getting some modified shredder to work! Indeed, if you look at the video, the sheets of paper that went into the shredder at the beginning are not the same as the ones that they took out at the end.

4. The 2002 experiment never happened as described. Ariely or one of his collaborators had that cool idea of modifying the shredder, but it wasn’t so easy to do so they gave up and just faked the study. Then when making the 2015 movie they just said they lost the shredder, and the moviemakers just did the reenactment as described in option 3 above.

5. I’m sure there are other possibilities I didn’t think of!

Is the video “fake”? I doubt it! I assume the video is either a clip from the movie or was filmed at the same time as the movie, and in either case it’s a reenactment of Ariely’s description of what happened in the experiment. In a reenactment if you show some sheets of paper fed into a shredder and then later you show some sheets of paper removed from the shredder, there’s no reason that they have to be the same sheets of paper. Similarly, if a movie reenacts a flight from New York to Chicago, and it shows a shot of a plane taking off from LaGuardia, followed by a shot of a plane landing in O’Hare, they don’t have to be the same plane. It’s just a reenactment! The fact that someone made a video reenacting a scene with a shredder does not imply that this shredding actually happened in the video—there’s really no reason to for them to have gone to the trouble to have done this shredding at all.

So, there are several possibilities consistent with the information that is currently available to us. You can use your own judgment to decide what you think might have happened in 2002 and 2015.

“Dishonesty can permeate through a system and show up not because of selfish interest but because of a desire to help.”

In the above-linked press release—the one that said, “we modified the shredder! We only shredded the sides of the page, whereas the body of the page remained intact”—Ariely also said:

There are pressures for funding. Imagine you run a big lab with 20 people and you’re about to run out of funding – what are the pressures of taking care of the people who work with you? And what kind of shortcuts would you be willing to take? Dishonesty can permeate through a system and show up not because of selfish interest but because of a desire to help.

With regards to research, I don’t think that most people think long-term and think to themselves that somebody would try to replicate their results and find that they don’t work. People often tend to “tweak” data and convince themselves that they are simply helping the data show its true nature. There are lots of things in academic publications that are manifestations of our abilities to rationalize. . . .

I think that the pressures of publication, funding, helping the group, and reputation are very much present in academic publications.

Which is interesting given the fraudulent projects he was involved in. Interesting if he was doing fraud and interesting if he was the unfortunate victim of fraud. He’s an admirably tolerant person. Fraud makes me angry; I would not be so quick to refer to it as “not because of selfish interest but because of a desire to help.”

Wassup, NPR? You raised concerns in 2010 but then promoted

One of the above sources links to this NPR article from 2010:

Should You Be Suspicious Of Your Dentist Or NPR’s Source? . . .

Last month, Dan Ariely, a behavioral economics professor, talked with All Things Considered host Robert Siegel, about how incredibly loyal, almost irrationally so, people are to their dentists – more so than with other medical professions. . . . when he appeared on NPR’s air, there was every reason to trust him.

Ariely offered information certain to unnerve listeners and anger dentists ¬– information based on a fact that he cannot back up.

If two dentists were asked to identify cavities from the same X-ray of the same tooth, Ariely said they would agree only 50 percent of the time.

Ariely cited Delta Dental insurance as his source. However, Delta spokesman Chris Pyle said there is no data that could lead to that conclusion. . . .

Here is what Ariely said:

Prof. ARIELY: And we asked both dentists to find cavities. And the question is, what would be the match? How many cavities will they find, both people would find in the same teeth? . . . It turns out what Delta Dental tells us is that the probability of this happening is about 50 percent. . . .

It’s really, really low. It’s amazingly low. Now, these are not cavities that the dentist finds by poking in and kind of actually measuring one. It’s from X-rays. Now, why is it so low? It’s not that one dentist find cavities and one doesn’t. They both find cavities, just find them in different teeth. . . .

“According to Dr. Ariely, he was basing his statement on a conversation he said he had with someone at Delta Dental,” said Pyle. “But he cannot cite Delta Dental in making that claim because we don’t collect any data like that which would come to such a conclusion.”

So what happened?

Ariely said he got that 50 percent figure from a Delta source who told him about “some internal analysis they have done and they told me the results. But they didn’t give me the raw data. It’s just something they told me.”

Ariely did not provide the name of the Delta medical officer, whom Ariely said was not interested in talking with me. . . .

Ariely told me he happened upon that figure when he was conducting research analyzing 20 years of raw data on Delta claims. . . . But Ariely did not see or analyze any data that would lead to a conclusion that dentists would agree only 50 percent of the time based on studying an X-ray.

Wow. The NPR report continues:

But what is NPR’s responsibility? . . . NPR can’t re-report and check out every thing that an on-air guest says. . . . In this case, the interview with Ariely was taped ahead of time and edited for air – but no one thought it necessary to challenge his undocumented statement.

ATC executive director Christopher Turpin said NPR had no reason to question Ariely, given his credentials as a tenured professor and an expert on how irrational human beings are. . . .

ATC has other pre-taped segments with Ariely, and those should be double-checked before they are aired. There’s no doubt that Ariely is both entertaining and informative about how irrational we humans are — but he also must be right.

It’s funny that, after all that, they write that there’s “no doubt” that Ariely is informative. I have some doubt on that one!

But here’s the interesting thing. The above warning was from 2010. Not 2022, not 2020. 2010. Fourteen years ago. But if you google *NPR Ariely*, you get a bunch of items since then:

2011: Is Marriage Rational? : Planet Money

2011: For Creative People, Cheating Comes More Easily

2012: TED Radio Hour: Dan Ariely: Why Do We Cheat?

2012: ‘The Honest Truth’ About Why We Lie, Cheat And Steal

2014: Dan Ariely: Where’s The Line Between Cheating A Little and Cheating A Lot?

2014: Rethinking Economic Theory: The Evolutionary Roots Of Irrationality : 13.7: Cosmos And Culture

2015: Dan Ariely: What Pushes Us To Work Hard — Even When We Don’t Have To?

2017: Dan Ariely: When Are Our Decisions Made For Us?

2018: Everybody Lies, And That’s Not Always A Bad Thing

2020: Why Some People Lie More Than Others

And then in the past year or so there have been some skeptical stories, such as this from 2023: Did an honesty researcher fabricate data?

But until recently NPR was running lots and lots of stories quoting Ariely completely without question—for years and years after they’d been explicitly warned back in 2010.

The scientist-as-hero narrative is just so strong that NPR kept going back to that well even after their own warning.

Why post on this?

God is in every leaf of every tree. It’s interesting how the more you look into this shredder story the more you can find. But there’s always some residual uncertainty, because there’s always some elaborate explanation that we haven’t thought of.

In the meantime, I’d recommend following the advice of that 2010 NPR report and asking people for their evidence when they make claims of scientific breakthroughs. There’s nothing you can do to stop people from flat-out lying, but if you can get purported experts to specify their sources, that should help.

If scientists know ahead of time that they’re expected to produce the shredder, as it were, maybe they’d be less likely to make things up in the first place.