The Danger of Face Validity - The Scholarly Kitchen

"... the standard explanation for Van Halen’s M&M rider — that it was a classic expression of bloated rock privilege — is a hypothesis with a great deal of face validity: it simply makes good intuitive sense, and is therefore easy to accept as true.

Like many hypotheses with a great deal of face validity, however, it turns out to be wrong. Not just imprecise or lacking in nuance, but simply wrong. The reason that the members of Van Halen put the M&M rider into their contract had nothing to do with exploiting their privilege or with an irrational aversion to a particular color of M&M. It had to do with the band’s onstage safety. As it turns out, other provisions of the band’s contract required the venue to meet certain safety standards and provide certain detailed preparations in terms of stage equipment; without these preparations, the nature of the band’s show was such that there would have been significantly increased danger to life and limb. The M&M rider was buried in the contract in such a way that it would easily be missed if the venue’s staff failed to read the document carefully. If the band arrived at a venue and found that there was a bowl of M&M’s in the dressing room with all the brown ones removed, they could feel confident that the entire contract had been read carefully and its provisions followed scrupulously — much more confident than they would have been if they had simply asked the crew 'You followed the precise rigging instructions in 12.5.3a, right?' and been told 'Yes, we did.'

What does this have to do with scholarly communication? Everything. In scholarly communication (as in just about every other sphere of intellectual life), we are regularly presented with propositions that are easy to accept because they make obvious sense. Sometimes these are accompanied by rigorous data; too often they are supported by sloppy data or anecdotes. Sometimes they aren’t supported at all, but are simply presented as self-evidently true because their face validity is so strong.

In scholarly communication, we are regularly presented with propositions that are easy to accept because they make obvious sense.

A classic example is the citation advantage of open access (OA) publishing. It seems intuitively obvious that making a journal article freely available to all would increase both its readership and (therefore) the number of citations to it, relative to articles that aren’t free. By this reasoning, authors who want not only broad readership but also academic prestige should urgently desire their articles to be as freely available as possible. But the actual data demonstrating the citation impact of OA is mixed at best, and the reality and significance of any OA citation advantage remains fiercely contested (for example, here, here, here, here, here, here, here, and here)."