“I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion.”

In a post entitled, “Just make it stop! When will we say that further research isn’t needed?,” Dorothy Bishop writes:

On Friday we had a great presentation from Lottie Anstee who told us about her Masters project on handedness and musicality. There have been various studies on this topic over the years, some claiming that left-handers have superior musical skills, but samples have been small and results have been mixed. Lottie described a study with an impressive sample size (nearly 3000 children aged 10-18 years) whose musical abilities were evaluated on a detailed music assessment battery that included self-report and perceptual evaluations. The result was convincingly null, with no handedness effect on musicality.

Interesting. Handedness is a funny area of research: the idea of correlation between handedness and various abilities makes a lot of intuitive sense; we are exposed to lots of anecdotal evidence on lefthandedness; the topic lends itself to theorizing; it’s cheap to gather data on handedness, but there are not a lot of good public data sources. Put all this together and you get a literature full of suggestive but shaky findings based on noisy measurements and small samples. We discuss an example in pages 220-222 of Active Statistics.

What I’m saying is, it’s plausible to me that there could be interesting systematic differences between lefties and righties (beyond differences in ability to write, draw, throw, etc., with the non-dominant hand), and it’s also plausible that there are no other stable differences between these groups of people.

Bishop continues the story:

What happened next was what always happens in my experience when someone reports a null result. The audience made helpful suggestions for reasons why the result had not been positive and suggested modifications of the sampling, measures or analysis that might be worth trying . . . perhaps a more nuanced measure would reveal an association? Should the focus be on skilled musicians rather than schoolchildren? Maybe it would be worth looking at nonlinear rather than linear associations? And even though the music assessment was pretty comprehensive, maybe it missed some key factor – amount of music instruction, or experience of specific instruments.

After a bit of to and fro, I [Bishop] asked the question that always bothers me. What evidence would we need to convince us that there is really no association between musicality and handedness? The earliest study that Lottie reviewed was from 1922, so we’ve had over 100 years to study this topic. Shouldn’t there be some kind of stop rule?

Bishop summarizes:

My own view is that further investigation of this association would prove fruitless. In part, this is because I think the old literature (and to some extent the current literature!) on factors associated with handedness is at particular risk of bias, so even the messy results from a meta-analysis are likely to be over-optimistic. . . .

I [Bishop] suspect that most of the exciting ideas about associations between handedness and cognitive or personality traits are built on shaky foundations, and would not replicate if tested in well-powered, preregistered studies. But somehow, the idea that there is some kind of association remains alive, even if we have a well-designed study that gives a null result.

I agree 100% on that last point regarding meta-analysis.

And she concludes:

I hope that as preregistration becomes more normative, we may see more null results getting published, and learn to appreciate their value. But I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion.

It’s an interesting question. One useful way to look at it, I think, is to recognize that each of us can make our own decision of when to give up on a field. Bishop can give up on studies of handedness and behavior, but this shouldn’t stop others from continuing to study it. Major funders can stop paying for research on this topic, but researchers can continue to work on it using their own funds, etc. Even cold fusion still gets some funding—it’s the whole high-risk, high-reward thing. In answer to Bishop’s question, “Shouldn’t there be some kind of stop rule?,” my response is that each of us has our own stop rule, and that’s a good thing.

To put it another way, we should be open to the possibility that a research area is a dead end. The history of science is full of wrong turns, many of which have happened during our professional lifetimes.

P.S. As a fun bonus, follow the link to see Bishop expressing disappointment with the theories of Noam Chomsky:

As someone who works on child language disorders, I [Bishop] have tried many times to read Chomsky in order to appreciate the insights that he is so often credited with. I regret to say that, over the years, I have come to the conclusion that, far from enhancing our understanding of language acquisition, his ideas have led to stagnation, as linguists have gone through increasingly uncomfortable contortions to relate facts about children’s language to his theories.

Good stuff, including some debate in the comment section.