Take two on Laura Arnold’s TEDx talk.

This post is by Keith.

In this post I try to be more concise and direct about what I found of value in Laura Arnold’s TEDx talk that I recently blogged about here.

Primarily it was the disclosure from someone who could afford to buy good evidence (and experts to assess it) that they did not think good enough evidence was actually available right now. Yes there is some real gold among the research outputs – but it just cannot be reliably distinguished from fools gold at present – so don’t buy now or buy at your own risk. Furthermore, believing high quality evidence was necessary in any reasonable attempt to make things better, there was an argument/decision that evidence quality needed to be rectified first. And at the end of their talk they did not blame reporters or careerist researchers but instead claimed it was _our_ fault and that _we_ needed to be prepared to contribute to it being resolved.

That it is up to the masses to fix the current faulty evidence generating machines very much rings true to me. I think that was an important turning point the AllTrials initiative “It immediately occurred to Tracey [Director, Sense about Science] that there had to be thousands of people who took part in clinical trials who felt “I want my participation to count for something—I want the results that you who ran the trial got because I took part to be used.” Here maybe – “I want people who might want to help me to actually have good evidence on how do that”.

I also have more nuanced or speculative take.

p.s. jrc added some comments that seemed to nicely clarify some points I was trying to make.

In particular this one – “Arnold Foundation ultimately cares about facts/relationships in the world, not methods, but has to care about methods because they can’t get the facts they want with the methods people are using.”  Right – that’s the point – if they can’t get the facts [evidence] they want with the methods people are using that’s a problem for _all of us_ [the masses]. Given its a problem for all of us (not just those of us working in research), I think it would be helpful if all of us were better informed about it and made some sort contribution to fixing the problems.

When research consumers fund what I called nuisance research (research to help others do better interest research) there is no downside of an implied admission of needing to do such research. (An annoyance yes, as they rather not have to do it, but no personal or professional embarrassment about having to do it.) On the other hand, now think of research producers funding nuisance research (research to help themselves do better interest research) – that seems like a disclosure of not currently knowing how to do the best interest research. Unfortunately, seemingly credible if not incontestable claims of currently knowing how to do the best interest research possible may be perceived as an absolutely necessity. Such claims are required by universities to support their faculty and present them as important assets to others, by funding agencies to justify who they fund, by faculty themselves to maximise their funding, publication and promotion opportunities, etc. So amongst many research producers there may be insurmountable impediments to any noticeable attempts to rectify the current inadequate quality of research outputs.

Now, I do remember running into this at the University of Toronto when I was looking into ways to provide a statistical/methodological resource for all faculty members (in 1997). It went sort like this – “our faculty are quite capable of doing research on their own – they would not be here if they weren’t!”.  Because of this and others reasons, we went with what we called and “After hours induction club” hosted at the university’s faculty club in the evening so that interested faculty could assess the resource “discretely”. There was an invited statistical expert who gave a short talk on areas they would be interested in applying in practice and then questions and informal discussion followed afterwards. Unfortunately, I left the University of Toronto very shortly after setting this up and only one occurred – it did seem to work well.

It also explains the extremely poor record of the funding of biostatistics in Canada -“Funding for biostatisticians comes from … the Canadian Institutes of Health Research (CIHR) for the clinical research. On the other hand, CIHR rarely funds the advancement of methodological research.”

Perhaps more importantly, given research consumers want to fund what I called nuisance research they will likely need to fund research producers to do this – who else would know how? The concern about this is that some percentage of them may be too well set in their ways as always promoting themselves as the best at currently knowing how to do the best nuisance research given they have successfully “faked” currently knowing how to do the best interest research for most of their careers. Those who have not maintained that reputation will unlikely still be a research producer. Not all or even many will have been “faking it” – but some will have.

 

The post Take two on Laura Arnold’s TEDx talk. appeared first on Statistical Modeling, Causal Inference, and Social Science.