Evaluating hypotheses: We are all Bayesians now (for appropriate definitions of “Bayesians”)

(lifted and edited from Phil Birnbaum’s comment section)

You should consider all relevant evidence when evaluating hypotheses. This seems an uncontroversial statement, even among journal editors. Is this necessarily Bayesian? Depends on one’s definition of Bayesianism, but to me the term implies something quantitatives: the use of Bayes’ theorem. If we consider any argument that goes outside the data Bayesian, the term seems too broad to be useful. In particular, if “Bayesianism” is used as an umbrella for any use of subjectivity, well, philosophers have been pointing that out for centuries that science can’t be entirely subjective. It’s necessary, however, to make clear what’s objective and what isn’t;  for scientists to use subjective priors (which, to be clear, few Bayesians endorse) obfuscates the difference. On the other hand, I’m totally on board with broadening the definition of “evidence”, though informal evidence should be used informally.

One thing that may or may not be relevant is it doesn’t matter what order you do the conditioning in. That is, in theory summarising all available evidence in a prior and then adjusting for the result of a new experiment gives the same posterior as starting with the experiment result then adjusting for all other evidence. Since there’s rarely an objective prior, you should post all the data and let anyone who wants to update their posterior do so. In practice, humans have all kinds of cognitive biases, not to mention they’re generally not great at integration. You should post the data, but you should help your readers out by providing informative and honest summaries of the data. Hypothesis tests can be nice, but graphs are often more useful.