“Making Statistics Work: Information Theory and Bayesian Inference”

I took a look at the above-titled book by economists Duncan Foley and Ellis Scharfenaker. It’s an interesting read, in many ways a throwback to the 1950s when a group of mathematicians brewed a heady mix of operations research, game theory, probability theory, and economics in an attempt to create a unified theory of social science, or to map the limitations of this effort. Important figures in this effort include John Maynard Keynes, John Von Neumann, Jimmie Savage, Milton Friedman, Duncan Luce, Howard Raiffa, Kenneth Arrow, Herbert Simon, Ed Jaynes, . . . a whole bunch of people who are still remembered today.

Back in the day, Bayes was seen alternately as Jesus or the Devil, and there were hopes of a grand synthesis of subjective probability and local information in markets, a connection between formal statistical inference and individual decision making.

In retrospect, the cognitive science of the 1950s wasn’t all there, and hierarchical modeling hadn’t been integrated into Bayesian inference. Also some key pieces such as posterior predictive checking and general-purpose Bayesian computing weren’t there. So any attempts at unification were premature. Not that it was a bad idea to try! Much is learned from incomplete efforts. It’s just clear in retrospect that any unified theories of the time were bound to fail.

From the perspective of seventy years later, we can see Bayesian inference as a useful part of the statistical toolkit, a way to place regularization (a central part of all modern machine learning) in the context of scientific modeling. Many problems that can be solved with an entirely Bayesian approach, and others can be viewed as approximate Bayes–or, to put it another way, Bayesian ideas can help with all sorts of statistical modeling problems, even when other inferential methods are used.

What I’m saying is, it’s a good idea for everyone doing statistics or machine learning to understand the basics of Bayesian inference and computation, prior and predictive checking, and Bayesian model expansion, for their own sake and also as a way to make sense of statistical learning.

“Making Statistics Work: Information Theory and Bayesian Inference” is one of the most unusual statistics books I’ve ever read. I don’t agree with much of it–for example, right on the second page they start talking about “prior beliefs,” which isn’t how I think of things at all (see here and here)–and it’s written in a mathematical style which seems old-fashioned to me but has a kind of charm. You could almost say that it’s the statistics book that William Feller would’ve written had he been converted to Bayesianism.

What I really like is that the book is what it is–an forthright attempt at a modern expression of the aimed 1950s synthesis of mathematical statistics, physics, and economics. It clocks in at a crisp 300 pages. It has zero overlap with Bayesian Data Analysis and Bayesian Workflow, and that’s just fine. As I said, I don’t really buy their synthesis myself, but I respect their attempt. You can judge it as you will.