The point of yesterday’s post on the three ways of attacking a statistical problem

I fear that people may have gotten lost in the details of the data and code for the football won/lost example, so I wanted to clarify why I wrote the post.

In general there are three ways of attacking a statistical problem:

1. Probability calculation. Set up a probability model and crank it through. This will require a bunch of assumptions, and you’ll also need to set parameters in your model to reasonable values.

2. Direct empirical calculation. This will work if you have enough data, and if these data are not subject to selection.

3. Statistical modeling. Kind of like method 1 above, except that you fit (“learn”) the parameters from the data; as a result you can fit a more complicated model. I include machine learning in this category too.

In statistics classes, we focus on method 3. No surprise, right? Statistical modeling encompasses probability calculation and direct empirical calculation; indeed methods 1 and 3 can be viewed as special cases of method 3. Method 1 is method 3 but with a simple model and a crude method of setting the parameters. Method 2 is method 3 but with a simple model assuming stationarity in all directions.

So, yeah, statistical modeling. There’s a reason my colleagues and I have written multiple books on the topic, spent innumerable person-hours developing and using Stan, etc.

But . . . it’s good to know about methods 1 and 2 as well.

Why? Four reasons.

First, methods 1 and 2 are simpler, and sometimes they work just fine.

Second, even beyond simplicity, methods 1 and 2 have fewer requirements. Method 1 does not require the data (which is how we were able to get a good answer to that football question in the first place), and method 2 does not require a data-generation model. In contrast, method 3 requires data and a model.

Third, even when estimates based on methods 1 and 2 are seriously flawed, they can be useful starting points and comparison points to better approaches. Indeed, sometimes when a probability calculation gives a ridiculous result, this can be useful in developing intuition. For example, the notorious calculation of a probability of a tied election as 10^-90 came from an inappropriate application of a binomial-distribution model, which motivates the development of models for statistical dependence among voters, while the failure of straight-up empirical estimates motivates models that combine probability modeling and empirics.

Fourth, when people are informally estimating things, they’re often using some version of method 1 or 2. Which is fine! But then I think it’s important to be aware of what you’re doing and to ask, What is the probability model you are assuming, or What is the frequency calculation you are making?

Those four reasons–that’s the point of yesterday’s post.