A Bayesian probability worksheet

What's new 2022-10-08

This is a spinoff from the previous post. In that post, we remarked that whenever one receives a new piece of information ${E}$ , the prior odds ratio ${\mathop{\bf P}( H_1 ) / \mathop{\bf P}( H_0 )}$ between an alternative hypothesis ${H_1}$ and a null hypothesis ${H_0}$ is updated to a posterior odds ratio ${\mathop{\bf P}( H_1|E ) / \mathop{\bf P}( H_0|E )}$ , which can be computed via Bayes’ theorem by the formula

$\displaystyle \frac{\mathop{\bf P}( H_1|E )}{\mathop{\bf P}(H_0|E)} = \frac{\mathop{\bf P}(H_1)}{\mathop{\bf P}(H_0)} \times \frac{\mathop{\bf P}(E|H_1)}{\mathop{\bf P}(E|H_0)}$

where ${\mathop{\bf P}(E|H_1)}$ is the likelihood of this information ${E}$ under the alternative hypothesis ${H_1}$ , and ${\mathop{\bf P}(E|H_0)}$ is the likelihood of this information ${E}$ under the null hypothesis ${H_0}$ . If there are no other hypotheses under consideration, then the two posterior probabilities ${\mathop{\bf P}( H_1|E )}$ , ${\mathop{\bf P}( H_0|E )}$ must add up to one, and so can be recovered from the posterior odds ratio ${o := \frac{\mathop{\bf P}( H_1|E )}{\mathop{\bf P}(H_0|E)}}$ by the formulae

$\displaystyle \mathop{\bf P}(H_1|E) = \frac{o}{1+o}; \quad \mathop{\bf P}(H_0|E) = \frac{1}{1+o}.$

This gives a straightforward way to update one’s prior probabilities, and I thought I would present it in the form of a worksheet for ease of calculation:

A PDF version of the worksheet and instructions can be found here. One can fill in this worksheet in the following order:

In Box 1, one enters in the precise statement of the null hypothesis ${H_0}$ .
In Box 2, one enters in the precise statement of the alternative hypothesis ${H_1}$ . (This step is very important! As discussed in the previous post, Bayesian calculations can become extremely inaccurate if the alternative hypothesis is vague.)
In Box 3, one enters in the prior probability ${\mathop{\bf P}(H_0)}$ (or the best estimate thereof) of the null hypothesis ${H_0}$ .
In Box 4, one enters in the prior probability ${\mathop{\bf P}(H_1)}$ (or the best estimate thereof) of the alternative hypothesis ${H_1}$ . If only two hypotheses are being considered, we of course have ${\mathop{\bf P}(H_1) = 1 - \mathop{\bf P}(H_0)}$ .
In Box 5, one enters in the ratio ${\mathop{\bf P}(H_1)/\mathop{\bf P}(H_0)}$ between Box 4 and Box 3.
In Box 6, one enters in the precise new information ${E}$ that one has acquired since the prior state. (As discussed in the previous post, it is important that all relevant information ${E}$ – both supporting and invalidating the alternative hypothesis – are reported accurately. If one cannot be certain that key information has not been withheld to you, then Bayesian calculations become highly unreliable.)
In Box 7, one enters in the likelihood ${\mathop{\bf P}(E|H_0)}$ (or the best estimate thereof) of the new information ${E}$ under the null hypothesis ${H_0}$ .
In Box 8, one enters in the likelihood ${\mathop{\bf P}(E|H_1)}$ (or the best estimate thereof) of the new information ${E}$ under the null hypothesis ${H_1}$ . (This can be difficult to compute, particularly if ${H_1}$ is not specified precisely.)
In Box 9, one enters in the ratio ${\mathop{\bf P}(E|H_1)/\mathop{\bf P}(E|H_0)}$ betwen Box 8 and Box 7.
In Box 10, one enters in the product of Box 5 and Box 9.
(Assuming there are no other hypotheses than ${H_0}$ and ${H_1}$ ) In Box 11, enter in ${1}$ divided by ${1}$ plus Box 10.
(Assuming there are no other hypotheses than ${H_0}$ and ${H_1}$ ) In Box 12, enter in Box 10 divided by ${1}$ plus Box 10. (Alternatively, one can enter in ${1}$ minus Box 11.)

To illustrate this procedure, let us consider a standard Bayesian update problem. Suppose that a given point in time, ${2\%}$ of the population is infected with COVID-19. In response to this, a company mandates COVID-19 testing of its workforce, using a cheap COVID-19 test. This test has a ${20\%}$ chance of a false negative (testing negative when one has COVID) and a ${5\%}$ chance of a false positive (testing positive when one does not have COVID). An employee ${X}$ takes the mandatory test, which turns out to be positive. What is the probability that ${X}$ actually has COVID?

We can fill out the entries in the worksheet one at a time:

Box 1: The null hypothesis ${H_0}$ is that ${X}$ does not have COVID.
Box 2: The alternative hypothesis ${H_1}$ is that ${X}$ does have COVID.
Box 3: In the absence of any better information, the prior probability ${\mathop{\bf P}(H_0)}$ of the null hypothesis is ${98\%}$ , or ${0.98}$ .
Box 4: Similarly, the prior probability ${\mathop{\bf P}(H_1)}$ of the alternative hypothesis is ${2\%}$ , or ${0.02}$ .
Box 5: The prior odds ${\mathop{\bf P}(H_1)/\mathop{\bf P}(H_0)}$ are ${0.02/0.98 \approx 0.02}$ .
Box 6: The new information ${E}$ is that ${X}$ has tested positive for COVID.
Box 7: The likelihood ${\mathop{\bf P}(E|H_0)}$ of ${E}$ under the null hypothesis is ${5\%}$ , or ${0.05}$ (the false positive rate).
Box 8: The likelihood ${\mathop{\bf P}(E|H_1)}$ of ${E}$ under the alternative is ${80\%}$ , or ${0.8}$ (one minus the false negative rate).
Box 9: The likelihood ratio ${\mathop{\bf P}(E|H_1)/\mathop{\bf P}(E|H_0)}$ is ${0.8 / 0.05 = 16}$ .
Box 10: The product of Box 5 and Box 9 is approximately ${0.32}$ .
Box 11: The posterior probability ${\mathop{\bf P}(H_0|E)}$ is approximately ${1/(1+0.32) \approx 75\%}$ .
Box 12: The posterior probability ${\mathop{\bf P}(H_1|E)}$ is approximately ${0.32/(1+0.32) \approx 25\%}$ .

The filled worksheet looks like this:

Perhaps surprisingly, despite the positive COVID test, the employee ${X}$ only has a ${25\%}$ chance of actually having COVID! This is due to the relatively large false positive rate of this cheap test, and is an illustration of the base rate fallacy in statistics.

We remark that if we switch the roles of the null hypothesis and alternative hypothesis, then some of the odds in the worksheet change, but the ultimate conclusions remain unchanged:

So the question of which hypothesis to designate as the null hypothesis and which one to designate as the alternative hypothesis is largely a matter of convention.

Now let us take a superficially similar situation in which a mother observers her daughter exhibiting COVID-like symptoms, to the point where she estimates the probability of her daughter having COVID at ${50\%}$ . She then administers the same cheap COVID-19 test as before, which returns positive. What is the posterior probability of her daughter having COVID?

One can fill out the worksheet much as before, but now with the prior probability of the alternative hypothesis raised from ${2\%}$ to ${50\%}$ (and the prior probablity of the null hypothesis dropping from ${98\%}$ to ${50\%}$ ). One now gets that the probability that the daughter has COVID has increased all the way to ${94\%}$ :

Thus we see that prior probabilities can make a significant impact on the posterior probabilities.

Now we use the worksheet to analyze an infamous probability puzzle, the Monty Hall problem. Let us use the formulation given in that Wikipedia page:

Problem 1 Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

For this problem, the precise formulation of the null hypothesis and the alternative hypothesis become rather important. Suppose we take the following two hypotheses:

Null hypothesis ${H_0}$ : The car is behind door number 1, and no matter what door you pick, the host will randomly reveal another door that contains a goat.
Alternative hypothesis ${H_1}$ : The car is behind door number 2 or 3, and no matter what door you pick, the host will randomly reveal another door that contains a goat.

Assuming the prizes are distributed randomly, we have ${\mathop{\bf P}(H_0)=1/3}$ and ${\mathop{\bf P}(H_1)=2/3}$ . The new information ${E}$ is that, after door 1 is selected, door 3 is revealed and shown to be a goat. After some thought, we conclude that ${\mathop{\bf P}(E|H_0)}$ is equal to ${1/2}$ (the host has a fifty-fifty chance of revealing door 3 instead of door 2) but that ${\mathop{\bf P}(E|H_1)}$ is also equal to ${1/2}$ (if the car is behind door 2, the host must reveal door 3, whereas if the car is behind door 3, the host cannot reveal door 3). Filling in the worksheet, we see that the new information does not in fact alter the odds, and the probability that the car is not behind door 1 remains at 2/3, so it is advantageous to switch.

However, consider the following different set of hypotheses:

Null hypothesis ${H'_0}$ : The car is behind door number 1, and if you pick the door with the car, the host will reveal another door to entice you to switch. Otherwise, the host will not reveal a door.
Alternative hypothesis ${H'_1}$ : The car is behind door number 2 or 3, and if you pick the door with the car, the host will reveal another door to entice you to switch. Otherwise, the host will not reveal a door.

Here we still have ${\mathop{\bf P}(H'_0)=1/3}$ and ${\mathop{\bf P}(H'_1)=2/3}$ , but while ${\mathop{\bf P}(E|H'_0)}$ remains equal to ${1/2}$ , ${\mathop{\bf P}(E|H'_1)}$ has dropped to zero (since if the car is not behind door 1, the host will not reveal a door). So now ${\mathop{\bf P}(H'_0|E)}$ has increased all the way to ${1}$ , and it is not advantageous to switch! This dramatically illustrates the importance of specifying the hypotheses precisely. The worksheet is now filled out as follows:

Finally, we consider another famous probability puzzle, the Sleeping Beauty problem. Again we quote the problem as formulated on the Wikipedia page:

Problem 2 Sleeping Beauty volunteers to undergo the following experiment and is told all of the following details: On Sunday she will be put to sleep. Once or twice, during the experiment, Sleeping Beauty will be awakened, interviewed, and put back to sleep with an amnesia-inducing drug that makes her forget that awakening. A fair coin will be tossed to determine which experimental procedure to undertake:

If the coin comes up heads, Sleeping Beauty will be awakened and interviewed on Monday only.

If the coin comes up tails, she will be awakened and interviewed on Monday and Tuesday.

In either case, she will be awakened on Wednesday without interview and the experiment ends.

Any time Sleeping Beauty is awakened and interviewed she will not be able to tell which day it is or whether she has been awakened before. During the interview Sleeping Beauty is asked: “What is your credence now for the proposition that the coin landed heads?”‘

Here the situation can be confusing because there are key portions of this experiment in which the observer is unconscious, but nevertheless Bayesian probability continues to operate regardless of whether the observer is conscious. To make this issue more precise, let us assume that the awakenings mentioned in the problem always occur at 8am, so in particular at 7am, Sleeping beauty will always be unconscious.

Here, the null and alternative hypotheses are easy to state precisely:

Null hypothesis ${H_0}$ : The coin landed tails.
Alternative hypothesis ${H_1}$ : The coin landed heads.

The subtle thing here is to work out what the correct prior state is (in most other applications of Bayesian probability, this state is obvious from the problem). It turns out that the most reasonable choice of prior state is “unconscious at 7am, on either Monday or Tuesday, with an equal chance of each”. (Note that whatever the outcome of the coin flip is, Sleeping Beauty will be unconscious at 7am Monday and unconscious again at 7am Tuesday, so it makes sense to give each of these two states an equal probability.) The new information is then

New information ${E}$ : One hour after the prior state, Sleeping Beauty is awakened.

With this formulation, we see that ${\mathop{\bf P}(H_0)=\mathop{\bf P}(H_1)=1/2}$ , ${\mathop{\bf P}(E|H_0)=1}$ , and ${\mathop{\bf P}(E|H_1)=1/2}$ , so on working through the worksheet one eventually arrives at ${\mathop{\bf P}(H_1|E)=1/3}$ , so that Sleeping Beauty should only assign a probability of ${1/3}$ to the event that the coin landed as heads.

There are arguments advanced in the literature to adopt the position that ${\mathop{\bf P}(H_1|E)}$ should instead be equal to ${1/2}$ , but I do not see a way to interpret them in this Bayesian framework without a substantial alteration to either the notion of the prior state, or by not presenting the new information ${E}$ properly.

If one has multiple pieces of information ${E_1, E_2, \dots}$ that one wishes to use to update one’s priors, one can do so by filling out one copy of the worksheet for each new piece of information, or by using a multi-row version of the worksheet using such identities as

$\displaystyle \frac{\mathop{\bf P}( H_1|E_1,E_2 )}{\mathop{\bf P}(H_0|E_1,E_2)} = \frac{\mathop{\bf P}(H_1)}{\mathop{\bf P}(H_0)} \times \frac{\mathop{\bf P}(E_1|H_1)}{\mathop{\bf P}(E_1|H_0)} \times \frac{\mathop{\bf P}(E_2|H_1,E_1)}{\mathop{\bf P}(E_2|H_0,E_1)}.$

We leave the details of these variants of the Bayesian update problem to the interested reader. The only thing I will note though is that if a key piece of information ${E}$ is withheld from the person filling out the worksheet, for instance if that person relies exclusively on a news source that only reports information that supports the alternative hypothesis ${H_1}$ and omits information that debunks it, then the outcome of the worksheet is likely to be highly inaccurate, and one should only perform a Bayesian analysis when one has a high confidence that all relevant information (both favorable and unfavorable to the alternative hypothesis) is being reported to the user.