Proof of Bayes' rule: Intro

by Eliezer Yudkowsky Feb 11 2016 updated Jul 10 2016

Proof of Bayes' rule, assuming you know the rule itself, and the notations for the quantities involved.

[todo: This page has been orphaned, for being insufficiently different from /p/1xr (and not much easier to read). It should probably be deleted.]

In what follows, $~$H_i$~$ and $~$H_j$~$ are two competing hypotheses, and $~$e$~$ is the evidence. Bayes' rule (in the odds form) says:

$$~$ \dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \cdot \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)} = \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)} $~$$

The quantity $~$\frac{\mathbb P(H_i)}{\mathbb P(H_j)}$~$ is the prior odds. E.g., in the Diseasitis problem, dividing 20% by 80% gives us 1/4, which represents Odds of (1 : 4).

The quantity $~$\frac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)}$~$ is the likelihood ratio. For example, in the diseasitis problem, we'd divide $~$\frac{\mathbb P({positive}\mid {sick})}{\mathbb P({positive}\mid \neg {sick})}$~$ to get the number 3, indicating "Sick patients are three times as likely as healthy patients to turn the tongue depressor black."

The quantity $~$\frac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)}$~$ is the posterior odds ratio. E.g., in the Diseasitis problem, $~$\frac{\mathbb P({sick}\mid {positive})}{\mathbb P(\neg {sick}\mid {positive})}$~$ works out to an odds ratio of 3/4, which are odds of (3 : 4). We indeed have that 1/4 * 3 = 3/4, so the rule is numerically true in this case.

The proof of Bayes's Rule in its general form proceeds by repeatedly applying the definition of conditional probability, $~$\mathbb P(X\mid Y) = \frac{\mathbb P(X \wedge Y)}{\mathbb P(Y)}.$~$

$$~$ \dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \times \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)} = \dfrac{\mathbb P(e \wedge H_i)}{\mathbb P(e \wedge H_j)} = \dfrac{\mathbb P(e \wedge H_i) / \mathbb P(e)}{\mathbb P(e \wedge H_j) / \mathbb P(e)} = \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)} $~$$

In the Diseasitis problem, the corresponding quantities would be:

$$~$\dfrac{20\%}{80\%} \times \dfrac{90\%}{30\%} = \dfrac{18\%}{24\%} = \dfrac{0.18 / 0.42}{0.24 / 0.42} = \dfrac{43\%}{57\%}$~$$

Visualizing the proof graphically:

bayes venn

Which can be read as saying: The ratio of the initial sick population (red) to the initial healthy population (blue), times the ratio of positive results (+) in the sick population to positive results in the blue population, equals the ratio of the positive-and-red population to positive-and-blue population. Thus we can divide both into the proportion of the whole population (grey) which got positive results, yielding the posterior odds of sick (red) vs healthy (blue) among only those with positive results.


Emile Kroeger

Visualizing the proof graphically:

… I don't understand this diagram. I understand the proof above fine, but the diagram just confuses me.