Conditional probability, total probability and Bayes

What "given" does

$P(A \mid B)$, read "the probability of $A$ given $B$", is the probability of $A$ once you know that $B$ has happened. Knowing $B$ throws away every outcome outside $B$: the sample space shrinks to $B$, and you ask what share of it also lies in $A$.

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0.$$

With counts, the same idea is simply "how many in both, out of how many in $B$". The denominator is the condition, never the whole population.

Order matters

$P(A \mid B)$ and $P(B \mid A)$ are different numbers with the same numerator and different denominators. In a class of 30 where 10 students play chess and 4 of those also play the piano, while 8 students play the piano in all: $P(\text{piano} \mid \text{chess}) = \frac{4}{10}$, but $P(\text{chess} \mid \text{piano}) = \frac{4}{8}$. Before computing, say aloud which group you are restricted to.

The multiplication rule and trees

Rearranging the definition gives

$$P(A \cap B) = P(B)\,P(A \mid B) = P(A)\,P(B \mid A).$$

This is what you use when an experiment happens in stages, such as drawing without replacement or choosing a supplier and then testing a part. A tree displays it: the first branches carry $P(B)$ and $P(\overline{B})$; the branches after them carry conditional probabilities; the probability of a path is the product along it. The branches leaving any point must add up to 1.

$A$ and $B$ are independent exactly when $P(A \mid B) = P(A)$: knowing $B$ changes nothing.

Total probability

If the sample space is split into separate cases $B_1, B_2, \dots, B_n$ that cover everything (three factories, two machines, "has the condition" and "does not"), then

$$P(A) = P(B_1)P(A \mid B_1) + P(B_2)P(A \mid B_2) + \dots + P(B_n)P(A \mid B_n).$$

On a tree, this is "add every path that ends in $A$". It is the sum rule and the multiplication rule working together.

Bayes' theorem

Often the question gives $P(A \mid B)$ and asks for $P(B \mid A)$: the defect rate of a machine is known, and you want the chance that a defective part came from that machine. Bayes' theorem reverses the condition:

$$P(B_k \mid A) = \frac{P(B_k)\,P(A \mid B_k)}{P(A)},$$

where the denominator $P(A)$ comes from total probability. In words: the path through $B_k$ that ends in $A$, divided by all paths that end in $A$.

The frequency table: the safest route

Formulas with several decimals are where marks are lost without a calculator. Instead, imagine a round number of cases, such as 1000 or 10 000, chosen so that every count is a whole number, and fill in a table.

	$A$	not $A$	Total
$B_1$
$B_2$
Total

Fill the row totals from $P(B_1), P(B_2)$, then each $A$ cell from the conditional rates, then the column total. Every conditional probability is now a cell divided by a total: $P(A \mid B_1)$ divides by a row total, $P(B_1 \mid A)$ divides by a column total. The same table also gives the joint and the overall rates, so one table answers every part of a multi-part question.

Base rates matter

When one group is much larger than the other, a small error rate in the large group can produce more cases than a large rate in the small group. That is why a test that is right 90% of the time can still produce mostly false alarms when the thing it detects is rare. A table makes this visible at once; intuition usually does not.

A routine for Bayes questions

1. Name the cases (the rows) and the observed event (the column).
2. Pick a total such as 1000 so the counts are whole.
3. Fill the rows, then the event column.
4. Divide the cell you want by the column total of the observed event.
5. Check: the probabilities of all cases given the event add up to 1.