Subgroups and totals: Simpson’s paradox

An overall rate is a weighted average

A table split by category usually shows a rate for each subgroup: defaults per loan, late deliveries per parcel, defects per unit. A group's overall rate is its total events over its total cases:

$$\text{overall rate} = \frac{\text{events}_1 + \text{events}_2 + \dots}{\text{cases}_1 + \text{cases}_2 + \dots}$$

Written another way, it is the subgroup rates weighted by each subgroup's share of the cases, which we will call the mix:

$$\text{overall rate} = s_1 r_1 + s_2 r_2 + \dots \qquad (s_1 + s_2 + \dots = 1)$$

So an overall rate depends on two things: how the group does inside each subgroup, and where its cases sit. The lesson on averages and weighted data showed the weighting; this lesson is about what happens when two groups have different mixes.

Lower in every subgroup, higher overall

Two banks report defaults by loan category:

Loan category	Bank X	Bank Y
Prime mortgages	1 of 100 (1%)	18 of 900 (2%)
Small-business loans	90 of 900 (10%)	12 of 100 (12%)
All loans	91 of 1,000 (9.1%)	30 of 1,000 (3.0%)

Bank X has the lower default rate in both categories, yet its overall rate is about three times Bank Y's. The arithmetic is right. Small-business loans default far more often than mortgages at both banks, and 90% of X's loans are small-business loans while 90% of Y's are mortgages. In weighted form, X is $0.1 \times 1\% + 0.9 \times 10\% = 9.1\%$ and Y is $0.9 \times 2\% + 0.1 \times 12\% = 3.0\%$. The overall rates mostly record what each bank lends, not how well it lends within a category.

This reversal is called Simpson's paradox. It is not a contradiction, only a surprise: an aggregate mixes a comparison of performance with a comparison of composition.

When a reversal can happen

Two conditions must hold together:

1. The subgroups differ a lot in their own rates (here 1–2% against 10–12%).
2. The groups being compared have different mixes across those subgroups.

If both banks had the same mix, the bank with the lower rate in every category would also have the lower overall rate, because each overall rate would be the same weighted sum, taken of smaller numbers for one bank. So the first thing to check in a split table is whether the sizes, or the shares, differ between the groups.

A second quick check: each group's overall rate lies between its own lowest and highest subgroup rates. X's overall rate can be anything from 1% to 10%, depending on its mix; an option outside that range is wrong at once.

A like-for-like comparison

To compare performance alone, give both groups the same mix and recompute. On X's mix (10% mortgages, 90% small business), Y would have $0.1 \times 2\% + 0.9 \times 12\% = 11.0\%$, against X's 9.1%. On Y's mix, X would have $0.9 \times 1\% + 0.1 \times 10\% = 1.9\%$, against Y's 3.0%. Whichever mix you choose, X comes out lower. Holding the mix fixed so that only the subgroup rates differ is called standardising.

Recomputing an aggregate by hand

• From counts: add the events, add the cases, divide once.
• From rates and sizes: turn each rate into a count first (10% of 900 is 90), then add.
• From rates and shares: take the weighted sum, measuring from the lowest rate if it helps: $1\% + 0.9 \times (10 - 1)\% = 9.1\%$.
• Backwards: from the overall rate and the two subgroup rates, the mix follows. X's 9.1% sits $\frac{9.1 - 1}{10 - 1} = \frac{8.1}{9} = 0.9$ of the way from 1% to 10%, so 90% of its loans are in the 10% category.

Never average the rates column unless the subgroups are the same size. And when a table gives rates but no sizes, the overall rate is not determined: you can only say it lies between the subgroup rates.

What a split table supports

Sort each statement by the level it speaks about.

• Subgroup statements ("X has the lower default rate on mortgages") are read straight from the rows.
• Overall statements ("Y has the lower default rate across all its loans") need the totals, recomputed from counts.
• Statements about counts need counts, not rates. X had more defaulted small-business loans than Y, 90 against 12, although X's default rate on them was lower.
• Claims of cause or quality ("X is the more careless lender", "small-business lending caused X's losses") go beyond any table of rates. The table describes outcomes; it does not say why they happened. A split also controls only for the variable it splits by: within "small-business loans" the two banks may still lend to very different firms, and a finer split could change the picture again.

The usual wrong answer is a statement that is true at one level, read as if it were true at another.

A routine for split tables

Compare the mixes first: if the sizes differ a lot between groups, expect the overall comparison to part from the subgroup ones. Recompute any aggregate from counts. Then read each statement for its level (subgroup, overall or count) and for any word of cause or quality, such as "better", "safer", "because" or "more effective", which the table alone cannot support.