Simpson's paradox
A Socratic walk-through of Simpson's paradox — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #How can a treatment help every subgroup and still appear to harm the population as a whole?
Suppose a treatment does better than its rival on small stones, and better on large stones, and those are the only two kinds of stone there are. It seems to follow that it does better overall. The inference feels like it cannot fail.
It fails. Here is a real published table, from a comparison of two kidney-stone treatments — open surgery and a keyhole procedure — reported by Charig and colleagues.
| | Open surgery | Keyhole removal | |---|---|---| | Small stones | 81/87 = 93% | 234/270 = 87% | | Large stones | 192/263 = 73% | 55/80 = 69% | | All patients | 273/350 = 78% | 289/350 = 83% |
Open surgery wins on small stones. It wins on large stones. It loses overall. Nothing in that table is a typo, and no arithmetic has been abused. So which number should a surgeon act on?
Reasoning it through
REASONING #First check that the reversal is not a trick of rounding. Each bottom row is simply its two top rows added: 81 plus 192 gives 273 out of 350, and 234 plus 55 gives 289 out of 350. The reversal survives.
So look at how they add up. The overall rate is not an average of the two subgroup rates; it is a weighted average, and the weights are the case mixes. Open surgery's 350 patients are three-quarters large stones. Keyhole's 350 are three-quarters small ones. And the two stone sizes have very different base rates — pooled across both treatments, small stones cleared 88 per cent of the time and large ones 72 per cent.
Now the reversal is not mysterious at all. Open surgery is being scored mostly on the hard cases and keyhole mostly on the easy ones. Each treatment beats the other on like-for-like patients, and then the aggregate hands keyhole a much friendlier exam.
Why was the mix lopsided? Because doctors chose. Large, difficult stones were sent to open surgery. That is the crux: the third variable, stone size, influenced both which treatment a patient received and how likely they were to be cured. It sits upstream of both.
The same shape drove the most famous case of all. Berkeley's 1973 graduate admissions showed about 44 per cent of male applicants admitted against about 35 per cent of female applicants. Bickel, Hammel and O'Connell went department by department and found no evidence of bias against women in the admissions decisions; if anything a small tilt the other way. What produced the aggregate gap was where the applications went — women applied disproportionately to departments that admitted a small fraction of everyone. Note what that does and does not say: the decisions were not the source of the gap, but why the application patterns differed remains a real question.
The analogy
THE ANALOGY #You and a friend drive the same two roads on the same day. You are faster than them on the motorway and faster on the town road — yet your average speed for the day is lower, because you spent most of your day in town and they spent most of theirs on the motorway. Your average speed reports the mixture as much as it reports your driving.
With speeds the resolution is obvious, because nobody believes average speed measures driving skill, so no paradox is ever felt. The difficulty in real cases is exactly that the aggregate number looks like it measures the thing — treatment efficacy, admissions fairness — and only a claim about why the mixture came out that way can tell you whether it does.
Clarifying the model
THE MODEL #The first thing to give up is the idea that one of the two tables is wrong. Both are accurate summaries of the same patients. The paradox is not in the arithmetic; it is in the assumption that a single number can answer a question we have not yet stated.
So state it. A surgeon asks: if I send this patient, whose stone I have already measured, to open surgery rather than keyhole, does she do better? For that question, the subgroup rows are the relevant ones — the patient's stone size is known, fixed, and not something the choice of treatment changes. The aggregate row answers a different and largely useless question: what happens if I assign treatments the way the 1980s surgeons did, which mostly asks how their referral habits distributed difficulty.
Now the part that is genuinely counterintuitive, because it means "always look at the subgroups" is not the lesson. Imagine the third variable were not stone size but something caused by the treatment — say a drug works by lowering blood pressure, and you split the patients by their blood pressure measured after taking it. Split that way, the drug can look useless within each stratum while plainly helping overall, because you have partitioned away the very route through which it acts. Here the aggregate is right and the subgroups mislead.
The two cases are arithmetically identical. Nothing in the numbers distinguishes them. What distinguishes them is a claim about causal structure: does the third variable sit upstream of the treatment, as stone size does, in which case conditioning on it removes a confound; or downstream, on the causal path, in which case conditioning on it removes part of the effect you are trying to see. This is the real content of the paradox, and it is a claim data cannot supply about itself. You cannot decide whether to aggregate by staring harder at the table.
A picture of it
THE PICTURE #How to readStart at the two treatments on the left; the width of each ribbon is a number of patients. The first stage shows the case mix — open surgery's flow runs overwhelmingly into large stones, keyhole's into small ones — and that lopsidedness is what the doctors' referral habits created. The second stage shows why it matters: small stones clear far more often than large ones, whatever treatment is used. Follow both stages together and the reversal is visible as geometry: keyhole's patients are routed through the easy channel. The per-cell success rates are in the table above, not the diagram — what the ribbons show is the mixture that the aggregate silently averages over.
What became clearer
WHAT CLEARED #Simpson's paradox is not a statistical anomaly to be caught and corrected; it is the ordinary behaviour of weighted averages when the weights differ between the groups being compared. Both tables are true. Which one answers your question depends on why the weights came out unequal — on whether the splitting variable is a common cause of treatment and outcome, or a consequence of the treatment. That is a causal claim, it must be argued from knowledge of the situation, and no amount of further data will settle it. The paradox is really a demonstration that a comparison is not fully specified until you say what you would be intervening on.
Where to go next
ONWARD #- Why randomised assignment dissolves the confounding case by making the case mixes equal in expectation.
- How collider bias produces the mirror-image error: a spurious association created by conditioning rather than removed by it.
Key terms
TERMS #| Term | What it means |
|---|---|
| Confounder | a variable influencing both the treatment received and the outcome; conditioning on it is what makes the subgroup comparison the right one. |
| Mediator | a variable on the causal path from treatment to outcome; conditioning on it hides part of the effect. |
| Weighted average | the aggregate of subgroup rates, weighted by subgroup sizes, which is where the reversal comes from. |
| Base rate | the underlying success rate of a subgroup irrespective of treatment, such as the higher clearance rate for small stones. |
Every term the collection defines is gathered in the glossary.