Simpson's Paradox

In plain terms

Split a dataset into groups and one story appears. Add the groups together and the story flips. Both sets of numbers are correct. That's Simpson's paradox, and it is less a paradox than a warning about what averages do when the groups being averaged are different sizes.

The mechanism is always the same. Some third variable determines both which group people land in and how the outcome usually goes there. If most of one population piles into the hard group and most of the other into the easy group, the pooled averages describe the split more than they describe the outcome.

The uncomfortable part is that statistics alone cannot tell you which number to believe. You need to know how the data came to be arranged that way.

Why it matters

The pooled number is the one that makes headlines, and it can be exactly backwards. A hospital with worse overall survival rates may be better at every individual procedure, if it takes the sickest patients. A treatment that wins in mild cases and wins again in severe cases can lose overall, if doctors gave it mostly to the severe ones.

The reverse error is just as available. Slicing data until an unwelcome result disappears is a real move, and "but look at it by subgroup" is not automatically the more honest view. Which level answers the question depends on what the split variable is doing.

There is a workable rule. If the variable you split on comes before the outcome and influences who ends up where, the split numbers are usually the ones to trust. If it comes after, and is itself affected by the thing you're studying, splitting on it can manufacture a reversal out of nothing.

Canonical example

The most cited real case is graduate admissions at the University of California, Berkeley, for autumn 1973, published by Bickel, Hammel, and O'Connell in Science in 1975.

In aggregate, 8,442 men applied and about 44% were admitted. 4,321 women applied and about 35% were admitted. A nine-point gap across more than twelve thousand applications is not noise, and on its face it looks like the university was rejecting women.

Department by department, the picture changed. Across the six largest departments, women were admitted at a higher rate than men in four of them. The pooled gap came from where the applications went. The two departments with the easiest admissions, both admitting over 60% of applicants, drew 825 and 560 male applicants against 108 and 25 female ones. The harder departments, some admitting under 10%, received far more applications from women. Even within just those six, the pooled rates were about 44% for men and about 30% for women, while the department-level rates mostly ran the other way.

The authors' conclusion was careful, and worth repeating accurately: the aggregate gap was explained by women applying disproportionately to departments with low admission rates, not by admissions committees at those departments treating women worse. They did not claim the matter was closed. They pointed out that the question moves upstream, to why the application patterns differed in the first place, which is a question about earlier schooling and field choice rather than about the admissions process. Anyone citing Berkeley as proof that a bias claim was simply wrong is overreading it. The study relocated the question. It did not delete it.

Counter-example (not a fallacy)

"Overall, the drug beat placebo. But among patients who stuck with the full course, placebo did better, and among patients who dropped out, placebo did better too. So the drug is actually worse."

Here the aggregate is the trustworthy number and the subgroups are the trap. Sticking with the course is not a fixed characteristic patients walked in with. It's an outcome, partly caused by how the drug made them feel. Splitting on it breaks the randomisation that made the comparison fair and can flip the sign for no real reason.

Compare the kidney stone version, where the split is legitimate: treatment A beat treatment B for small stones and beat it again for large stones, yet lost overall, because surgeons sent the large stones to A. Stone size existed before treatment and determined the assignment. That's a genuine confounder, and the split numbers are the right ones.

The line: does the variable you're splitting on come before the outcome and shape who ends up where, or is it downstream of the thing you're measuring? Before, split. After, don't.

How to fix it

If you've been linked here, show both levels and say which one you think answers the question and why. "In aggregate X, by department Y, and here's what's driving the difference" is a stronger claim than either number alone, and it heads off the objection instead of waiting for it. If you presented only the pooled figure, go and check whether the groups being pooled are lopsided in size. If they are, the average may be describing the composition of your sample rather than the effect you're claiming.

If you're on the receiving end, ask for the breakdown rather than asserting the number is wrong: "What does this look like by department?" And be ready for the answer to cut the other way, because the person disaggregating is sometimes the one skipping a step.