P-Hacking

In plain terms

P-hacking is testing your data enough different ways that something eventually comes out "significant", then writing up that one result and leaving the rest in a drawer. Drop the outliers. Add a control variable. Split by age, then by region. Stop collecting data the week the number looks good. Any single one of those moves can be defensible. Doing them until the answer arrives is not.

It helps to know what a p-value actually says, because most popular explanations get it wrong. Assume for a moment there's no real effect at all, and assume the rest of your model and sampling assumptions hold. The p-value answers: how often would data like this, or data even further from no effect, turn up anyway? A p of 0.03 means about 3 percent of the time.

That's the whole claim. It is not the probability that your hypothesis is true. It is not the probability that the result was chance. It is not the probability the finding will replicate. It's a statement about how ordinary or unusual your data would look in a world where nothing is going on.

Why it matters

The 0.05 convention only works if you asked one question. Run 20 independent tests on data with no real effect in it and you'd expect one to land under 0.05 by chance. The odds that at least one does are about 64 percent. Report only that one and you've published a coin flip as a discovery, with a p-value that's now meaningless, because it was calculated as if the other 19 tests never happened.

You don't have to be sneaky to do this. Andrew Gelman and Eric Loken called the honest version the garden of forking paths. A researcher runs one analysis, but the choices along the way, which cases to exclude, which variable to adjust for, whether to log-transform, which subgroup to lead with, were made after seeing the data. Had the data come out differently, different choices would have looked reasonable. The effective number of comparisons is large even though only one test was ever run.

This is a good part of what sits behind the replication crisis. When teams went back and tried to reproduce a hundred published psychology findings, roughly a third of the replications came out statistically significant, against 97 of the original 100. Similar shortfalls turned up in preclinical cancer biology and economics. The original results weren't fraud. Many were the product of a literature that rewarded significance and never counted the tests that didn't make it.

Canonical example

"The new checkout flow showed no overall lift. But among mobile users in Canada on weekends, conversion rose 4 percent, p = 0.03. Ship it."

The headline test failed, so the analyst went looking. Slicing by device, by country, and by weekday versus weekend gives 60 subgroups. If the new flow does nothing at all, you'd still expect about three of those 60 tests to come in under 0.05, purely from noise. One did. The 0.03 was computed as though that Canadian weekend mobile slice was the only question ever asked, which is exactly what it wasn't.

The tell is the shape of the finding. A real effect that shows up in one oddly specific corner and nowhere else is more often the corner where the noise happened to pile up.

Counter-example (not a fallacy)

"We registered one primary outcome and the exact analysis before collecting data. It came in at p = 0.02. We also ran fifteen subgroup analyses afterward and reported all fifteen, flagged as exploratory and uncorrected."

Nothing wrong here, and note that they still ran fifteen extra tests. Exploring data is how hypotheses get generated; a dataset nobody pokes at is a wasted dataset. The difference is that the confirmatory test was fixed in advance, so its p-value means what it says, and the exploratory work is labeled as exploratory rather than smuggled in as proof. Whatever those fifteen turned up is a lead for the next study, not a finding.

The line: were the analysis decisions locked in before you saw the data, and did you report everything you tried?

How to fix it

If you've been linked here, the repair is mostly bookkeeping. Write down the analysis you intend to run before you look at the outcome, and pick one primary question. If you test many things, say how many and adjust the threshold accordingly, or just report every test you ran and let the reader see the denominator. Decide your stopping rule for data collection in advance, because "check the numbers weekly and stop when it's significant" will find significance in pure noise given enough weeks. If a result only surfaced after you went looking, present it as a lead worth a fresh test rather than as a conclusion. That's not a weaker claim. It's a claim that will still be standing in a year.

If you're on the receiving end, ask how many analyses were run and how the subgroup was chosen. "Was this the question you set out to answer, or the one that worked?" A preregistered study can point to the record. A hacked one changes the subject.