What is p-hacking, and how does it create false findings?

P-hacking means making statistical decisions or methodology choices during research that artificially produce statistically significant results. These decisions raise the probability of false positives, where a study appears to show an effect that does not exist. One paper defines it broadly as any measure a researcher applies to render a previously non-significant p-value significant. [1] [2]

A p-value is part of a significance test and indicates how surprising or extreme a result is. In frequentist hypothesis testing, if the p-value falls below a threshold called alpha, the result is labelled significant and the null hypothesis can be rejected. Because significant results are notable and publishable, some researchers have an incentive to push the value under the threshold. [3] [2]

The practice belongs to a wider family often called data dredging or fishing. The term p-hacking is reserved for p-values, while similar practices in Bayesian statistics have been called B-hacking or posterior hacking. [2]

The core mechanism is the multiple comparison problem. With alpha at 0.05, researchers expect only 5 percent of tests on true null effects to come out significant. But the chance of at least one false positive rises with each added test. For independent tests it is 1 minus 0.95 raised to the number of tests: about 9.75 percent for two tests, 22.62 percent for five, and 40.13 percent for ten. P-hacking runs many tests without correcting for this accumulation and reports only the significant ones. [4] [2]

Almost every analysis involves choices, such as how to handle outliers, whether to combine groups, and which covariates to include. Many sets of choices can be defensible if fixed before the analysis. Probing the data and then reporting a result as if it had always been the plan dramatically raises the chance of a false positive. [5]

One form is changing the sample size. Researchers check for significance, add or drop participants, and test again, which violates the stopping rules of significance testing. Because the repeated tests share most of their participants, they are not independent, but different datasets can turn up as false positives each time, so the overall rate exceeds 5 percent. [4]

A second form is grouping participants after the fact along unplanned variables, which effectively multiplies the number of tests. Both forms were illustrated in a tutorial using data drawn from true null distributions, which still generated many false positives. [4]

The literature often portrays p-hacking as trial-and-error fishing driven by incentives, continuing until the desired result appears or the researcher runs out of patience. [2]

The American Statistical Association states that reporting only analyses with certain p-values makes the reported p-values essentially uninterpretable and leads to a spurious excess of significant results in the literature. It notes that no formal multiple testing is needed for the problem to arise: any choice of what to present based on statistical results compromises interpretation if readers are not told. It recommends disclosing the number of hypotheses explored, data collection decisions, all analyses and all p-values computed. The BMJ article similarly recommends prespecifying decisions in a public protocol, or being transparent when that is not possible. [6] [5]

The term appeared shortly after the onset of the replication crisis, and p-hacking has been discussed as one driver of false-positive results in the social sciences and beyond. [2]