Differentiating Statistical and Clinical Significance

There is a difference between statistical significance and clinical or practical significance that should be kept in mind with any type of statistical analysis.

With a sufficient amount of data, even the smallest differences in effects can be found and the difference declared ‘statistically significant’.  But this doesn’t necessarily mean that whatever the difference is matters in the real world.  Let’s look an example.

Let’s say a drug manufacturer is examining a new medication that treats hypertension.  The manufacturer is comparing the new drug to an existing drug that has been on the market for 40 years and is well understood.  Both drugs are being compared on their ability to lower blood pressure.  We’ll skip all the details and cut to the results:

• Old drug blood pressure reduction: 27 ± 3 mmHg
• New drug “X” blood pressure reduction: 29 ± 3 mmHg

The study was a huge clinical trial across many clinics and with the large power from the large sample the difference in the two groups is statistically significant at p=0.0442 (95% CI: 1-3) ^[1]I’m making these numbers up for the sake of the example.  The investigators thus correctly conclude that Drug X offers a significantly greater reduction in blood pressure than the old drug.  So, assuming no difference in side-effects, should everyone be switched to the new drug?

Here’s where clinical significance comes into play.  The investigators have shown that there is a statistically significant difference of about 2 mmHg between the groups.  The 95% CI shows us that this difference is likely 1 to 3 mmHg.  So the question we must ask is: “Does the difference between a 27 mmHg reduction in blood pressure and a 29 mmHg reduction in blood pressure matter?”, and the answer is probably ‘it depends’.

If the new drug costs 10 times what the old drug costs, no, that clinical difference is probably not meaningful in light of how expensive the drug is.  If the new drug is the same price or less, then there is probably a marginal benefit to switching if all other factors are equal.

Thus, statistical significance only tells us the effect we see appears to be real by some statistical criteria, but it does not tell us whether the difference we see matters.

This difference may be especially important (or difficult to detect) in studies that report relative risk or odds ratios, because the numbers can be so impressive looking. Fox example, a study might conclude there is a 3x relative risk of some disease from taking a particular medication.  Headlines in the newspapers will scream “medication TRIPLES risk of disease!!!”.  But if that disease naturally occurs at a rate of 1 person in ten million, and the medication reduces risk in one in five patients that take it, then the clinical significance of the increased risk might be acceptable.