Fat Tails Are The Norm

Scientific papers of all stripes routinely assume that the likelihood of low frequency events is "Gaussian" which is to say that the probability distribution of approximately a "normal" distribution which can be parameterized fully by a mean value and a standard deviation.

But, a recent study shows that, in real life, in complex fact patterns, probability distributions routinely have "fat tails" (a.k.a. "long tails") (i.e. the likelihood of extreme events is much greater than a normal distribution would suggest).

Published recently in the journal Royal Society Open Science, the study suggests that research in some of the more complex scientific disciplines, such as medicine or particle physics, often doesn't eliminate uncertainties to the extent we might expect. 

"This is due to a tendency to under-estimate the chance of significant abnormalities in results." said study author David Bailey, a professor in U of T's Department of Physics. 

Looking at 41,000 measurements of 3,200 quantities -- from the mass of an electron to the carbon dating of a sample -- Bailey found that anomalous observations happened up to 100,000 times more often than expected. 

"The chance of large differences does not fall off exponentially as you'd expect in a normal bell curve," said Bailey.

The paper and its abstract are as follows:

Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how well these have been evaluated and what they imply. Reported scientific uncertainties were studied by analysing 41 000 measurements of 3200 quantities from medicine, nuclear and particle physics, and interlaboratory comparisons ranging from chemistry to toxicology. Outliers are common, with 5σ disagreements up to five orders of magnitude more frequent than naively expected. Uncertainty-normalized differences between multiple measurements of the same quantity are consistent with heavy-tailed Student’s t-distributions that are often almost Cauchy, far from a Gaussian Normal bell curve. Medical research uncertainties are generally as well evaluated as those in physics, but physics uncertainty improves more rapidly, making feasible simple significance criteria such as the 5σ discovery convention in particle physics. Contributions to measurement uncertainty from mistakes and unknown problems are not completely unpredictable. Such errors appear to have power-law distributions consistent with how designed complex systems fail, and how unknown systematic errors are constrained by researchers. This better understanding may help improve analysis and meta-analysis of data, and help scientists and the public have more realistic expectations of what scientific results imply.

David C. Bailey. "Not Normal: the uncertainties of scientific measurements." Royal Society Open 4(1) Science 160600 (2017).

It isn't that the statistical law sometimes called the "law of averages" that make a Gaussian distribution seem reasonable, is inaccurate. But, the assumptions of that law often do not hold as tightly as our intuition suggest.

This could be because the events are not independent of each other, because systemic error is underestimated, because the measurements aren't properly weighted, because the thing being measured does not have sufficiently quantitatively comparable units, because look elsewhere effects aren't properly considered, or because the underlying distributions of individual events that add up to form the overall result are not simple binomial probabilities.

In particle physics this is handled by setting standards for nominal significance in error estimates assuming a Gaussian distribution that are far higher than what ought to be necessary to constitute a discovery (i.e. 5 sigma).

Lubos Motl also has a recent post on a similar subject which I won't attempt to summarize here, which distinguishes between probabilities with "fat tails" (when extreme events are actually more likely than in a normal distribution) and the application of the "precautionary principle" (which is used to justify assuming that unlikely bad events have relatively high probabilities and should be regulated when the exact probability can't be determined exactly) to justify regulations in a cost-benefit analysis.