Distinguishing Between Systemic and Statistical Error Mathematically

Background

It is not common in experimental research to make two error estimates for each measured quantity.

One is "statistical error" which can often be determined with precision based upon sample size and a few other assumptions, most of which are quite robust (i.e. big errors in many of the other assumptions have only a minor impact on the statistical error estimate for estimates in the range of values usually considered with these methods).

There other is systemic error, which is the notion that all of the measurements made may have varied from the accurate value by a certain amount due to limitations of the measurement tool.  Systemic error also includes what social scientists would call "bias."  Often this is determined non-mathematically (e.g. based on the precision with which a calibrating value is capable of being measured).

Distinguishing Statistical From Systemic Error Mathematically

It is also sometimes possible to distinguish statistical error from systemic error mathematically.  For reasons explained in any advanced undergraduate or graduate level statistics class, statistical errors in samples with large samples always trend towards a "Gaussian distribution" which is to say that they can be accurately approximated by a normal distribution (i.e. a "Bell Curve").

In contrast, any errors that cannot be well fit to a normal distribution are necessarily not statistical errors.  Thus, to the extent that error distributions are non-Gaussian, you know that this component of the experimental error must be systemic.

These concept are briefly alluded to here which is the blog post that inspired this post.