Physicists' Dirty Little Secret

It isn't widely known, but a post by Lubos Motl reminds us, that there are actually some papers that come out of the latest high energy physics experiments in which the results do not neatly match their predictions. The latest involves the production of pairs of top quarks, but it actually happens much more frequently than most people familiar with high energy physics knows or admits.

Almost all of these cases involve quantum chromodynamics, the Standard Model theory of the strong force. And, the reason this doesn't make headlines is that for all practical purposes, in anything more than the most idealized highly symmetric situations, it is impossible to do calculations that make actual predictions with the actual Standard Model equations of QCD. Instead, you must use one or more of several tools for numerically approximating what the Standard Model equations predict, each of which has its flaws, and some of which aren't fully compatible with each other.

Since each of these numerical approximations has been well validated in their core domains of applicability, there is nothing deeply wrong with any of them. But, none of them are perfect, even in isolation.

But, towards the edges of their domains of applicability, or in circumstances when you have to apply more than one not fully compatible approximation to the same problem to get an answer, both of which are present in the experiment the Motl describes in his recent post, sometimes the results can be wildly off. Also, lots of key QCD constants are only known to precisions of 1% of less, which also doesn't help produce accurate predictions from first principles.

Yet, this isn't terribly notable, because everyone knows that the relevant sources of theoretical prediction error in these situations often far exceeds the combined statistical and systemic experimental error. 

Hence, in QCD, we often know that the experimental measurements are sound but have deep doubts about the soundness of our predictions, while the situation is the other way around in all other parts of QCD physics. QCD is a whole different ball game in Standard Model physics.

In somewhat related news, it turns out that a method of doing calculations in QCD that was widely assumed in conventional wisdom to be the most efficient is actually much less efficient than an old school approach that takes a little more thought. The old school methods approaches the theoretical maximum of calculation efficiency, which makes it possible to calculate the infinite series approximations common in quantum mechanics calculations to far more terms and thus to achieve much greater precision and accuracy with the same amount of calculation work. So, progress is being made in fits and starts on the theoretical front, even though it can be painful to get there.