Significance Of 750 GeV Bump May Be Greatly Overstated

The 750 GeV Bump

The ATLAS experiment has reported an apparent resonance at 750 GeV with local significance of about 3.5 to 3.9 sigma, which the CMS experiment has also reported, but with a lower significance of about 2 sigma. (Previous coverage at this blog is found here and here and here and here).

A new paper discussed below, however, suggests that this may be grossly overestimated due to a subtle flaw in the way that ATLAS determined the margin of error in its estimation of the diphoton background to which the actual number of events seen experimentally were compared.

Previous Discussions Of The Statistical Significance Of The Bump

A great deal of the initial discussion about this find focused on the extent to which the look elsewhere effect reduced the significance of this discovery, and the extent to which the fact that a bump was found in the same place by both the ATLAS and CMS experiments tamed the look elsewhere effect.

Normally, the look elsewhere effect would greatly reduce the significance of an individual experiment's locally significant resonance.  But, the likelihood that an experiment will have one highly significant bump in a great many trials simply due to random chance is much, much greater than the likelihood that two independent experiments will have a significant bump in the same place simply due to random chance.

But, the initial discussion largely took at face value the 3.9 sigma local significance of the ATLAS bump before adjustment for look elsewhere effects.  This has now been seriously questioned.

Doubts About the Significance Of The ATLAS Bump.  Is It Really Only 2 Sigma?

A new paper argues on quite technical physics driven grounds that ATLAS used the wrong methodology to calculate the local significance of the 750 GeV bump. Basically, they argue that the margin of error in the diphoton background was underestimated by almost a factor of two, because the absolute magnitude of the margin of error in lower energy events (which were oversampled in estimating the margin of error) is smaller than the margin of error in higher energy events (which were undersampled in estimating the margin of error).

Therefore, the new paper argues that the actual local significance of the 750 GeV bump at ATLAS was just 2 sigma.  But, a 2 sigma bump, even when replicated in an independent experiment at the same significance, followed by slight discounting for the look elsewhere effect, is very quite likely to be a mere fluke.

As the abstract of the new paper explains, theoretical error in calculating the margins of error in the background expectations is probably at fault in this case.

We investigate the robustness of the resonance like feature centred at around a 750 GeV invariant mass in the 13 TeV diphoton data, recently released by the ATLAS collaboration. We focus on the choice of empirical function used to model the continuum diphoton background in order to quantify the uncertainties in the analysis due to this choice. We extend the function chosen by the ATLAS collaboration to one with two components. By performing a profile likelihood analysis we find that the local significance of a resonance drops from 3.9σ using the ATLAS background function, and a freely-varying width, to only 2σ with our own function. We argue that the latter significance is more realistic, since the former was derived using a function which is fit almost entirely to the low-energy data, while underfitting in the region around the resonance.

Jonathan H. Davis, et al., "The Significance of the 750 GeV Fluctuation in the ATLAS Run 2 Diphoton Data" (January 13, 2016).

Background: Conventional Wisdom About Statistical Significance In High Energy Physics

Flukes with a local two sigma significance come and go all of the time in collider physics, and even bumps with a local three sigma significance tend to have a less than 50% chance of turning out to be real in the long run, although they start to warrant some serious attention.  Four sigma results are quite promising, but are still not considered a sure thing.  Only a result with five sigma significance after considering the look elsewhere effect is considered a "discovery" that is truly "real" in particle physics.

The gross discounting of the significance of experimental data (two sigma should mean a 95% chance that it is not a fluke, and three sigma should mean a 99% chance that it is not a fluke) that should be extremely unlikely given a Standard Model background and the mathematical rules of probability, flows from three main reasons.

It flows first, from look elsewhere effects (the fact that unusual results will happen randomly sometime if you do a lot of trials which is notoriously hard to quantify because rigorous definition of what constitutes a trial in complex experiments is notoriously elusive).

Second, it flows from underestimations of what are usually fairly modest theoretical calculation errors (which exist due to numerical approximations in the calculations and error in the measurement of fundamental constants used as inputs in them).

Third, it flows from systemic measurement errors. The statistical component of the margin of error, however, is almost always calculated with a precision that is empirically indistinguishable from perfect, because the mathematical formulas needed to calculate this are fairly straightforward and well understood, and it is purely a mathematical calculation that does not rely on the underlying physics.

Implications For Fundamental Physics

If the analysis by Davis and his colleagues is correct, then the 150+ papers devising beyond the Standard Model theories that can accommodate a new 750 GeV scalar or pseudoscalar boson are much ado about nothing, the fact that there have not been noticeable signals in four lepton or mixed lepton-photon channels (as would be expected in connection with a strong diphoton signal) requires no elaborate work around, and it is very likely that the 750 GeV bump seen by ATLAS and CMS will disappear as more data are collected in this year's portion of LHC Run-2 data collection.

Put another way, particle physics is at a great fork in the road right now. If the 750 GeV bump is real, then the LHC has just made the most profound discovery in physics since the development of the Standard Model in the early 1970s that explained everything except neutrino oscillation in the next forty year and had predicted the existence of the Higgs boson that was finally discovered in 2012.

A new 750 GeV boson would portend a whole new world of BSM physics at the electroweak scale that many people had hoped for in some form or another, but there had been no widespread consensus in the physics community predicting, and it becomes imperative to immediately start building a more powerful collider that can measure the new phenomena that are just around the corner.

In contrast, if the 750 GeV boson turns out to be just another fluke, the prospect that beyond the Standard Model physics do not exist all of the way up to the GUT scale or Plank scale are greatly heightened, and it is likely to there is very little left for a new collider to show us about fundamental physics, because a collider capable of disclosing GUT or Plank scale physics are far beyond the power of physics to construct in the foreseeable future in light of humanity's current technological and economic constraints.  A new collider would have to be ten orders of magnitude more powerful than the current one to get a good look at that scale of new physics.

How Likely Is The 750 GeV Bump To Be Real?

Unfortunately, in light of the new paper by Davis and the lack of corroboration of the 750 GeV bump in other channels, the betting odds that the 750 GeV bump have gone from perhaps 3-2 to odds of somewhere in the range of 9-1 to 99-1, in my estimation.

The previously emerging conventional wisdom that there is no phenomenology outside of neutrino physics that can't be fully explained by the Standard Model up to the GUT scale (10¹⁵ GeV vs. the 10³ to 10⁴ GeV scale measured at the LHC) described in the early 1970s is again well on its way to being our reality once again. The discovery of the Higgs boson mass of about 125 GeV made this theoretically possible (many of the different possible Higgs boson masses the Standard Model would have given rise to non-sensical results such as producing probabilities of something happening that don't add up to 100% at high energies), and the failure of experiments to discover convincing hints that there are any new physics out there hints that this is not only possible, but is probably true.