The answer may very well lie in a more detailed mathematical analysis of the way that the non-Newtonian aspects of gravity that follow directly from the principles upon which the theory was formulated (although not necessarily precisely the same formulation of the equations derived from those principles).
Deur's Remarkable Work On Gravity
I previously noted at this blog Alexandre Deur's very important work on the phenomenological implications of the fact that massless spin-2 gravitons in a particle based quantum gravity realization of quantum gravity interact not only with other particles containing mass-energy, but also with each other.
GR and QCD Inspired Self-Interactions Of Gravity With Gravitational Energy May Explain Dark Matter
Deur's analysis suggests that this effect should give rise to essentially all of the phenomena attributed to dark matter and is a function to the extent to which baryonic matter distributions are not spherically symmetric and their scale with more massive systems exhibiting proportionately stronger dark matter effects and less spherically symmetric systems exhibiting stronger dark matter effects.
His results which rely on a low order approximation of the gravitational self-interaction term of a general relativity Lagrangian that is constructed by analogy to the Lagrangian of gluon self-interactions in QCD, reproduce the successes and predictions of theories like Milgrom's MOND theory in galactic scale systems from ellipical galaxies to spiral galaxies to dwarf galaxies, while overcoming MOND's failures in galactic clusters and the Bullet Cluster.
Moreover, while MOND adds one new fundamental parameter to general relativity and the Standard Model, Deur, as noted below, either adds none or takes one away.
In contrast, all dark matter models need, at a minimum, an average dark matter density (which has been determined empirically), and an average dark matter particle mass. Many also need a mass for a massive dark matter self-interaction boson, and its coupling constant, and/or a cross-section of interaction between one or more dark sector particles and non-dark sector particles.
Subtle Non-Newtonian GR Effects May Explain Dark Energy Without A Cosmological Constant
Furthermore, in a point that I didn't emphasize sufficiently before, he explains how graviton self-interactions that pull gravitons emitted from ordinary matter towards strong gravitational field lines and away from the destinations we would expect them to take in the absence of gravitational self-interactions weakens the gravitational pull between systems that show dark matter effects and other clumps of matter outside those systems.
This effect is equivalent to dark energy, and according to his heuristic analysis should have the right order of magnitude, although he has not been reconciled his observation in detail with the astronomical data.
A suggestion that the order of magnitude of the non-Newtonian implications of General Relativity (possibly generalized slightly) may be sufficient to explain the entire dark sector comes from Hong Sheng Zho in a preprint last modified on June 9, 2008 and originally submitted on May 27, 2008 arXiv:0805.4046 [gr-qc] that "the negative pressure of the cosmological dark energy coincides with the positive pressure of random motion of dark matter in bright galaxies."
Another indication that these effects may be of the right order of magnitude to explain dark energy as well as dark matter comes from Greek scientists K. Kleidis and N.K. Spyrou in their paper "A conventional approach to the dark-energy concept" (arXiv: 1104.0442 [gr-qc] (April 4, 2011). They too note that energy from the internal motions of the matter in the universe (both baryonic and dark) in a collisional dark matter model are of the right scale to account for existing observational data without dark energy or the cosmological constant.
It is also worth noting that the cosmological constant is small enough that other kinds of careful analysis of sources for dark energy effects in the Standard Model and non-Newtonian effects in general relativity other than the cosmological constant may explain some or all of it.
For example, Ralf Schutzhold in an April 4, 2002 preprint at arXiv:gr-qc/0204018 in a paper entitled "A cosmological constant from the QCD trace anomaly" noted that "non-perturbative effects of self-interacting quantum fields in curved space times may yield a significant contribution" to the observed cosmological constant. The calculations in his four page page conclude that: "Focusing on the trace anomaly of quantum chromo-dynamics (QCD), a preliminary estimate of the expected order of magnitude yeilds a remarkable coincidence with the empirical data, indicating the potential relevance of this effect."
This Approach Eliminates One Fundamental Measured Physical Constant And Adds No New Ones.
Even more powerfully, these non-Abelian quantum gravity effects are derived from first principles, without introducing any non-Standard Model particles other than the plain vanilla massless spin-2 graviton that has been widely expected to exist for many decades, and without introducing any new experimentally measured physical constants that weren't already present in the Standard Model and General Relativity. Indeed, it is quite possible that this analysis could reduce the number of fundamental physical constants in the two combined theories, by eliminating the need for a separate cosmological constant, which is one of the two physical constants specific to General Relativity.
Everything else would just be math. Hard math, admittedly, but probably nothing beyond what can be accomplished using numerical simulations of the type currently used to for Lattice QCD with existing or near future computing power.
The Only Physical Constants Left To Measure (In Neutrino Physics) May Be Known In A Decade
If this analysis is correct, then the only gaps in our measurements of the physical constants in the fundamental laws of physics that govern everything in the universe that remain to be meaningfully measured are four physical constants related to neutrino physics: (1) the mass hierarchy of the neutrino masses ("normal" or "inverted"), (2) the absolute mass of at least one of the neutrino mass eigenstates (something already quite constrained by Planck data), (3) the quadrant of one of the theta angle parameter of the PMNS matrix (there are two possibilities), (4) the CP violating phase of the PMNS matrix, and (5) the Majorana or Dirac nature of neutrino masses. This task is likely to be accomplished in the next four to ten years by experiments that are currently being conducted, or have been designed and funded and are currently in the process of being constructed.
The Standard Model Particle Set, Plus The Graviton, May Be The Complete Set
If this worked, it would strongly suggest that the Standard Model particles plus the graviton is the complete set of particles that exist in the universe:
Three generations four kinds of fermions, each with their matter and antimatter counterparts, in three color charge variations each for the two kinds of quarks, and in two parity variations each for each of the three kinds of charged fermions, plus photons, eight color charge variants of gluons, the Z boson, the W+ boson, the W- boson, the Higgs boson and the graviton.
Thus any additional fundamental particle content in any beyond the Standard Model theory seeking to unify these forces, other that preons that can produce this and only this set of composite particles, could be rejected immediately as incorrect.
Is Deur's Analysis A Modification of General Relativity?
While Deur's analysis of gravitational self-interactions follows the principles of General Relativity very closely, it is ultimately an incomplete quantum gravity theory, rather than classical General Relativity itself.
Deur's treatment of gravitational self-interaction is non-standard, and can probably be more fairly described as a modification of general relativity, rather than a straight application of Einstein's equations, manipulated into a different form.
As the leading textbook on general relativity, Gravitation by C.W. Misner, K.S. thorne and J.A. Wheller (1973), explains in Section 20.4, there are several arguments in favor of the proposition that the energy of a gravitational field cannot be localized and hence cannot be considered in the same way that all other mass-energy is considered in the energy-momentum tensor of general relativity.
This widely held textbook assumption that gravitational energy cannot be localized may be the best explanation for why it has taken so long to seriously explore approaches to incorporate gravitational force self-interaction in phenomenological analysis, because the self-interaction effects that Deur argues can be deduced from a graviton model of gravity can only work when gravitational energy is localized, with quanta, or as classical fields or as classical curvature self-interactions.
But, as A.I. Nikishov of the P.N. Lebedev Physical Institute in Moscow states in an updated July 23, 2013 version of an October 13, 2003 preprint (arXiv:gr-qc/0310072), these arguments "do not seem convincing enough." For example, Feynman's lectures on gravitation assumed that gravity was mediated by a graviton that could be localized with a self-interaction coupling strength equal to the graviton's energy, just as the graviton would with any other particle. String theory and supergravity theories, generically make the same assumptions.
Nikishov also made the same analysis of Deur in his paper "Problems in field theoretical approach to gravitation" dated February 4, 2008 in its latest preprint version arXiv:gr-qc/04100999 originally submitted October 20, 2004, when he states in the first sentence of his abstract that:
We consider gravitational self interaction in the lowest approximation and assume that graviton interacts with gravitational energy-momentum tensor in the same way as it interacts with particles.Deur and Nikishov are not the only investigators to note the potential problems with the anomalous ways that conventional General Relativity treats gravitational self-interactions, and they are not alone in this respect. Carl Brannen has also pursued some similar ideas.
As another example, consider this statement by A.L. Koshkarov from the University of Petrozavodsk, Russia in his November 4, 2004 preprint (arXiv:gr-qc/0411073) in the introduction to his paper entitled "On General Relativity extension."
But in what way, the fact that gravitation is nonabelian does get on with widely spread and prevailing view the gravity source is energy-momentum and only energy-moment? And how about nonabelian self-interaction? Of course, here we touch very tender spots about exclusiveness of gravity as physical field, the energy problem, etc. . . .All the facts point out the General Relaivity is not quite conventional nonabelian theory.Koshkarov then goes on to look at what one would need to do in order to formulate gravity as a conventional nonabelian theory like conventional Yang-Mills theory.
Alexander Balakin, Diego Pavon, Dominik J. Schwarz, and Winfried Zimdahl, in their paper "Curvature force and dark energy" published at New.J.Phys.5:85 (2003), preprint at arXiv:astro-ph0302150 similarly noted that "curvature self-interaction of the cosmic gas is shown to mimic a cosmological constant or other forms of dark energy."
Balakin, et al., reach their conclusions using the classical geometric expression of general relativity, rather than a quantum gravity analysis, suggesting that the overlooked self-interaction effects do not depend upon whether one's formulation of gravity is a classical or a quantum one, but the implication once again, is that a failure to adequately account for the self-interaction of gravitational energy with itself may account for all or most dark sector phenomena.
As noted above, there is a rich academic literature expressing dissatisfaction with the precise way that General Relativity was formulated by Einstein on the grounds that it lacks one or another subtle aspects of rigor or theoretical consistency, or makes a subtle assumption that is unnecessary, or needs to be tweaked to formulate it in a way that formulates gravity in a quantum manner.
Some of the modifications proposed are more ambitious than others and there are at least half a dozen serious contenders for ways to reformulate General Relativity, adopting all or more of the core theoretical axioms from which its equations were derived, in a way that is usually indistinguishable from Einstein's equations in all contexts that have been experimentally measured.
What makes the effort by Deur stand out is that his quite simple and naive approach, despite being incomplete and not very rigorous, manages to draw out non-Newtonian effects that replicate much or all of the observed phenomenology of dark matter and dark energy, without adopting any core axioms that aren't extremely well motivated and natural.
The only axiom he adds to Einstein's formulation is that gravitational energy is localized in massless spin-2 gravitons that couple to each other with a strength proportional to their mass-energy, the same coupling that gravitons are proposed to have to all of the Standard Model's other particles.
This assumption is pretty much the most conservative and banal axiom that one could add to General Relativity and has been a mainstream assumption of physicists for decades (the graviton was named in 1934 and considerable research has been applied to associate it with properties that replicate most features of general relativity). Honestly, there is really no other principled stance to take than the one in which gravitational energy is self-interacting with a strength proportionate to its energy, in just the way that it does with all other forms of matter and energy, as a wide swath of investigators assume.
And yet, remarkably, it turns out that this subtle additional axiom which naively wouldn't seem to have any phenomenological impact on general relativity at all, actually seems to have shockingly immense phenomenological consequences when analyzed properly. The impact is so great that this insight may be enough to tie up all of the loose ends in fundamental physics, even without formulating a complete and rigorous theory of quantum gravity.
Indeed, even a slightly weaker assumption that gravitational energy is localized and that gravity self-interacts with this energy at a strength proportional to its energy just as it does to all other forms of mass and energy (which avoids the need to formulate the assumption as part of a quantum gravity theory), is a sufficient assumption to produce the same phenomenological consequences, although that approach is less intuitive and makes the math harder.
While the core concepts of General Relativity are a key pillar of fundamental physics, the exact manner in which Einstein expressed those core concepts mathematically has far less respect within the ranks of expert physicists than they do among educated laymen. If Deur hasn't missed something really basic in his analysis, in a part of general relativity theory that has been widely avoided based upon the textbook lore that it was a dead end, it turns out that this scorn has been well deserved.
Prospects For Future Fundamental Physics Research
Renormalization With Quantum Gravity
One of the notable features of the Standard Model is that many of the physical constants in it are functions of the energy scale of the interactions in which they are measured. All of the Standard Model's mass constants (with the possible exception of the neutrino masses if they are not Dirac masses), and all of its coupling constants "run" with the energy scale at which they are measured.
In a quantum gravity theory, there is every reason to expect that the gravitational coupling constant (or equivalently, the Planck mass which is the inverse of the square root of eight pi times the gravitational coupling constant G), runs with energy scale as well. In one theoretically well motivated analysis, the square of the Plank mass at energy scale "k" is expected to run to a value of the square zero energy Plank mass plus the k^2 times a physical constant approximately equal to 0.05.
The incorporation of gravity into the Standard Model would also add a term to each of the other beta functions of the Standard Model that govern the running of each of its physical constants that run with energy scale. At energy scales much smaller than the Plank scale, the impact of these additional terms in the beta functions is negligible. But, at high energies approaching the Plank scale, these contributions would be appreciable.
Mikhail Shaposhnikov and Christof Wetterich, in their ground breaking paper "Asymptotic safety of gravity and the Higgs boson mass" (arXiv:0912.02088 submitted January 12, 2010) made one of the most accurate predictions of the Higgs boson mass (126 GeV +/- 2.2 GeV) before it had been measured, using this kind of analysis with an incomplete fragment of a hypothetical quantum gravity theory.
The Higgs boson mass falls as energy scales get higher. They assumed, for some well motivated reasons, that the Higgs boson mass would hit zero at the Plank scale, and then used the Standard Model beta function for the Higgs boson, modified by a term to account for the gravitational impact on the running of the beta functions, to calculate the minimu Higgs boson mass that could be calculated back from that boundary condition, using the top quark mass and strong force coupling constant as calibration points.
If the gravitational coupling constant does run and does influence the running of the other Standard Model physical constants at high energies, which the success of Shaposhnikov and Wetterich's prediction would tend to support, and the complete set of fundamental laws of physics really does amount to the Standard Model because a GR-like graviton, then the only thing we need to add to our knowledge to predict the way that the law of physics works at high energies approaching the Plank scale is to correctly determine the form of the gravitational terms added to the Standard Model beta functions.
Shaposhnikov and Wetterich didn't need to determine those to reach their conclusion, because they found a way to make all of the beta function terms that they didn't know cancel out in their calculations. But, if the universe turns out to be fully explained by the Standard Model plus a graviton, then these calculations become the biggest unsolved problem in fundamental physics apart from neutrino physics, and fall almost exclusively to the theoretical physics community rather than to the experimentalists.
There still isn't a really solid analysis of how inserting gravity terms into the beta functions of the Standard Model, as Shaposhnikov and Wetterich did at an ansatz level, impacts gauge coupling unification in the Standard Model. If the introduction of these terms can be shown to lead to a gauge unification of the Standard Model coupling constants at the Planck scale (a feature of SUSY already preserved in SUGRA theories at the GUT scale), then the case that the Standard Model really is complete, even if a more elegant underlying preon or string model may explain it, becomes truly compelling.
Impact On Astronomy Research
If Deur's work is mathematically validated and confirmed when the observational evidence is re-evaluated in light of his analysis, it will dramatically take the wind out of the sails of large numbers of direct dark matter detection experiments, and the order of the day will be to reconsider a wealth of astronomy data already existing in this new framework and to identify new astronomy observations that could test it.
For example, it makes observations designed to test the strength of gravity between isolated pairs of galaxies or galactic clusters whose masses can be estimated accurately by looking at the behavior of isolated intergalactic stars a critical priority for astronomers, that otherwise might not even have been examined with any real precision.
It would also prompt a minor revolution in cosmology that would likely lead to the demise of the reigning six parameter lambda CDM model, even though it fits the available data very well.
Impact of High Energy Physics and Theoretical Physics
While Deur's work alone wouldn't directly challenge any of the beyond the Standard Model physics theories that are being tested at the LHC, most of which already include massive spin-2 gravitons that are necessary in his General Relativity extension, it would eliminate the single most powerful motivating reason to look for beyond the Standard Model physics: the near certainty that there must be one or more new beyond the Standard Model particle out there in the dark sector that needs to be explained.
Suddenly, there is no longer a need for even a singlet keV scale mass sterile neutrino to explain the experimental evidence. Suddenly, it is possible without further experiment, to calculate how Standard Model particles should behave all of the way up to the Plank scale in a manner that fully incorporates all four known forces in the universe.
Suddenly, SUSY and supergravity theories that preserve or nearly preserve R-parity in a way that gives rise to a long lived lightest supersymmetric theory, look like they have a bug, rather than a feature. Similarly, stable axions, which are now touted as a potential explanation of both dark matter and the strong CP problem, no longer look nearly as attractive.
The hierarchy problem, in light of the analysis of Shaposhnikov and Wetterich, strengthened in a Baysean manner by their accurate Higgs boson mass prediction and by the strong inference from Deur's analysis that gravitational energy really is localized in gravitons, looks like a natural side effect of asymptotic safety at the Plank scale.
More generally, the experimental motivation for pursuing any kind of theoretical research program that suggests the existence of beyond the Standard Model particles or forces collapses. It may take a generation for the old guard members of the academy to die for this conclusion to really sink in, but the next generation of string theorist and GUT modelers will likely find it much more attractive to ruthlessly limit their inquiries to models that can exactly reproduce the Standard Model plus graviton particle set, than the generation that preceded them did.
Less glamorous challenges, like trying to understand the physics of dense condensed matter object in space like neutron stars, looking for exotic hadrons, explaining scalar and axial vector mesons, the theory of patron distribution functions, will start to look more promising than the grand race to find a GUT theory that merely post-dicts Standard Model and gravitational physics that we already understand, except for the "why" we have the laws of physics that we do instead of something different. Increased precision in the measurements of the soon to be completely measured set of fundamental physical constants will also narrow down even the options for those kinds of inquiries.
In principle, these conclusions should only slightly refine research into the baryogenesis and leptogenesis, and cosmological inflation. But, psychologically, it is going to make research programs that try to tease these phenomena out of the already known laws of the universe, like Higgs inflation theories, look considerably more attractive relative to those that propose new physics, than is the case today.
Obviously, if Deur's work does pan out, the attractiveness of working on trying to turn incomplete theories of quantum gravity into comprehensive ones blossoms, in light of a powerful hint in the right direction, although many existing branches of theoretical work in general relativity and quantum cosmology, such as massive gravity theories, wither on the vine.
[UPDATE September 2, 2014] Physics Forum has a nice analysis of the issue of self-gravitation in Einstein's equations at a thread here.