ΛCDM Predicts The Wrong Mix Of Dark Matter Halos

Stacy McGaugh has another enlightening article at his blog on the discrepancies between the ΛCDM model prediction and the empirical evidence that has been apparent since the 1990s and not resolved within the ΛCDM model.

In a nutshell, if you try to apply the ΛCDM[1] to predict the "halo mass function" which is the mix of dark matter halos in the universe by mass, scaled to the proper quantity of dark matter overall, the ΛCDM prediction is wrong.

ΛCDM basically predicts a steady power law relationship between halo masses of a given size at all scales. But, if you interpret the observational data on galaxies (e.g. their luminosity and rotation speed) in a model dependent way to fit a dark matter particle paradigm, you see a steep drop off in observed very large halos in real life that is not predicted by the theory. In real life there are also too few small dark matter halos which should manifest as dwarf and satellite galaxies, relative to the theoretical expectation.

Incidentally, power law relationships with a sharp drop in frequency above a certain maximum threshold, similar to the ΛCDM prediction for the halo mass distribution, but tweaked to have the cutoff scale observed empirically, are quite common in ordinary life. For example, this describes the relationship between magnitude and frequency of earthquakes, floods and tornadoes on Earth.

Some of the key summaries of the data are in the charts below:

Note the logarithmic scale. Thus, the predicted frequency in the chart (the solid line) above of 1,000,000,000 solar mass halos (eyeballing it), is about 500 times too high relative to the observed data (blue squares and purple triangles). The solid prediction line is too high across the board and is too steep at low masses and too shallow at high masses.

The discrepancy is less serious at low masses, but remains profoundly off for high mass halos, even when halos mass is estimated directly using rotational velocity as show below (which isn't very informative for very low mass galaxies because its sample doesn't include them), which is a more direct calculation than the luminosity based estimates shown above.

We can fit a "Schechter function" to the actual data, but that is not a principled theoretical prediction drawn from first principles and is instead merely an empirical phenomenological fit to the data (much like MOND, which works the vast majority of the time over a broad range of applicability from solar system scales to large single galaxy scales but fails in galactic clusters, but is just an observed fit to the data without an underlying theoretical mechanism).

The ΛCDM model predicts that the most massive galaxies should be several hundred times more common than they are and it is harder for astronomers to miss hundreds of huge galaxies than it is for them to miss large numbers of tiny ones. In fairness to the ΛCDM model, however, none of its serious competitors does a particularly good job of explaining why we observe no galaxies with masses in excess of 10,000,000,000,000 solar masses either (although this issue hasn't been seriously analyzed one way or the other, for lack of a sufficient group of scientists actively investigating alternatives, for many of the non-paradigm alternatives). Stacy McGaugh answered my question at his blog regarding the best available hypothesis for why this is the case, stating:

The predominant idea for this goes back to the mid-70s, and essentially says that the gas cooling time has to be shorter than the gravitational collapse time. Basically, there has to be time for the gas to collapse. This comes out to about the right upper mass limit for individual galaxies, but is rather crude in many respects – the cooling time is very sensitive to the metallicity of the gas; a spherical cow that ignores hierarchical substructure is assumed, and so on. Still, that’s the best I’ve heard.

There is also an outstanding issue regarding whether the largest halos ought to be associated with structures like galaxy clusters, rather than individual galaxies, that complicates the analysis.

Other issues with the ΛCDM model.

A review article from 2017 cited in the blog post linked above examines more problems with the "Standard Model of Cosmology."

The dark energy plus cold dark matter (ΛCDM) cosmological model has been a demonstrably successful framework for predicting and explaining the large-scale structure of Universe and its evolution with time. Yet on length scales smaller than ∼1 Mpc and mass scales smaller than ∼1011M⊙, the theory faces a number of challenges. For example, the observed cores of many dark-matter dominated galaxies are both less dense and less cuspy than naively predicted in ΛCDM. The number of small galaxies and dwarf satellites in the Local Group is also far below the predicted count of low-mass dark matter halos and subhalos within similar volumes. These issues underlie the most well-documented problems with ΛCDM: Cusp/Core, Missing Satellites, and Too-Big-to-Fail. The key question is whether a better understanding of baryon physics, dark matter physics, or both will be required to meet these challenges. Other anomalies, including the observed planar and orbital configurations of Local Group satellites and the tight baryonic/dark matter scaling relations obeyed by the galaxy population, have been less thoroughly explored in the context of ΛCDM theory. Future surveys to discover faint, distant dwarf galaxies and to precisely measure their masses and density structure hold promising avenues for testing possible solutions to the small-scale challenges going forward. Observational programs to constrain or discover and characterize the number of truly dark low-mass halos are among the most important, and achievable, goals in this field over then next decade. These efforts will either further verify the ΛCDM paradigm or demand a substantial revision in our understanding of the nature of dark matter.

James S. Bullock, Michael Boylan-Kolchin, "Small-Scale Challenges to the ΛCDM Paradigm" (July 13, 2017, last updated September 2, 2019) arXiv 1707.04256

See also:

We present rotation curve fits to 175 late-type galaxies from the Spitzer Photometry & Accurate Rotation Curves (SPARC) database using seven dark matter (DM) halo profiles: pseudo-isothermal (pISO), Burkert, Navarro-Frenk-White (NFW), Einasto, DC14, cored-NFW, and a new semi-empirical profile named Lucky13. We marginalize over stellar mass-to-light ratio, galaxy distance, disk inclination, halo concentration and halo mass (and an additional shape parameter for Einasto) using a Markov Chain Monte Carlo method. We find that cored halo models such as the DC14 and Burkert profiles generally provide better fits to rotation curves than the cuspy NFW profile. The stellar mass-halo mass relation from abundance matching is recovered by all halo profiles once imposed as a Bayesian prior, whereas the halo mass-concentration relation is not reproduced in detail by any halo model. We provide an extensive set of figures as well as best-fit parameters in machine-readable tables to facilitate model comparison and the exploration of DM halo properties.

Pengfei Li, Federico Lelli, Stacy McGaugh, James Schombert, "A comprehensive catalog of dark matter halo models for SPARC galaxies" (January 28, 2020). arXiv 2001.10538

Other phenomena that are hard to explain with the ΛCDM model are (1) dwarf galaxies with almost no apparent dark matter (which MOND predicted in the early 1980s and explains with the external field effect), (2) the behavior of wide binary star systems in which the stars appear to be bound by a gravitational force stronger than predicted by general relativity, (3) the fact that inferred dark matter halo shapes differ from the theoretically predicted NFW distribution, and (4) the fact that inferred dark matter distributions are tightly linked in a predictable manner to observed ordinary matter distributions even at a fine scaled level that is basically impossible to explain with truly collisionless dark matter particles that interact exclusively through gravity in a manner prescribed by General Relativity.

This tends to imply that dark matter particles, if they exist, must either be subject to a force that acts between dark matter particles (self-interacting dark matter), probably (based on cluster collision data) with a coupling strength on a order similar to that of the electromagnetic force, or a small in coupling strength fifth force governing interactions between dark matter and ordinary matter, or both. So, viable dark matter particle theories need one or two new forces (presumably with at least one beyond the Standard Model non-graviton carrier boson each) in addition to at least one new, beyond the Standard Model, non-graviton dark matter particle. All of these possibilities are significant extensions of the ΛCDM model that materially impact what it would predict. 

There are also some issues with the ΛCDM model when compared with observational data at the larger cosmology scale. Most notably, (1) galaxies form too quickly (the impossible early galaxies problem), (2) 21cm wavelength radiation data that demonstrates the temperature of the universe at 180 million to 280 million years after the Big Bang, seems inconsistent with the model because the universe was much colder than predicted and is instead consistent with a no dark matter hypothesis, and (3) the velocity of colliding galaxy clusters is also too often higher than it predicts.

There are also serious concerns about the cosmological constant part of the ΛCDM model which may be overestimated or non-existent due to several methodological issues with how the value of this parameter is determined based upon the available data. See, e.g., a post at this blog here. The jury is out regarding whether these concerns are well founded or lead to the conclusions that the people proposing them have suggested. This is a fairly recent and still hotly debated issue in cosmology and astronomy, and beyond the scope of this post.

In fairness, while the ΛCDM model is contradicted by the observational evidence at small scales in many respects, it is still a decent zeroth order approximation of reality, that could conceivably be tweaked to better fit observation through some mass assembly process that is currently not understood and by factors such as gravitational interactions with distributions of ordinary matter, although this possibility looks quite doubtful given everything we know so far.

Most strongly supporting it, the ΛCDM model compares favorably in predicting the dynamics of particles near the fringes of spiral galaxies that are outside the main plane of the spiral, relative to many leading modified gravity theories. 

What the does the ΛCDM model gets right?

The ΛCDM model does do an excellent job of predicting the nature of the cosmic microwave background radiation (which has slight warm and cold spots) that is observed, although it turns out that this is easier to fit than one might naively assume. This is because some of the parameters are determined relative to other parameters, rather than being truly independent of each other, so fewer degrees of freedom involved that have to be fit than one would expect just looking at the raw data in a superficial manner.

The ΛCDM model also does a decent job of explaining why observable matter has a web-like or cell-like structure at the largest scales with filaments of higher matter density surrounding comparative voids of matter.

What observational data sometimes touted as a problem for dark matter particle theories does not materially undermine the ΛCDM model?

The lack of direct detection of dark matter particles in dark matter detection experiments confirms that if dark matter particles exist that they do not have meaningful electromagnetic, weak force, or strong force interactions with other particles of ordinary matter, but this is entirely consistent with the standard, unmodifed ΛCDM model. 

The direct dark matter detection experiments do tell us that any interaction that dark matter particles have with ordinary matter must be much weaker than the weak force of the Standard Model between ordinary matter particles (which is mediated by Z boson exchange). 

The observed frequency of decay products of Higgs bosons, W and Z bosons likewise supports the hypothesis that if dark matter particles exist, dark matter particles do not decay via the weak force if individual particles have a mass of 62.5 GeV or less (with the available galaxy dynamics data favoring a dark matter particle mass on the order of 1-10 keV over heavier masses if dark matter is created through the most popular thermal freeze out hypothesis). 

Given evidence that dark matter particles, if they exist, are stable or metastable over time lines comparable to the age of the universe (or are created and destroyed at almost exactly identical rates, perhaps due to some sort of conservation law), the lack of weak force decays is also unsurprising.

Likewise, because the ΛCDM model takes no position on how dark matter particles come into being, whether they decay, or how it can be destroyed, the lack of dark matter annihilation signal detections and the lack of collider signatures of dark matter neither contradicts nor supports the ΛCDM model.

The available data does not clearly favor models in which dark matter particles are fermions, over models in which dark matter particles are bosons, but the ΛCDM model is agnostic on that issue.

[1] Pronounced lambda "C" "D" "M", in which lambda is the cosmological constant, and CDM stands for "Cold Dark Matter" but really means "warm" or "cold" dark matter which is "nearly" collisionless with ordinary matter and other dark matter and hence has dynamics that are predominantly a function of gravity alone.  The aggregate amount of CDM in the model is fixed from moments after the Big Bang to the present, while the amount of dark energy (described by the cosmological constant) is proportional to the spatial volume of the universe at any given point in time, and hence grows over time. This model is also known as the Standard Model of Cosmology and is the prevailing paradigm, despite an ever growing list of discrepancies with the empirical evidence. ΛCDM has a good (but not perfect) track record relative to observational data at scales larger than galaxy clusters, but a very poor track record at scales of galaxies and smaller systems.