I agree with basically every single word of Sabine Hossenfelder's recent post at her blog "Backreaction" entitled "Do we need a theory of everything?" (with the possible exception of her stylistic choices in her associated podcast video which I didn't watch).
TOEs in a nutshell
She argues, and I agree, that we know we need a theory of quantum gravity, but we do not need a "Grand Unified Theory" (GUT) that finds a common symmetry group for the entire Standard Model, nor do we need a theory of quantum gravity and a the Standard Model forces to be unified in a single symmetry group constituting a "Theory of Everything" (TOE) in sensu stricto.
I agree that all GUTs and TOEs proposed to date that are capable of being falsified have been falsified, and that there is no positive unexplained experimental evidence that can only be solved by a GUT or TOE.
Early GUTs looked promising, but the most naive and obvious efforts to construct them predicted things like proton decay, flavor changing neutral currents at the tree level, neutrinoless double beta decay, sterile neutrinos, and electroweak scale supersymmetric particles, that have not been observed at a statistically significant level in replicated results at this time.
I agree that trying to find a GUT or TOE by building an expensive new collider employing thousands of scientists does not itself justify that effort relative to other ways that scarce funds available for scientific research could be spent, although there are other scientific justifications for these projects even though they are less compelling (they keep thousands of HEP physics employed testing the limits of science, bring greater precision to our measurement of Standard Model parameters, and promote the development of new technologies and calculation methods developed for the purpose of conducting the experiments that have value of their own).
I agree that "beauty" and "naturalness" have a pretty poor track record of motivating break throughs in physics, and in particular, have not been useful guides for high energy physics theorists in the last four decades. But, I am not quite as bearish as Dr. Hoffenfelder about the potential for these kinds of intuitions to provide at least some useful guidance in hypothesis generation in fundamental physics.
I would add a few additional observations, however, because I wouldn't have a science blog if that wasn't what I did.
No part of Core Theory has been falsified, despite decades of attempts to do so.
General relativity with a cosmological constant is a century old theory. The lion's share of the Standard Model of Particle Physics is forty years old and the modifications to it since then have been relatively minor. These combined constitute "Core Theory"
No experimental evidence has ever disproved any part of the current version of the Standard Model of Particle Physics, mostly put in place in the late 1970s, as modified later to include three generations of fundamental fermions, and either two or three massive, active, Standard Model neutrinos with three weak force flavors and three mass eigenstates that oscillate pursuant to the PMNS matrix.
Likewise, no experimental or observational evidence has ever disproved General Relativity with a cosmological constant as currently operationalized.
I use the phrase "as currently operationalized" because some credible physicists think that the axioms of General Relativity are conceptually sound and correct descriptions of reality but that there are flaws in how we apply or "operationalize" General Relativity to real world phenomena that are incorrect in some manner and that if operationalized correctly, we would have a more accurate description of reality. By using this limitation to what I am talking about when I talk about General Relativity, I am treating proposals to change how we operationalize General Relativity that lead to different predictions than General Relativity as currently operationalized as a form of "BSM" physics.
There are a number of currently unresolved experimental tensions with Core Theory that are not yet significant enough to necessitate "new physics."
This isn't to say that there aren't tensions in the observational and experimental data that could cause this reality to cease to be true. But, none of those tensions have met the gold standard of five standard deviation departures from Standard Model predictions that have been replicated by credible independent experimental groups.
Some of the most notable of these tensions, although certainly not anywhere near a complete list, are:
(1) the discrepancy between the experimentally measured value of the anomalous magnetic moment of the muon and the Standard Model prediction,
(2) suggestions of violation of charged lepton universality (i.e. that electrons, muons and tau leptons behave identically except for their different masses), and
(3) tensions in measurements of the Hubble constant and other astronomy observations favoring explanations of dark energy phenomena other than a simple cosmological constant.
There are at least half a dozen to a dozen other statistical significant experimental tensions with the Standard Model that fall in the two to five standard deviation from the predicted value range.
Some previously significant tensions, like the Pioneer anomaly and the muonic hydrogen radius problem, have disappeared as better measurements and better analysis has eliminated these tensions without resorting to "new physics." Conventional wisdom is that the same fate awaits most of the existing experimental tensions with Core Theory.
There are other experimental claims that are nominally more statistically significant than the five sigma discovery threshold that have not been replicated.
For example, a Moscow experiments claims to have seen neutrinoless double beta decay, that other experiments claims to have ruled out. Other experiments claims to have directly detected dark matter particles or sterile neutrinos that other experiments claim to have ruled out. And, another experiment claims to have detected a 17 MeV particle known as X17 that other experiments purport to have ruled out. An experiment in Italy claimed to have seen superluminal neutrinos which would be contrary to the Standard Model and special relativity, until a flaw in its experimental equipment was discovered.
Conventional wisdom is that each of these apparent discoveries are either actually cases of experimental or theoretical error of an undetermined nature by the scientists making the claims, or will confirmed as soon as another group of scientists gets around to trying to replicate the results (which can take years and millions or even billions of dollars of funding to do).
Core Theory doesn't answer every question we'd ideally like it to be able to answer.
This also isn't to say that the "Core Theory" of the Standard Model and General Relativity with a cosmological constant as currently operationalized, is a "complete" theory that explains everything that we would like it to explain. Most notably, Core Theory:
* does not explain the reason that its roughly two dozen experimentally measured parameters take the values that they do;
* does not explain the mechanism by which Standard Model neutrinos acquire mass;
* does not explain the process by which matter in the form of baryons and leptons came into existence in a manner in which almost all baryons and charged leptons are made of matter rather than antimatter; and
* does not answer other questions about how the conditions of the universe at times prior to Big Bang Nucleosynthesis came to be.
Of course, there are also myriad other scientific questions that Core Theory doesn't answer, because they deal with complex phenomena which should be capable of being understood with Core Theory working from first principles, but in practice, are too complex to be derived from the bottom up in that manner, even though Core Theory can qualitatively inform our understanding of these phenomena.
For example, you can't directly apply Core Theory to learn how many species of dolphins there are in the world, or how to build a better battery, or how to cure cancer, or what the best way is to predict major earthquakes are far as possible in advance. Core Theory ideally gives us the fundamental laws of Nature, but applying them gives rise to emergent phenomena that can't be easily predicted from knowledge of those laws of Nature alone.
We know that we need BSM physics to explain reality and haven't found it yet.
In addition to things left unexplained by Core Theory, we also know that some parts of Core Theory must be wrong because the Standard Model and General Relativity have theoretical inconsistencies and fail to explain, at a minimum, dark matter phenomena.
The most profound and glaring is unsolved problem in fundamental physics is the observation of "dark matter phenomena" which at far more than a five sigma "discovery" threshold of statistical significance cannot be explained with the Standard Model and with General Relativity with a cosmological constant as currently operationalized. Beyond the Standard Model and beyond "core theory" physics (i.e. "BSM physics" or "new physics") are needed to explain these phenomena.
No complete dark matter particle theory, and no gravity modification or quantum gravity theory, and no combination of these theories, have solved the dark matter problem in a manner that has gained wide acceptance among physicists.
All of the most popular explanations of dark matter phenomena fail in some respect or other to explain some dark matter phenomena in a manner consistent with observational evidence. Many of the less popular explanations of dark matter phenomena have simply not be vetted well enough to know whether or not there is evidence that these theories cannot explain, and are thus, not ripe to receive wide acceptance in the scientific community.
This fact alone means that even if there is a TOE in sensu stricto out there to be discovered that it would not simply be the Standard Model attached to a theory of quantum gravity that exactly replicates General Relativity with a cosmological constant except in technical respects that can't currently be observed. A TOE would have to cover everything explained by Core Theory and also some additional BSM physics.
There is good reason to think that a correct theory of quantum gravity, should we devise one, will shed light on a number of outstanding unsolved problems in physics, in addition to the mere technical and logical inconsistencies between classically formulated General Relativity with a cosmological constant.
A theory of quantum gravity might explain dark matter phenomena, might provide an alternative explanation of dark energy phenomena, and might also resolve one or more other unsolved problems in physics, particularly in the subfield of cosmology.
In particular, Deur's work on quantum gravity purports to formulate an alternative that largely reduces to general relativity without a cosmological constant in the classical strong field limit and to explain dark matter and dark energy phenomena as second order quantum gravity effects visible only in very weak gravitational fields, as well as resolving a number of other less pressing unsolved problems in astrophysics and cosmology. It is one of the only outstanding theory of which I am aware which purports to be able to do so in all circumstances that has not been falsified in any situation. But because it hasn't been sufficiently vetted yet by other scientists, the lack of falsification could be due to lack of scientific attention, rather than due to the soundness of the theory.
We have a Core Theory explanation of dark energy, but it is hard to reconcile with many plausible solutions to the quantum gravity problem.
It is also worth observing that while "dark energy" is commonly viewed as an unsolved problem in physics, that General Relativity plus a cosmological constant is one explanation of "dark energy" that is consistent with all observed dark energy phenomena to within experimental uncertainties (although there are some developing tensions that could grow strong as our astronomy observations grow more precise and change this status quo).
But it is much easier to formulate a theory of quantum gravity that replicates, in the classical limit, General Relativity as currently operationalized without a cosmological constant, than it is to do so with a cosmological constant. This is because most quantum theories are described locally, while the cosmological constant is a global rather than a local aspect of the classical theory that is General Relativity.
It isn't impossible to add a cosmological constant to a quantum gravity theory. One way to do so is to add a particle in addition to a graviton to a quantum gravity theory to describe the cosmological constant. There is also a class of quantum gravity theory where it is somewhat less problematic to include a cosmological constant type term, because they utilize quanta of the space-time background (one way to describe the cosmological constant is as the innate curvature of a space-time vacuum or brane), rather than, or in addition to, a carrier boson of the gravitational force known as the graviton which is analogous to a photon (in the case of the electromagnetic force in the Standard Model) or a gluon (in the case of the strong force in the Standard Model).
For this reason, active investigation of alternative ways to explain dark energy phenomena goes hand in hand with research in to a theory of quantum gravity. It seems quite plausible that we may need to undo some of the grand and simple General Relativity with a cosmological constant solution to the question of gravity in order to get to a theoretically consistent theory of quantum gravity.