Which Ancestries Does 23andMe Botch Most Badly?

There is a somewhat widely known problem with the 23andMe ancestry estimation techniques for Koreans. It turns out, that the company's own data confirms this fact. 

The Company has a chart measuring precision and recall for each ethnic category. Precision is the likelihood that a particular genetic unit estimated to be from someone of a particular ethnicity really is from a person of that ethnicity. Recall is the likelihood that a gene from a person of a particular ethnicity is accurately classified as such. Here is the breakdown (with poor performance noted in bold):

POPULATION	PRECISION (%)	RECALL (%)
Sub-Saharan African	99	99
West African	97	96
East African	95	89
Central & South African	100	89
East Asian & Native American	99	99
Native American	99	86
East Asian	97	97
Japanese	98	92
Korean	86	62
Yakut	96	78
Mongolian	89	53
Chinese	93	91
Southeast Asian	95	70
European	99	99
Northern European	95	85
British & Irish	90	39
Finnish	95	86
French & German	78	8
Scandinavian	86	34
Southern European	93	66
Balkan	88	42
Iberian	92	51
Italian	88	50
Sardinian	96	62
Eastern European	90	50
Ashkenazi Jewish	97	93
Middle Eastern & North African	95	83
Middle Eastern	90	76
North African	95	77
South Asian	99	95
Oceanian	100	95

Koreans are known to be often misclassified and sure enough, the numbers are bad. Only French and German rates lower than Korean by predictive accuracy. There are less accurate recall rates than those for Koreans, but 62% is nothing to write home about and it is far less accurate than for other East Asian ancestries.

There are also known problems with Southern and Eastern Europeans, all of whom have very low recall rates. Among Northern Europeans, only Finnish people really stand out as distinct.

Wicked Problems v. Trivial Problems

As my regular readers know, this blog focuses mostly on the broad issues of prehistory and fundamental physics. Why not other areas of science?

A useful way of understanding the class of problems that interest me, at least from a blogging perspective, is that I am interested in what this blog post has called "wicked problems", in contrast with "trivial problems," of the kind that their blog focuses upon.

Two major technical fields I know have areas that call themselves “complexity”—ours and dynamical systems. . . . [Another sense] reflects the way we use “complex” in ordinary speech: big, involved, hard-to-describe, requiring much effort even to comprehend. It included a cool notion that I think you may like called wicked problems . . . . This is different from our notion of a problem being wickedly hard—it has to do with framing the problem itself. . . .
The key is the notion of a wicked problem(WP). It is not that easy to define what they are, but here is the standard list of properties they have:

*The problem is not understood until after the formulation of a solution.
*Solutions to wicked problems are not right or wrong.
*Every wicked problem is essentially novel and unique.
*Every solution to a wicked problem is a “one shot operation.”
*Wicked problems have no given alternative solutions. . . .

In 1967 West Churchman introduced the concept of wicked problems in a journal article. . . . Horst Rittel and Melvin Webber formally described the concept of wicked problems later in 1973 where they contrasted WP’s from “trivial” problems such as those from mathematics, chess, or puzzle solving.

Put another way, I enjoy the unstructured fumbling around trying to figure out how to determine what is going on and how to make sense of it and problem defining part of the process from unstructured or contradictory data, as much as I do the colder and more purely rational process of solving a well defined, but possibly very difficult, problem.  I like big, messy, unsolved "one shot" problems.

Three senses of the word "complexity"

Put another way, the post distinguishes between three kinds of complexity, the first of which is the wicked problem sense (called it "real-world based" complexity) described above.

Their blog focuses on complexity in a second sense of problems that are wickedly hard to solve (generally the address whether it is theoretically possible to solve a kind of problem in a reasonable amount of time defined in a technical way, which is used to evaluate computer algorhythms and cyptography strength) because they are resource intensive to solve:

When we say “complexity” we almost always are referring to the resources needed to compute and solve some problem. The resources measured can be: time, space, randomness, nondeterminism, rounds, quantum operations, or other resources. Indeed sometimes we mix these together to get measures that limit two or more resources.

They also mention a third sense of the term which they call behavior based complexity, which is also a more common sense of the word than theirs, as used by places like the Sante Fe Institute, and which they claim is based upon:

[T]he notion of “a complex system.” . . . When many say complexity theory, they usually are referring to the type of behavior of a system. This type of complexity theory is all about behavior, about prediction, about chaos, about the tremendous forking of paths (bifurcations) that can arise from even simple systems that evolve over time.

I would quibble with their definition here. "Chaos theory" generally refers to deterministic dynamical systems whose outcomes are hard to predict because they are highly sensitive to initial conditions (like the weather), even if you know the often simple equations that govern them exactly, causing their outcomes to seem quasi-random, and happens to be related for non-obvious reasons to fractals.

Complexity theory is a kindred but analytically distinct endeavor (closely aligned with what sociologists and business management professors mean when they talk about "complex organizations"), and while chaos theory typically involves simple underlying equations, complexity theory does not.  Complexity theory involves notions like "emergence," the conditions that cause systems to be complex, and how to apply simply strategies to deal with complex systems.

Still, while I quibble with their definition, I know what they mean and I agree that it is distinct. Indeed, I think that complexity is a pretty poor term to define the study of how intrinsically difficult certain classes of problems are to solve computationally, in terms of the resources necessary to solve them. It is a very valid area of study, but "complexity" theory is a poor name for it, because the problems that are addressed are generally straightforward and well defined.

The Strategic Aspects Of A Research Agenda

In designing a research program, it is useful to distinguish between uncanny and mundane predictions. An uncanny prediction is one that is incongruous with both current scientific understanding of the phenomenon and any corresponding folk models, i.e., the prediction specifies aspects of the world that are hitherto unrecognized. In contrast, a mundane prediction is one that is consistent with our current understanding of the world, be it scientific or folk. If supported, uncanny predictions have a large impact on scientific knowledge – they provide substantial prima facie evidence supporting the hypothesis from which they were derived, and they open up new areas of empirical exploration. In contrast, when mundane predictions are supported, they have far less impact on scientific knowledge. Typically, a variety of existing perspectives can account for familiar phenomena, hence supported mundane predictions provide marginal evidence for the hypothesis from which they were derived; likewise, because the given effects are already familiar, such findings do not lead to new areas of empirical exploration. Most uncanny predictions will fail, and most mundane predictions will succeed. This is because existing scientific perspectives, and many folk models, will generally be accurate, hence hypotheses that are incongruent with such knowledge will often be incorrect, while hypotheses that are congruent with it will often be correct. Phrased in cost/benefit terms, uncanny predictions are thus a high-risk, high-yield enterprise, while mundane predictions are a low-risk, low-yield enterprise.

From here.

Pinning Down Archaic Admixture Population Models

There are many outstanding disputes, critical to understand the demographic history of Eurasian modern humans in the Upper Paleolithic era, related to the population models that are used to describe how Neanderthal genes could have ended up in almost all modern Eurasians at frequencies on the order of 2%-4% of our autosomal genome in a sample made up of many thousands or tens of thousands of individuals, despite a complete absence of Neanderthal mtDNA or Y-DNA in any genetically tested modern human, from a large sample of tens or hundreds of thousands of individuals, including hundreds of ancient DNA samples. This population genetic data has been accumulated in a collective scientific enterprise that has deliberately oversampled populations that are likely to be genetically diverse outliers in both data sets, although there are far more outlier populations and ancient DNA populations that are undersampled for autosomal genetics than there are that have been undersampled for mtDNA.

One of the confounds in estimating what kind of process gave rise to the introgression of Neanderthal DNA into modern humans is the question of how much of the Neanderthal DNA originally present in hybrid individuals has been purged over time from modern humans, either due to random genetic drift in admixed modern human populations, or due to selective disadvantage associated with particular Neanderthal genes.

It helps, in comparing possibilities that we have significant shares of the Neanderthal genome from ancient DNA to compare against modern genomes.

Neanderthal genes that could have introgressed into modern humans can be broken into one of four categories: (1) genes in which the Neanderthal genome and modern human genome are indistiguishable (which is a very substantial share of the total, probably on the order of 95% or more), (2) Neanderthal genes with a positive selective advantage (there is some early indication that this may mostly consist of HLA genes which are related to the immune system, (3) Neanderthal genes that have a selective disadvantage relative to modern human genes, which statistically should have been removed from the human genome over the relevant time spam of at least 30,000 years or so, and quite possible two to four times as long as that, even if the selective disadvantage is very modest, particularly as disadvantageous genes slowly become separated from nearby genes that may have selective advantage through the recombination process over many generations, and (4) Neanderthal genes that are selectively neutral.

One can determine in modern populations which Nenderthal genes are present at elevated frequencies indicative of selective advantage and which are only present at a baseline level, in order to both estimate the true selectively neutral baseline level of admixture before selection started to act on the genes in modern humans with Neanderthal ancestry, and to estimate the magnitude of the advantage associated with those genes present at elevated frequency. This task is somewhat harder than it seems because one has to address statistical noise that elevates the frequency of some random genes for reasons unrelated to selective advantage, but is well within the capabilities of well established statistical methods.

One can also search, by direct comparison, for distinguishably Neanderthal genes that have not ended up in any modern human at all. There are basically three ways that this could happen: (1) the genes were never transferred in an admixture event because there were a finite number of admixture events and only an approximately random half of the Neanderthal genome was transferred in each event, so some genes may never have been transferred in any of the events, (2) the genes were transferred in an admixture event and left the modern human genome via random genetic drift, (3) the genes were transferred in an admixture event but due to selective disadvantage associated with the genes, they were culled from the modern human genome. The percentage of Neanderthal specific genes known to exist which are found in no modern human populations can provide a very accurate estimate of the combined impact of these three factors, although by itself, it doesn't do much to tell you how much of each factor plays a part.

It is mathematically trivial to relate the impact of the first factor to the number of admixture events that took place, and the relationship between the percentage of genes never transferred in admixture events and the number of admixture events is highly non-linear. For one admixture event, the percentage of 50%. For two it is 25%. In general, the never transmitted proportion of the genome is 1/(2^n) where n is the number of admixture events. In any scenario where there are seven or more admixture events in all of human history, the percentage of Neanderthal specific genes never transmitted in admixture events is below 1% and at somewhere on the order of twelve to fourteen admixture events ever in all of modern human history, the impact of this factor would be completely undetectable with any level of statistical significance in an autosomal genome data set as large as the one that is currently in existence.

If the effective population size of the modern human populations that admixed with Neanderthals was on the order of four hundred to seven hundred and fifty individuals, the effect of non-transmission of specific genes in any admixture event should be negligable, and even at an effective population size as low as two hundred, the impact of this factor should be a very small proportion of the total number of Neanderthal genes not observed in any modern human population. Yet, most estimates of the effective population size of the founder population of modern human Eurasians are at least in the single digit thousands, and archaic admixture itself, while it would inflate the apparent effective population size of the founder population of modern human Eurasians, at the 2.5%-4% of the total population size would not have an effect so significant that it would bring the effective population size of the founding population of modern human Eurasians to the low three digits, particularly to the extent that the estimates are corroborated by mtDNA and Y-DNA based estimates that have on archaic component.

This means that essentially all of the "missing" Neanderthal DNA (at least outside the sex chromosomes where there are clearly population structure and demographic history factors that are non-random at play) must statistically derive from either genetic drift or selective disadvantage.

We can then work to estimate both components separately using a variety of population genetic parameters, and work to look at the parameter space of assumptions that can produce outcomes consistent with the percentage of missing Neanderthal DNA that we observe.

Random drift of selectively neutral genes is easy to model with very accurate results using just a handful of parameters, either analytically, or numerically with Monte Carlo methods. Some of the key parameters are generation length, effective modern human population size at the time of admixture, number of admixture events, spacing of admixture events, boom and bust variability in effective modern human population size, and population growth (which can be quite accurately estimated in the long run from a variety of evidence, even if fine grained variability in this rate is hard to determine).

For populations that experience growth in the long run (as modern humans in Eurasia obvious did), where the number of generations is very large, it turns out that generation length doesn't actually matter very much, because when you have a number of generations in excess of one thousand with a population that reaches the many millions sometime in the Upper Paleolithic, and an overall percentage of admixture that is at least on the order of the 2.5%-4% it has reached at long term fixation, which has apparently been reached for all Eurasian given the supercontinental uniformity present in that percentage, the amount of genomic loss that takes place due to random drift bceomes insensitive to the number of generations because random drift is much more powerful an effect, in a non-linear manner, when populations are small. At a leading order estimate, the likelihood of losing a gene entirely from a population in any given span of generations is a non-linear function of the absolute number of individuals in the population who carry that gene. Basically, the percentage likelihood that a gene will leave the population by random drift is roughly proportional to the probability that a random sample from the effective population equal to the absolute number of gene carriers in the population would be zero. Once the absolute number of carriers and zero is several sample error standard deviations apart from a sample of that size, the probability of loss of a gene entirely due to random drift approachees zero.

Complicating this is a factor that also looks like random drift, which is mutation. While not listed as a separate factor, another way that a gene can be removed from the gene pool is through a mutation at that locus. The probability of this happening is a function of the number of generations involved and the effective population size of each generation, divided by the number of carriers of a particular gene, and discounted for the fact that lots of mutations are lethal and never enter the gene pool. This is the method used to make estimates of the age of mtDNA and Y-DNA haplogroups and it isn't very accurate, but there is a considerable body of empirical evidence that put order of magnitude bounds on the size of this effect. So, whlle the error bars on this component of the random loss of selectively neutral genes from the population might have extremes that vary by as much as a factor of two to ten if we were really being realistic about who precise our methods of mutation dating have proven to be in practice (perhaps more, given that the timing of the admixture event has something on the order of a factor of two uncertainty in it to begin with and that our estimates of generation length in Upper Paleolithic modern humans aren't terribly accurate and our effective population chart also has a pretty fuzzy line), if the effect is at an order of magnitude lower than other sources of removals of genes from the population's genome, we can safely ignore it, even if the precise magnitude of the effect is not known with a great deal of certainty.

From the other direction, there have been a number of reasonably useful estimates of the proportion of genes in the human genome, and the proportion of genes in a variety of other species, which do, or do not, show indications of having a selective effect at any given time (which basically consists of genes that have not reached fixation in the species for which there is no good reason to believe that selection produces multiple varieties in stable proportions as it does for HLA genes). In general, these studies have shown that the proportion of genes that are currently experiencing active selective pressures at any given time appear to be fairly modest, but not negligable either.

There is no really good way to estimate the relative numbers of selectively advantageous archaic genes to selectively disadvantageous archaic genes. There argument for more good genes than bad is that Neanderthals had more time to adapt to the new environment that modern humans were entering. The argument for more bad genes than good is that Neanderthals went extinct while modern humans didn't, so overall, modern humans had a selective advantage of some form over Neanderthals. But, it isn't unreasonable to infer that there should be an order of magnitude similar number of each. There is also no particularly good reason to think that the proportion of the genome that is selectively neutral at any given point in time has changed very much or was much different from Neanderthals than it is for modern humans. So, an examination of the number of Neanderthal genome genes present in elevated levels that hence show signs of selective advantage could cast some light, at least, on the proportion of Neanderthal genes that gave rise to selective disadvantages and were purged from the modern human genome. The early indications from this kind of analysis are that the proportion of Neanderthal genes still in the modern human genome which show signs of having been positively selected for is small relative to the total number of Neanderthal genes in the modern human genome.

Despite the fuzziness of all of these reasoning, from a quantitative perspective, the bottom line in all of this analysis is that we would expect a significantly disproportionate share of the proportion of missing genes from Neanderthal genome to have been lost due to selectively neutral random drift rather than natural selection, and that even this crude bound allows us to make fairly specific numerical estimates of the proportion of Neanderthal specific genes that were lost because they were selectively disadvantageous and the proportion of Neanderthal specific genes that were lost due to one of a couple of forms of random genetic drift.

Placing numerical bounds and maximum likelihood estimates on the proportion of Neanderthal specific genes that were lost due to random genetic drift with this kind of analysis, in turn, allows us to significantly narrow the parameter space of population model assumptions that could produce the observed amount of random genetic drift. The observed proportion of random genetic drift in the Neanderthal genome would be particularly relevant in placing bounds on the paramater space for assumptions about effective modern human population size at the time of admixture, number of admixture events, spacing of admixture events, and the boom and bust variability in effective modern human population size. And, there are independent ways to provide additional bounds on many of these parameters from other lines of population genetic data and anthropology and the physical anthropology of Neanderthal remains, so the flexibility in one paramater doesn't inject too much flexibility into other paramaters.

Also, a reasonably tightly bound overall estimate of the magnitude of random genetic drift from the proportion of the Neanderthal genome that has been purged from modern humans provides a robust, and fairly direct estimate, from the longest time period for which ancient DNA is available for hominins, that can be used to inform estimates of the rate at which selectively neutral genes are purged by genetic drift in modern humans that is relatively population model independent for use in analysis of non-Neanderthal admixture population genetics (e.g. in estimates related to Denisovian admixture, putative African archaic admixture, admixtures of modern human populations in the Upper Paleolithic era, and the accuracy of estimates of the probability that a chance in the proportion of a particular gene in a population was due to random genetic drift or selection), since the error bars on this direct measure of random genetic drift in autosomal genes over that time period would be much smaller than the error bars around estimates of any of the specific parameters in parameter space that could be used to estimate it from first principles using population models alone. Thus, making this estimate in the Neanderthal case would materially improve the statistical power of all of our long term population genetic estimates, a contribution that may be unique and may not be available with greater precision from any other set of data for the foreseeable future.

Explicitly estimating the impact of selective effects and the loss of genes due to random genetic drift is also likely to establish that the total number of archaic admixture events was larger than an estimate that ignores these effects, because, on balance, these effects tend to reduce the number of Neanderthal genes in the modern human genome. Thus, the process of estimating these numbers of likely to reveal that Neanderthals and modern humans had sex more often that a crude back of napkin estimate would suggest. And, if the kind of process assumptions (Haldane's rule which also impacts fertility assumptions, and predominantly female modern human mothers for hybrid children born into modern human tribes that averted extinction, which implies that there are large numbers of uncounted cases where Neanderthal mothers that were erased from modern human populations in the present) that most naturally explain the disconnect between autosomal genetic data and uniparental genetic data are also incorporated into the analysis, the amount of cross species sexual activity between Neanderthals and modern humans may have been quite a bit higher indeed than the current percentage of our autosomal genome attributable to Neanderthal genes would suggest, probably on the order of a factor of two to five, which would be roughly the difference between a once in a generation event (a crude estimate without these considerations) and something like a once every few years event.

My intuition is that the amount of allele loss due to random genetic drift acting on selectively neutral genes that is actually observed in the Neanderthal case would suggest that the magnitude of the impact of random genetic drift in purging selectively neutral genes from modern human populations is quite a bit smaller than could safely be inferred by a naiive estimate based on other existing data and pure population modeling not supported by this kind of empirical calibration. Thus, I suspect that this data will, generally, favor findings that it is more likely that a given chance in gene frequency was a selective effect rather than a random one, and that populations not subject to selective pressures are more genetically stable than one might naiively expect even with a fairly careful theoretical analysis.

The Population Genetics Of Mutts

Population genetic studies that are designed to capture pre-historic genetic diversity and the origins of modern "peoples" focuses on "pure blooded" individuals whose ancestors (typically at least back to four grandparents) have known and identical origins in the locale to which the sample is assigned. There are studies of admixed populations (e.g. mestizos and African-Americans, the people of Madagascar and "colored" Anglo-Caribbeans and South Africans), but those studies usually focus on populations where the admixture was population-wide, took place half a dozen generations or more ago, and may have involved considerably less of a choice element than is found in modern mixed couples.

For the purpose of discerning pre-historic population structure, this kind of selection makes sense.

But, it would be interesting to see, from a genetic diversity and heredity perspective, if there are systemic differences between monoancestral people and multiancestral people, in general.

For example, are there genetic trades related to psychological makeup (e.g. traits linked to the Big Five personality trait of openness to experience, or traits linked to novelty seeking) that are found at markedly different frequencies in monoancestral and multiancestral people?