Losing Neanderthal mtDNA Through Drift (An Old Result)

The Facts Of Our Neanderthal Genetic Inheritance

Not a single modern human today has Neanderthal mtDNA and not a single sample of ancient mtDNA from a modern human is derived from a Neanderthal either. Likewise, not a single man on Earth today, or a single modern human's ancient DNA harbors Neanderthal Y-DNA. Yet, about 4% of the autosomal DNA on non-Africans is derived from Neanderthals and some ancient DNA samples have elevated levels of autosomal Neanderthal DNA.

A Random Genetic Drift Explanation For The Lack Of Neanderthal mtDNA

How could this happen?

A Letter to the Editor of the journal American Human Genetics from Magnus Nordborg entitled On the Probability of Neanderthal Ancestry, written in 1998 before we knew that humans had autosomal Neanderthal DNA discusses this point (starting at page 22 in the linked pdf):

Two simple scenarios for human demography were used—namely, constant population size and constant ancient-population size followed by exponential growth 50,000 years ago. For both cases, the effective number of females in the constant population was assumed to be 3,400, growing exponentially to for the latter 8 5 # 10 case. These parameters were chosen so that the probability would be high that Te lies within the range 100,000–200,000 years, when a generation time of 20 years is assumed. The age of the sampled Neanderthal, ts, was assumed to be 30,000–100,000 years (the recovery of DNA more ancient than 100,000 years seems highly doubtful [Krings et al. 1997]). I argue below that the absolute values of all these parameters are of considerably lesser importance than their relative values. Table 1 gives the results for models of random mating. As expected, the probability that both a compatible topology and an extreme difference between Te and Tr would be observed is low, and, therefore, the hypothesis that modern humans and Neanderthals were a randomly mating population may be rejected. However, closer inspection reveals the more interesting fact that the topology alone may not be unlikely. The reason for this is that, unless the sampled Neanderthal lived long after human populations had started to grow exponentially, most of the modern mtDNA lineages would have coalesced at ts: if, for example, the modern sample only had two ancestors who were contemporary with the sampled Neanderthal, it would not be surprising if they were monophyletic (probability of 1/3). A large difference between Te and Tr, on the other hand, is always unlikely under random mating. . . .  

For the constant–population-size model, for example, assume that Neanderthals and anatomically modern humans merged 1 coalescent-time unit ago (equivalent to tm = 68,000 years, for the population size used above) and that Neanderthals composed 25% of the new population. Then, the probability that all Neanderthal mtDNA was lost through drift is .52 (the probability that Neanderthal mtDNA was not in the sample [calculated as above] is the same, to two decimal places). At the same time, each nuclear locus, for which the coalescence-time scale is four times slower, would have lost all Neanderthal alleles with probability .10 and would have become fixed for them with probability 9.8*10^-5. Thus, 90% would still be segregating for Neanderthal alleles. In conclusion, data such as those shown in figure 1 shed little light on the issue of replacement versus interbreeding, unless the number of ancestors of the sample was large throughout the periods of interest. This is part of a general problem: in order to estimate gene flow, a large sample is needed, and, in order to estimate ancient-gene flow, a large ancient sample is needed. According to coalescent theory, large ancient samples usually cannot be obtained by the sampling of modern populations. The rate of coalescence is quadratic in the number of ancestors and linear in the inverse of the population size. Thus, the expected number of ancestors of a sample usually decreases rapidly as earlier time periods are studied. Exceptions include exponentially growing populations, in which the number of ancestors may be large shortly after the onset of growth (reviewed in Donnelly and Tavare´ 1995; Marjoram and Donnelly 1997). In the present case, it seems clear that the statistical power to detect interbreeding that took place before the human population started to grow exponentially is close to zero.

In fact, I am skeptical that modern humans were not experiencing exponential population growth around the time of Neanderthal admixture and personally believe that a more restrictive admixture model is what kept Neanderthal mtDNA out of the modern human gene pool.

A Social Explanation For The Lack Of Neanderthal mtDNA

Specifically, I think that almost all Neanderthal-modern human admixture involve short term sexual encounters (which may or may not have been consensual) in which the female in the couple remained a part of a tribe belonging primarily to her own species, rather than in long term marital relationship or merged tribes.

Thus, Neanderthal hybrids born into modern human tribes almost all had modern human single mothers, and Neanderthal hybrids born in Neanderthal tribes almost all had Neanderthal single mothers.

Of course, neither model needs to be 100% true. 

The percentage of autosomal Neanderthal DNA in modern humans suggests that the Neanderthal fraction of the autosomal DNA of the initial modern population was probably 12% or less, rather than the 25% used to determine that there was a chance of Neanderthal mtDNA due to drift during period of more or less constant population.

Suppose also that two-thirds of Neanderthal-modern human hybrids fit the model I have suggested, while one-third were born to Neanderthal-modern human couples integrated into the mostly modern human tribe as modeled in the article, and that the long term couples collectively equally split between Neanderthal women and modern human men, and Neanderthal men and modern human women, then the effective percentage of Neanderthal contribution to the mtDNA gene pool would be not 25% but 2%, resulting in a much greater probability of losing Neanderthal mtDNA to random drift.

The Absence of Neanderthal Y-DNA in Modern Humans and Haldane's Rule

Needless to say, this model does not by itself resolve the absence of Neanderthal Y-DNA in modern humans, although its probability of loss to random drift would be on the order of the 52% estimated if the population was constant and Neanderthals contributed 25% to the total ancestry and there was a constant population.

But, here Haldane's rule comes into play and while the question of whether Neanderthals and modern humans were sufficiently genetically differentiated for Haldane's rule to apply was widely debated when this was first proposed, careful analysis of Neanderthal v. modern human Y-DNA and the pattern of autosomal Neanderthal DNA distribution across the human genome now strongly corroborates with direct evidence the hypothesis that Haldane's rule did indeed cause a disproportionate share of Neanderthal male-modern human female parent hybrids to be female and hence to lack Neanderthal Y-DNA.

Again, Haldane's rule doesn't have to operate with perfect efficiency to achieve the observed result with a high probability. 

If 80% of hybrid children were girls and 12% or less of the total ancestry is Neanderthal, then the probability of Y-DNA drift is equivalent to that of a randomly mating community with 4.8% Neanderthal ancestry, which makes the likelihood of loss of Neanderthal Y-DNA due to drift much, much higher than the 52% in a constant population model that you would expect in a community with 25% Neanderthal ancestry. If 90% of the hybrid children at girls, this falls to 2.4%, and both of those estimates are before considering the possibility that hybrid Neanderthal boys in a mostly modern human tribe may have had more of a selective fitness deficit than hybrid Neanderthal girls in a modern human tribe, which also seems quite plausible.

One can have very plausible parameters and end up with a very high probability of the loss of Neanderthal Y-DNA to random drift is some combination of a fairly effective Haldane's law, and greater selective fitness for hybrid girls relative to hybrid boys in modern human tribes for hybrids with a high percentage of Neanderthal admixture, are true.

In Nature, where Haldane's rule does apply, the preference for girls (in species with similar gender coding to primates) is much higher than 90%, so the estimates above are conservative. For example, male mules (i.e. horse and donkey hybrids) are "one in a million". A wide variety of studies across many species examined in a meta-analysis have demonstrated that the gender disfavored by Haldane's rule either doesn't happen at all, or produces sterile offspring, rather than fertile hybrid males being just significantly less likely than females (in primate-like sex determination schemes). For example, in a case of two species of primates in the same genus in the wild, Haldane's rule was found to apply to hybrid individuals:

Aguiar et al. (2008) studied species identity in groups of the howler monkeys Alouatta caraya and A. clamitans within their area of sympatry. They found that adult individuals of hybrid origin (on the basis of their mosaic coloration pattern) were almost always females. This led them to conclude that this (male-heterogametic) species adheres to HR.

There is not a consensus regarding the exact mechanism by which Haldane's rule operates, but it does tend to be an all or nothing affair except in extremely rare cases.