Neanderthals And Modern Humans Experienced Hybrid Incompatability

Also know as Haldane's law, heterozygotes for sex determination genes are strongly disfavored as children of cross-species hybrids. And, the evidence is strong that this hybrid incompatibility prevented Neanderthal-human hybrids from being born. This is strong evidence that Neanderthals and modern humans were biologically separate species.

As the author explains in an open access summary:

After years of sequencing the genomes of female Neandertals, researchers have finally got their first good look at the Y chromosome of a male Neandertal—and found that it is unlike that of any other Y in modern humans living today. Even though Neandertals and modern humans interbred several times in the past 100,000 years, the DNA on the Y chromosome from a male Neandertal who lived at El Sidrón, Spain, 49,000 years ago has not been passed onto modern humans, researchers report today in The American Journal of Human Genetics. The finding fits with earlier studies that have found that although living Asians and Europeans have inherited 1% to 3% of their DNA from their ancestors’ interbreeding with Neandertals, they are missing chunks of Neandertal DNA on their Y chromosomes. This has suggested that female modern humans and male Neandertals were not fully compatible and that male Neandertals may have had problems with sperm production. The new study finds a clue to why: The El Sidrón Neandertal had mutations in three immune genes, including one that produces antigens that can elicit an immune response in pregnant women, causing them to reject and miscarry male fetuses with those genes. So even though male Neandertals and female modern humans probably hooked up more than once over the ages, they may have been unable to produce many healthy male babies (such as the reconstruction of this Neandertal boy from fossils from Gibraltar)—and, thus, hastened the extinction of Neandertals.

There are also no Neanderthal mtDNA sequences in modern humans. This is probably because the children of hybrid couples were probably raised in the tribe of the mother. Hybrid children with modern human mothers were raised in modern human tribes and left descendants who are alive today. Hybrid children with Neanderthal mothers were raised in Neanderthal tribes that ultimately went extinct. 

This inferred pattern of matrilocality for hybrid children, despite the fact that Neanderthals were predominantly patrilocal, also favors the possibility that hybrid children were largely a result of rape or brief encounters, rather than than marriage-like couple relationships. There is not a single example archaeologically of a hominin tribe containing both full blooded Neanderthal and full blooded modern human members.