Was There An Almost Failed First Modern Human Out Of Africa Wave?

Pagani (2016) makes the case based upon the most recent common ancestry date determined by comparing parts of Papuan autosomal DNA, compared to the TMRCA of that DNA in other modern humans, that the ancestors of the Papuans admixed with humans from an earlier wave of modern human migrants to Asia ca. 100,000 years ago. This is from a population which is also one of the few to show signs of admixture with Denisovans, a form of archaic hominin that diverged from modern humans before the oldest genetic or archaeological evidence of the existence of anatomically modern humans.

This new data point could be the solution to a potentially vexing paradox. 

There has long been archaeological evidence of a modern human presence in places like the Levant from 100,000 to 75,000 years ago. But, more recently, archaeological evidence of a modern human presence has been found in the Arabian interior from 100,000 to 125,000 years ago, in South Asia from more than 75,000 years ago, and arguably even China from 100,000 to 125,000 years ago. 

But, analysis of modern human DNA, and efforts to date Neanderthal admixture with modern humans including efforts based on non-mutation rate methods using ancient DNA, put a common ancestor of all non-Africans at more like 50,000 to 65,000 years ago, which corresponds to archaeological evidence of the first modern human presence in Australia and Papua New Guinea (which were a single land mass at the time).

There is then a gap in archaeological record in the Levant from around 75,000 years before present to about 50,000 years before present, so until very recently, at least, the mainstream explanation for the earlier archaeological evidence in the Levant, used to be that the early Levantine archaeological remains were an "Out of Africa that failed" and that all modern non-Africans descend from a second Out of Africa that prospered wave. 

But, the increasingly widespread archaeological evidence for a modern human presence in this 25,000 year gap period, genetic evidence in Altai Neanderthal ancient DNA indicating an admixture with modern humans ca. 100,000 years ago, and now this new data point in Pagani (2016), suggest that the simple version of the "Out of Africa that failed" theory are wrong.

Pagani (2016) instead suggests that there was a first wave of pre-Upper Paleolithic humans that spread across parts of Eurasia who admixed with Neanderthals and didn't make much of an ecological difference (although arguably, they could have led to the extinction of Homo Erectus in Asia). This first wave of modern humans outside Africa provided only a small part of the ancestry of a small subset of modern human. But, tens of thousands of years later, the remnants of these first wave modern humans did admix with the second wave of more successful Upper Paleolithic modern humans who ended up in Papua New Guinea, a wave whose expansion into Eurasia was permanent and successful, even thought they were mostly replaced by these second wave modern humans.

High-coverage whole-genome sequence studies have so far focused on a limited number of geographically restricted populations, or been targeted at specific diseases, such as cancer. Nevertheless, the availability of high-resolution genomic data has led to the development of new methodologies for inferring population history and refuelled the debate on the mutation rate in humans. Here we present the Estonian Biocentre Human Genome Diversity Panel (EGDP), a dataset of 483 high-coverage human genomes from 148 populations worldwide, including 379 new genomes from 125 populations, which we group into diversity and selection sets. We analyse this dataset to refine estimates of continent-wide patterns of heterozygosity, long- and short-distance gene flow, archaicadmixture, and changes in effective population size through time as well as for signals of positive or balancing selection. We find a genetic signature in present-day Papuans that suggests that at least 2% of their genome originates from an early and largely extinct expansion of anatomically modern humans (AMHs) out of Africa. Together with evidence from the western Asian fossil record, and admixture between AMHs and Neanderthals predating the main Eurasian expansion, our results contribute to the mounting evidence for the presence of AMHs out of Africa earlier than 75,000 years ago. 

Pagani, et al., "Genomic analyses inform on migration events during the peopling of Eurasia", Nature (Published online 21 September 2016). Hat tip: Marnie at Linear Population Model.

The paper is not open access, but Marnie at Linear Population Model provides quotes from some key passages of the paper and follows up with full citations included abstracts of some of the key sources cited therein.

For example, Marnie provides the citation and abstract for the Altai Neanderthal paper (which I am reprinting in a reformatted manner with emphasis added):

It has been shown that Neanderthals contributed genetically to modern humans outside Africa 47,000–65,000 years ago. Here we analyse the genomes of a Neanderthal and a Denisovan from the Altai Mountains in Siberia together with the sequences of chromosome 21 of two Neanderthals from Spain and Croatia. We find that a population that diverged early from other modern humans in Africa contributed genetically to the ancestors of Neanderthals from the Altai Mountains roughly 100,000 years ago. By contrast, we do not detect such a genetic contribution in the Denisovan or the two European Neanderthals. We conclude that in addition to later interbreeding events, the ancestors of Neanderthals from the Altai Mountains and early modern humans met and interbred, possibly in the Near East, many thousands of years earlier than previously thought.

Martin Kuhlwilm, et al., "Ancient gene flow from early modern humans into Eastern Neanderthals", Nature, Volume 530, Pages 429-433 (25 February 2016).  

Marnie also provides some of the language clarifying the key original insights of Pagani (2016) (citations and internal cross references omitted without editorial indication, emphasis mine):

Using fineSTRUCTURE, we find in the genomes of Papuans and Philippine Negritos more short haplotypes assigned as African than seen in genomes for individuals from other non-African populations. This pattern remains after correcting for potential confounders such as phasing errors and sampling bias. These shorter shared haplotypes would be consistent with an older population split. Indeed, the Papuan–Yoruban median genetic split time (using multiple sequential Markovian coalescent (MSMC)) of 90 kya predates the split of all mainland Eurasian populations from Yorubans at ~75 kya. This result is robust to phasing artefacts. Furthermore, the Papuan–Eurasian MSMC split time of ~40 kya is only slightly older than splits between west Eurasian and East Asian populations dated at ~30 kya. The Papuan split times from Yoruba and Eurasia are therefore incompatible with a simple bifurcating population tree model. 

At least two main models could explain our estimates of older divergence dates for Sahul populations from Africa than mainland Eurasians in our sample: 1) admixture in Sahul with a potentially un-sampled archaic human population that split from modern humans either before or at the same time as did Denisova and Neanderthal; or 2) admixture in Sahul with a modern human population (extinct OoA line; xOoA) that left Africa after the split between modern humans Africa after the split between modern humans.

We consider support for these two non-mutually exclusive scenarios. Because the introgressing lineage has not been observed with aDNA, standard methods are limited in their ability to distinguish between these hypotheses. Furthermore, we show that single-site statistics, such as Patterson’s D, and sharing of non-African Alleles (nAAs), are inherently affected by confounding effects owing to archaic introgression in non-African populations. Our approach therefore relies on multiple lines of evidence using haplotype-based MSMC and fineSTRUCTURE comparisons (which we show should have power at this timescale). 

We located and masked putatively introgressed Denisova haplotypes from the genomes of Papuans, and evaluated phasing errors by symmetrically phasing Papuans and Eurasians genomes. Neither modification changed the estimated split time (based on MSMC) between Africans and Papuans. MSMC dates behave approximately linearly under admixture, implying that the hypothesized lineage may have split from most Africans around 120 kya. 

We compared the effect on the MSMC split times of an xOoA or a Denisova lineage in Papuans by extensive coalescent simulations. We could not simulate the large Papuan–African and Papuan–Eurasian split times inferred from the data, unless assuming an implausibly large contribution from a Denisova-like population. Furthermore, while the observed shift in the African–Papuan MSMC split curve can be qualitatively reproduced when including a 4% genomic component that diverged 120 kya from the main human lineage within Papuans, a similar quantity of Denisova admixture does not produce any significant effect. This favours a small presence of xOoA lineages rather than Denisova admixture alone as the likely cause of the observed deep African–Papuan split. We also show that such a scenario is compatible with the observed mitochondrial DNA and Y chromosome lineages in Oceania, as also previously argued. 

We further tested our hypothesized xOoA model by analysing haplotypes in the genomes of Papuans that show African ancestry not found in other Eurasian populations. We re-ran fineSTRUCTURE adding the Denisova, Altai Neanderthal and the Human Ancestral Genome sequences to a subset of the diversity set. FineSTRUCTURE infers haplotypes that have a most recent common ancestor (MRCA) with another individual. Papuan haplotypes assigned as African had, regardless, an elevated level of non-African derived alleles (that is, nAAs fixed ancestral in Africans) compared to such haplotypes in Eurasians. They therefore have an older mean coalescence time with our African samples.

I find no fault in the analysis in Pagani (2016) which is thoughtfully done by a method that should be reliable.