In principle, Y-DNA lineages can expand much more rapidly than mtDNA lineages. The maximum number of children that a group of women with identical mtDNA can have in a lifetime on average is much lower than the maximum number of children that a group of men with identical Y-DNA can have in a lifetime.
But, how often does this happen and does the historic anthropology support the suggestion from the genetic evidence.
Commentators like Razib Khan (who has migrated Gene Expressions to a new website as of yesterday) speak of androcide - the near complete exclusion of substrate population men from having male children who go on to reproduce themselves as at the heart of this process, while substrate population women are more often appropriate by men of incoming superstrate populations via polygyny of the equivalent (multiple women per man).
Both history and ethnography document mass population collapse in concert with an androcide of the Amerindians. By this, I mean that European males took Amerindian women as concubines, and engaged in de facto polygyny in the New World.Polygyny remains common even today in much of Africa.
The evidence is sketchier in prehistoric populations like the Bell Beaker and Corded Ware peoples whose civilizations were superseded by later populations before it was possible for those civilizations to be documented in writing.
The alternative possibility that there was merely segregation of people into a rapidly expanding largely monogamous superstrate population and a stagnant largely monogamous substrate population, with some level of bride exchange between the two populations, is also possible. In that scenario, bride exchange brings substrate women into the rapidly expanding population mix, while bride exchange in the other direction is hard to discern from genetic evidence in modern populations since it is hard to distinguish from superstrate women who remained in their home population and because exchanged brides would make up a small part of a stagnant population whose descendants are becoming an increasingly small proportion of the total population.
If the growth rate for the expanding population is much greater than for the segregated stagnant population, it is hard to distinguish a polygyny and androcide scenario from a scenario with monogamy and moderate levels of equitable bride exchange from each other on the basis of genetic evidence alone.
It ought, in principle, to be easier to distinguish a scenario in which migration and conquest was overwhelmingly by men (e.g. an invading army) from one that is a mixed gender migration - as there would be no non-indigenous mtDNA lines introduced in great frequencies while there was a great change involving a newly introduced group of Y-DNA lineages. A male dominated conquering population could support a scenario of androcide (often literally with indigeneous men of reproductive age killed in war and their widows appropriated by the conquerors) without significant polygyny.
There is evidence of polygny and the equivalent in European cultures at the dawn of the historic era like the Celts (alluded to in several comments here and also inferred here and here). This may have also been present in early Bell Beaker and Indo-European populations, and on the to do list is an effort to track down better authority on the extent to which it is likely that there was much polygyny in those prehistoric or ancient cultures. There is also legendary history evidence of androcide in wars of conquest.
But, it is hard to tell. Polygny leaves a far fainter trail of artifacts in archaeological digs than technologies like metal use or food production technologies.
A cursory search reveals that polygyny was practiced, at least among elites, in almost all of the cultures known at the dawn of the historic era in Eurasia, although sometimes secondary wives or concubines were permitted only if they held slave status. The Mycenean Greeks and Trojans, the ancient Macedonians, the Hittites, the Celts, early Nordic pagans, the first century Merovingians and Carolingians, the early Jews, the Roman, the pre-Christian Slavs, Muslims and pre-Muslim pagans, the pre-Communist Chinese, the early Aryans (a controversial point to some), the pre-Zoroastrian religion of Iran, and the elites of ancient Egypt. There is genetic evidence of prior polygamy in French Basques and many other modern populations. Also, monogamous societies are a decided minority among anthropologically attested cultures (only about one in six fit that description). The status of Avesta era Iran is unclear and may have been monogamous and polygamy was expressly forbidden by the legal code of the Visigoths as of the 1st century according to Tactius (although not by any other the other Germanic tribes of that time). There is evidence of religious fertility rights involving annual divine marriages of Sumerian kings to temple priestesses in addition to their ordinary wives, but evidence regarding polygamy per se in ancient Sumeria is sketchy.
Still, as a Bayesian prior, we would strongly expect Chalcolithic and Bronze Age cultures of Europe to have been polygynous rather than monogamous. Monogamy is a practice that seems to have been exceedingly rare as a universal rule of any culture prior to the Iron Age. An academic case has been made for Proto-Indo-European monogamy, but I personally find it implausible given the prevalence of polygyny in the earliest attested Indo-European cultures.
However, if polygny was a practice largely restricted to kings and chiefs, rather than being widespread, the number of wives was not too immense, and the scale of chiefdoms was fairly large in this time period, the cumulative impact of this practice on the gene pool may have been fairly modest and this may not have been an important factor in explaining the expansion of Y-DNA lineages. The considerable diversity of polygynous practices makes it hard to model.
Polygny isn't absolutely necessary to estimates of Y-DNA R1b expansion rates in Europe which, while high, were not so high that they couldn't be consistent with a sustained period of largely monogamous expansion by a population without the male selective androcide and without the polygyny that necessarily goes hand in hand with that pattern.
The expansion of mtDNA H in Europe apparently coincident to the R1b expansion (in both cases from a source in Iberia whatever prior sources those expansion had, perhaps the Bell Beaker population*) did not bring mtDNA H to quite as dominant a position a Y-DNA R1b in Western Europe, but without knowing what the mtDNA breakdown of the source population for the R1b expansion in Western Europe was and having detailed mtDNA haplogroup subtype data for non-mtDNA H haplogroups in Western Europe (both available to some extent but beyond the scope of this post), it is hard to know how much of the Y-DNA R1b expansion involved displacement of indigenous men and how much involved expansion of multiple mtDNA lineages in the source population complemented by bride exchange between growing and stagnant communities on an equitable basis.
The 2008 analysis by Hammer et al. of sex differences in breeding ratio in six populations including the French Basque is most on point (internal references to figures and other papers of the paper omitted without acknowledgement of the omissions):
Our method uses a population genetic model (i.e., the coalescent) to account for the inherent uncertainty in estimating diversity and divergence rates from sequence data. For three out of six populations (Basque, Melanesians and Mandenka), the 95% confidence intervals for the ratio of X-linked and autosomal effective population sizes does not include 0.75 [Ed. the expected value in a monogamous population model.] (p = 0.001, 0.005 and 0.030 for the Basque, Melanesians and Mandenka, respectively). One interpretation of these results is that there is strong evidence for an unequal female and male Ne in at least three of our six populations, with estimates of the breeding sex ratio (i.e., the effective size of females to males) ranging from 2.1 in the San to 12.5 in the Basque. If the observed differences in nucleotide variability on the X chromosome and autosomes are caused by long-term (demographic) processes, then the estimates of Nx/Na presented will be highly correlated due to shared population history. When we use the intersection of all six confidence intervals (0.87–1.02) to estimate the range of Nx/Na values that are consistent with the data from all six populations, we estimate the range of the breeding sex ratio to be 2.4–8.7. We also note that even with a conservative Bonferroni correction, a 1:1 breeding sex ratio is rejected in two out of six populations.Naively, this suggests that the ancestors of the French Basque were, at some point, massively polygnyous. There is an alternative explanation, however.
We also employ a separate method for estimating the breeding sex ratio in each population that does not allow for intra-locus recombination but does permit independent mutation rates across loci. This method produces similar results to those described above, with estimates of the ratio of female to male effective population size ranging from 1.8 in the San to 14.0 in the Basque. We interpret this as additional evidence that the unusual patterns observed in our data are real and require explanation.
This is that while the pairings in some ancestral era of the French Basque were mostly of French Basque men and indigeneous women, that rather than being massively polygamous, the French Basque men were simply highly inbred and closely related. In other words, so many of the men were genetically identical or nearly so that they look in effective population size genetic statistics, like a single man.
This could happen, for example, if a Basque founding population were composed almost entirely of a group of migrating men who were all part of a patrilocal extended family group that had expanded fairly rapidly elsewhere in their insular community of origin, prior to migrating together to Iberia, while the indigeneous women of the pre-existing megalithic culture were highly diverse due to many generations of fairly long distance bride exchanges prior to the Basque male founding population arrived.
For example, suppose the early copper working in Central Europe or West Asia were organized on a patrilocal clan basis, so that only sons of existing (and prosperous and expanding) copper workers could become copper workers. Then, suppose that word of rich unexploited copper resources in Southern Iberia reached them. Large numbers of these men from the copper worker extended family might leave, en masse, in a Gold rush style, male dominated copper rush to Iberia that in this case proved to be not a myth, but a reality that led to multiple generations of migration to Iberia from the same patrilocal clan as those who arrived there prospered and married local women.
While the founding Bell Beaker population which I claim is proto-Basque probably was polygamous to some extent like other pre-Iron Age cultures, it seems highly unlikely to me that Basque men had, on average, for several hundred years, a dozen or so wives each.
After all, while the growth rate of the Western European specific clades of Y-DNA are phenomenal, they are merely at the very high end of what could be attained with a very fertile monogamous scenario. Mildly polygamous family patterns (e.g. 1.5 or so wives per Basque man on average) fit the observed rates of R1b expansion much better than massively polygamous family patterns in which each man could conceivably have had as many as a dozen or more wives on average for ten or twelve generations in a row.
The patrilocal clan originating founding population scenario that I suggest seems like a more likely source for most of the disparity in effective male and female population sizes that is evident in French Basque DNA.
* The old conventional wisdom was that the Bell Beaker people had predominantly a cultural impact on Western Europe and was largely a trading civilization with only modest demographic impact, while the subsequent Indo-Europeans may have had a major demographic impact. Maju (who is blogging again after taking a hiatus for a while) has argued this case multiple times. Ancient DNA evidence and modern population genetic studies seem to be favoring the reverse scenario - one with a major Bell Beaker demographic impact and a fairly shallow additional Indo-European wave demographic impact - although the ancient DNA data are still too fragmentary and inconclusive to definitively favor either scenario or some third scenario. There is also some debate over the question of whether the Indo-Europeans may have themselves been Indo-European linguistically and culturally, a position that I consider a minority and disfavored view. In my view, the Bell Beaker people were probably Vasconic (i.e. proto-Basque), and were themselves probably a second rather than an initial wave of farming people to expand in Western Europe.