Most researchers accept that before the arrival of anatomically modern humans, Neanderthals had adopted several transitional technocomplexes. Two of these, the Uluzzian of southern Europe and the Châtelperronian of western Europe, are key to current interpretations regarding the timing of arrival of anatomically modern humans in the region and their potential interaction with Neanderthal populations. They are also central to current debates regarding the cognitive abilities of Neanderthals and the reasons behind their extinction.
However, the actual fossil evidence associated with these assemblages is scant and fragmentary, and recent work has questioned the attribution of the Châtelperronian to Neanderthals on the basis of taphonomic mixing and lithic analysis.
Here we reanalyse the deciduous molars from the Grotta del Cavallo (southern Italy), associated with the Uluzzian and originally classified as Neanderthal. Using two independent morphometric methods based on microtomographic data, we show that the Cavallo specimens can be attributed to anatomically modern humans. The secure context of the teeth provides crucial evidence that the makers of the Uluzzian technocomplex were therefore not Neanderthals.
Maju discussed some of the earlier literature referred to in the abstract excerpt quoted above (such as this 2005 article by Mellars) at his old blog, and discusses this article at his new blog.
Mellars argues in particular that transitional European industry stone tools look very similar to Middle Stone Age tools unambiguously associated with modern humans from Africa (from tens of thousands of years earlier), and that it was unambiguously clear that the transitional industries did not appear in Europe until modern humans arrived on the scene (as deduced from carbon dating of nearby European sites definitively associated with modern humans). Thus, it would be an "impossible coincidence" if modern humans didn't have some connection to the transitional industries which were themselves only associated by fairly thin evidence with Neanderthals.
Some of the older literature comes from Denver's resident expert on Neanderthals, Julien Riel-Salvatore, who cast doubt on the Neanderthal association with the Uluzzian with multiple lines of strong circumstantial evidence as opposed to the direct evidence of the most recent paper. Riel-Salvatore also noted in his 2010 paper the doubtfulness of prior associations of the teeth found with Neanderthals, which today's research bears out.
Riel-Salvatore also discussed at his blog the case against Neanderthal involvement in the Châtelperronian industry. The association of Neanderthals with the Châtelperronian is based on just two digs and the association of one of the two sites was seriously questioned by by Bar-Yosef, O., & Bordes, J.G. in their article "Who were the makers of the Châtelperronian culture?" Journal of Human Evolution (2010) DOI: 10.1016/j.jhevol.2010.06.009, on the theory that there is strong evidence of muddling of the stratiography between an older clearly Neanderthal layers and the Châtelperronian that disfavors that associatioon.
Bar-Yosef and Bordes were less trenchant in their examination of the second site, however. He notes in that post that, "Ultimately, though, the revision proposed by Bar-Yosef and Bordes needs to be taken cautiously, especially as it concerns the St. Cesaire remains. One thing it doesn't do is associate the Chatelperronian with modern humans," as the latest research does for the Uluzzian. More particularly, he explains that:
At St. Césaire, where the the secondary burial of a Neanderhal was found in the Chatelperronian, Bar-Yosef and Bordes argue that the presence of distinct 'Mousterian' and 'Chatelperronian' components of the lithic industry and the fact that not all artifacts show a similar state of preservation suggest caution is needed before we can accept that it was not a mix of Mousterian and Chatelperronian levels. Because the burial is found in the upper (of two) Chatelperronian level, they argue that it was probably not deposited by occupants of the site anyway, implying that Neanderthals who did not make Chatelperronian tools might have buried one of their deceased at St. Césaire, perhaps in an effort to mark the site as theirs following the arrival of whoever made the Chatelperronian.
In my view, Bar-Yosef and Bordes' case is much stronger for Grotte du Renne than it is for St. Césaire, especially since bone refitting at St. Césaire has recently demonstrated that mixing between the Mousterian and the Chatelperronain was a negligible occurrence at the site (Morin et al. 2005). The need to invoke an explanation that doesn't depend strictly on stratigraphic of artifactual data also weakens their overall argument. I say this because if their argument is followed to its logical end, it would mean that, even if you found that unheard of goody that would be a site containing only Chatelperronian layers and thus could exclude stratigrpahic mixing as an explanation, the presence of Neanderthal remains (or at least of a secondary Neanderthal burial) could still be explained away as a result of self-affirming Neanderthals claiming a stake to a given bit of territory. That's not to say that it's impossible, of course, and I will concede that the St. Césaire burial is the only relatively undisputed case of a Neanderthal secondary burial, so the practice was never common and who knows what it really might have meant, as the authors concede. Given that this is the case, though, you wouldn't expect a very unusual behavior to be the preferred manner in which Neanderthals would have marked their territories in the context of encounters with new groups of hominins.
The debate now, is pretty much limited to whether the Châtelperronian was actually associated entirely with modern humans (a possibility discounted entirely until a couple of years ago), or simply involved efforts by Neanderthals (or hybrid Neanderthals perhaps) to copy contemporaneous modern human industries.
Alternately, a hybrid Neanderthal at the modern human-Neanderthal boundary might be a natural candidate for an unusual secondary burial at a possibly abandoned Châtelperronian site (don't forget that all modern humans were semi-nomadic hunter-gatherers at the time), if that industry was a modern human one, that would be better motivated than a simple effort of the Neanderthals to mark their territory. The secondarily buried Neanderthal at that site might, for example, have had a modern human father who lived there when the buried individual was conceived (or the mother of a hybrid child from such a union). We know that some Neanderthals did bury their dead and that there was some symbolic element to this practice (as indicated, for example, by grave goods).
If no Neanderthals, or at least no non-hybrid Neanderthals, engaged in transitional industries, and they instead were unable to innovate beyond the 200,000 year old Mousterian stone industry even in the face of exposure to more advanced modern human tool making, the natural inference is that they really were significantly less intelligent than modern humans. In particular, they may have either lacked the ability to learn and innovate, instead acting more on instinct, or may have lacked the mental capacity to make more sophisticated tools in general because they involved some sort of evolutionary leap in ability (perhaps in mental ability, or perhaps in fine motor skills) that they couldn't handle.