One idea given for the mysterious disappearance of Neanderthals around 40,000 years ago is a difference in brain power compared with early modern humans (Homo sapiens), who invaded their territory in Eurasia and outcompeted them. But a new study of brain variation reveals that Neanderthals and humans were much more alike than previously assumed.

Neanderthal skulls are quite obviously different in shape from those of early modern humans. While Neanderthals had longer, lower skulls with heavier brow ridges and larger nasal openings, modern humans have more globular skulls with smaller facial features. The inside of the skull, called the endocranium, also looks different in Neanderthals versus modern humans.

Because of the anatomical differences in the skulls, many experts have assumed that Neanderthals couldn't talk as well as humans, had poor planning skills, and had limited short-term memory. But, problematically, these differences "were not put into the context of modern human populational variation in brain anatomy, which is known to be substantial," the researchers wrote.

To better understand variation in brain anatomy, the researchers compared two large MRI datasets of the brains of living people: 100 ethnic Han Chinese and 100 Americans with European ancestry. In nearly 70% of the brain regions the researchers assessed, they found that the volume differences between the group of Chinese brains and American brains were larger than those previously found between Neanderthals and early modern humans. These results suggest a very large degree of overlap in brain anatomy and, therefore, cognitive abilities between Neanderthals and humans, the researchers wrote.

"If modern human population differences are not evolutionarily significant (which is what is generally assumed), then any similarly sized Neanderthal/early anatomically modern Homo sapiens brain differences also should not be considered evolutionarily significant," they wrote.

The researchers noted that even small differences in behavior and brain size can potentially have significant evolutionary consequences. One of the largest differences between Neanderthal and H. sapiens brains was found to be correlated with attention and inhibition, suggesting Neanderthals may have had a slightly lower executive functioning ability. But given how quickly Neanderthals disappeared after modern humans arrived, "it is not clear that such small differences could actually have meaningfully contributed to their replacement," the researchers wrote.

Given that human brains differ between living populations today to a greater degree than Neanderthal and early modern human brains differed some 40,000 years ago, the researchers don't think Neanderthals died out because they lacked the intelligence to adapt.

The new study "strongly points to demography and genetic swamping — possibly as a result of some kinds of cultural differences — and not to innate differences in cognitive ability as the most likely cause of Neanderthal replacement," the researchers wrote. This idea — in which the genes of a minority species are overwhelmed by those of the majority species — echoes recent research that has modeled the integration of H. sapiens into Neanderthal populations, which could have led to the latter's disappearance within as little as 10,000 years.

More research along these lines may be warranted, the researchers concluded, since they only compared the brains of Chinese and American people.

"It is entirely possible that even larger differences exist among modern human populations, further calling into question the evolutionary significance of the estimated Neanderthal differences in brain anatomy with contemporary anatomically modern H. sapiens," they wrote.

How much do you know about our closest relatives? Find out with our Neanderthal quiz!

Kristina Killgrove is a staff writer at Live Science with a focus on archaeology and paleoanthropology news. Her articles have also appeared in venues such as Forbes, Smithsonian, and Mental Floss. Kristina holds a Ph.D. in biological anthropology and an M.A. in classical archaeology from the University of North Carolina, as well as a B.A. in Latin from the University of Virginia, and she was formerly a university professor and researcher. She has received awards from the Society for American Archaeology and the American Anthropological Association for her science writing.