The jury’s still out on AI’s effectiveness as a learning tool, but research so far paints a grim picture. Using AI chatbots can impair critical thinking, result in lower brain activity during cognitive tasks, and has been linked to memory loss.

There was one prominent study, however, that provided a glimmer of hope for AI advocates. Published in the journal Nature, it purported to show that using AI like OpenAI’s ChatGPT can have a “large positive impact on improving learning performance” and a “moderately positive impact on enhancing learning perception and fostering higher-order thinking.”

The takeaway to the authors was clear: “ChatGPT should be actively integrated into different learning modes to enhance student learning, especially in problem-based learning,” they enthused.

But nearly a year after the study was first published, it’s now been unceremoniously retracted. Springer Nature, the journal’s publisher, cited “concerns regarding discrepancies” for why it pulled the paper, in a retraction note published late last month, which “ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions.”

Needless to say, it’s a blow to advocates of AI in education.

“The paper’s authors made some very attention-grabbing claims about the benefits of ChatGPT on learning outcomes,” Ben Williamson, a senior lecturer at the Centre for Research in Digital Education and the Edinburgh Futures Institute at the University of Edinburgh in Scotland, told Ars Technica. “It was treated by many on social media as one of the first pieces of hard, gold standard evidence that ChatGPT, and generative AI more broadly, benefits learners.”

The paper was not an experimental study, but a meta analysis that synthesized the findings of 51 existing studies on the subject, comparing the cognitive effects of participants who used ChatGPT and those that didn’t. This, as many commentators and experts have noted, already put the study on shaky ground, since ChatGPT was still a novel phenomenon and scientists were only just beginning to research its cognitive effects.

“It is not feasible that dozens of high-quality studies about ChatGPT and learning performance could have been conducted, reviewed, and published in that time,” Williamson told Ars.

“In some cases it appears it was synthesizing very poor quality studies, or mixing together findings from studies that simply cannot be accurately compared due to very different methods, populations, and samples,” he added. “It really seemed like a paper that should not have been published in the first place.”

The retraction comes as the AI industry continues to make a heavy push into the classroom. Companies like OpenAI have partnered with colleges and schools to provide students with free access to their AI tools, and even versions of their chatbots that are tailor-made for specific schools. OpenAI, Anthropic, and Microsoft have poured millions into teachers unions to train them how to use AI models. Ohio State University now requires all students in every major to enroll in an “AI fluency” course. 

That’s despite widespread complaints from teachers about AI being used for rampant cheating, and pushback from parents who are concerned about letting their children be unwitting test subjects in a large scale AI experiment.

Williamson called the retracted study “hugely frustrating for those of us trying hard to make sense of what AI means for learning, teaching, and education more generally.”

“We have had several years of hype about AI in education, but what we have really needed is high-quality research that can actually show us what kinds of impacts AI is having in classrooms and learning practices,” he told Ars.

This isn’t Springer Nature’s first AI-related controversy. Last year, it began approaching study authors and offering to sell them AI-generated “Media Kits” that that summarized their papers. Like many academic journals, it’s also struggled to weed out shoddy AI-generated content.

More on AI: Usually, Young People Embrace New Technology. Gen Z’s Attitude Toward AI Should Worry the Entire Tech Industry