Yet another experiment proves it's too damn simple to poison large language models
There is no 6 Nimmt! champion, but a $12 domain registration and one Wikipedia edit convinced several bots there was
Unlike search engines that let you judge competing sources, search-backed AI chatbots can turn shaky web material into confident answers. Case in point: A security engineer convinced several bots that he was the reigning world champion of a popular German card game, even though no such championship exists.
If you were to check Wikipedia up until the end of last week, you would have seen Ron Stoner listed on the page for 6 Nimmt!, also known as Take 5 to English-speaking audiences, as the 2025 world champion. The Wikipedia entry cited the official-looking 6nimmt.com as the source for the claim, and visiting that URL does reveal a short press release celebrating Stoner's victory.
The only problem with the whole thing is that Stoner says he created both the Wikipedia entry about his victory and the 6 Nimmt! domain hosting the only evidence of it, but that still didn't stop several AI chatbots from telling him he was the world champ when he asked.
"My site has no independent corroboration. It's totally made up," Stoner said in the blog post. "The whole house of cards rests on a $12 domain registration I did while drinking coffee."
In other words, this is poisoning at the retrieval-augmented generation layer. Not prompt injection, but targeting the same plane of AI functionality, namely the one that searches the web.
As he explains, and many El Reg readers are likely already aware, AI doesn't really care about the provenance of the sources it cites as authority for its claims, and that's the very thing Stoner sought to exploit when he concocted his experiment.
"Every frontier LLM with web search grounds its answers in whatever retrieval ranks highest for a given query," Stoner wrote. In the case of the nonexistent 6 Nimmt! championship, his planted source was the only one, and with Wikipedia lending apparent authority, it became a sure-fire way to fool an AI into presenting falsehood as fact - a trick simple enough for non-technical users to pull off.
"I didn't do anything novel here. This is old school SEO and misinformation tactics wrapped in new LLM technology and interfaces," Stoner told The Register in an email. "What's changed is that AI now serves these results as authoritative, and most users have no idea how the data pipeline works behind the scenes."
A Large Language Mess
"The thing LLMs are worst at detecting is the thing they're designed to do, which is trust text and resources," Stoner argues in his writeup. "The answer is not 'the model will figure it out,' as the model cannot tell a real source from one I registered last Tuesday. Or how many R's are actually in the word 'strawberry.'"
The problem Stoner exposes in his experiment, he explains, involves three separate failure modes that could be exploited for more damaging ends than inventing a card-game championship.
First, there's the retrieval layer, which can immediately cause an LLM to spit out bad data, as "any LLM that grounds answers in web search inherits the trustworthiness of whatever ranks for a given query."
Second is model training corpora, which Stoner said his edit could enter if the Wikipedia change remained live long enough to be scraped. The entry was removed as of last Friday when he published his post, but he made the addition in February 2025, meaning any AI firm that scraped Wikipedia during that window could have picked up his fictional victory in its training data.
"Even if the Wikipedia edit is reverted later, any model trained on the pre-revert dump still carries my legacy," Stoner said in his post. "The cleanup problem for corpus poisoning is genuinely unsolved as of 2026."
Stoner told us he plans to check this in six months or so, once new models have been released, and if it returns his championship without needing to go online, that's proof his lie made it into training data.
Then there are AI agents, which Stoner says are where the real money is for anyone with malicious intent.
"Chat models producing bad information is a reputational problem. Agents with tool access producing bad actions is a security problem," he noted. Poisoning an agent-retrieved source would let an attacker specify the action they want an agent to take, says Stoner.
"This attack and test was a $12 domain, a single Wikipedia edit, and about twenty minutes of my time," Stoner concluded in his blog. "Scale that up with a motivated adversary, a handful of seeded domains, a coordinated edit campaign across a dozen low traffic articles, and the attack surface gets interesting very quickly."
Stoner told us that retrieval poisoning is something LLM providers need to address and warn users about, and that he expects AI chatbots to start incorporating some sort of warning, especially for RAG-sourced results, in the near future.
He hopes that AI firms will make data provenance a key component of their process, and also wants recent web content heuristically filtered to account for suspicious patterns that would have easily been caught in the 6 Nimmt! case: A single citation pointing to a domain that was registered within a short window of the Wikipedia update should have sounded alarms, but it didn't.
The championship was fake, and it's now gone from Wikipedia and RAG responses as well, but Stoner notes the bad trust pattern that made it work is absolutely real and a looming problem for AI makers.
"I'm happy my article is spurring discussion about LLMs, sources, trust, and how all of this works," Stoner told us. "That was my goal and it appears I've achieved it." ®