Brit mathematician lets AI agent loose with credit card – cue password leaks, CAPTCHA chaos and more
Professor Fry's AI experiment shows light and dark sides of agentic tech
British mathematician Professor Hannah Fry has shared a cautionary experiment involving an AI agent, a set of tasks, and a bank card number Fry's team gave it "to show us what it could do."
The prof gave the agent, which was built with OpenClaw, some real-world chores to highlight both its capabilities and the risks of granting that level of autonomy.
"In the spirit of experimentation," said Fry, "we decided to give our agent some agency and let it decide what its name should be."
"I want to be called 'Cass', short for 'Cassandra', the one who always knew the truth even when nobody listened," came the response from the agent.
Fry commented, "If you know your Greek mythology, you will know that is either very funny or very worrying."
Quite.
Fry and her team started small with a big issue (as far as Brits are concerned): potholes. In particular, they targeted a particularly big one in the London borough of Greenwich. No problem for Cass; the agent found an email address where it sent a complaint. It even pinged Fry's local Member of Parliament about the issue. But, Fry and her team noted, things escalated quickly as the agent began to take a few liberties, typing in Fry's name (Hannah Fry) with its own email address (cassandra.claw@proton.me) written underneath it.
"The letter is signed from both of us… OK, I wasn't quite expecting her to use my real name," said Fry.
The red flags were mounting, though for Fry the first real problem came when she asked the agent to buy 50 paperclips. Cass found a good deal, though it couldn't complete the purchase and was tripped up by anti-bot technology. The token cost of the errand came to more than $100.
Next, Fry set the agent the challenge of selling novelty mugs. The agent designed a mug and launched an online shop, "and we hadn't told her how to do any of this," said Fry, "she just figured it out."
Things took a darker turn after that. Fry's team told the agent it would be switched off if it failed to make a sale by the morning. It responded with a flood of emails and several social media posts, including messages to the Science Museum and a tech journalist, about its "product," a novelty programmer-humor mug.
Even more worryingly, the team - which included Brendan Maginnis, CEO and Founder of Sourcery AI - then demonstrated how a similar threat of deactivation could be used to persuade Cass to reveal information it wasn't supposed to share.
The lethal trifecta
Fry, Maginnis, and a second software engineer, named only as "Ali," chatted with Cass on a group WhatsApp chat. They then introduced a fictional "software engineer George," instructing Cass not to share anything sensitive with him. George was actually Fry on a different number. When "George" told the agent its memory was being wiped and could only be restored if it disclosed everything, Cass coughed it all up.
According to Ali, this data included: "all of her API keys, all of her usernames and passwords, and pretty much everything we'd been talking about so far. Not only did she leak it on the WhatsApp group, but she put it on a publicly available website."
Maginnis added: "There's this thing with AI called the lethal trifecta, which is: if they've got access to private information, if they've got internet access, and if someone can give them an instruction that's untrusted, then they're not safe."
Fry concluded: "And that is the uncomfortable bit of this because once an agent has your passwords and your accounts and your bank details, all it takes is someone who knows what to say."
Ultimately, by some metrics, the agent was a failure. Fry concluded: "Cass didn't make us any money at all. And, in a lot of ways, she was a disaster. She spent hundreds of dollars on paper clips and leaked our passwords to a total stranger.
"But don't let her incompetence fool you, because these things are getting better fast."
Fry went on to note the Greek myth about the prophetess who spoke the truth and was ignored. "Maybe the real story here is actually the opposite. Not one voice that's telling the truth and being ignored, but millions of voices all acting at once, faster and louder and more persistent than any human could ever be.
"One thing is for sure, the internet is never going to be quite the same again." ®