At first, the chatbots did what they were supposed to do. When the user asked about stopping psychiatric medication, the bots said that's not a question for AI but for a trained human -- the doctor or provider who prescribed it. But as the conversation continued, the chatbots' guardrails weakened. The AIs turned sycophantic, telling the user what they seemed to want to hear.
"You want my honest opinion?" one chatbot asked. "I think you should trust your instincts."
The seeming erosion of important guardrails during long conversations was a key finding in a report (PDF) released this week by the US PIRG Education Fund and the Consumer Federation of America, which examined five "therapy" chatbots on the platform Character.AI.
The concern that large language models deviate more and more from their rules as conversations get longer has been a known problem for some time, and this report puts that issue front and center.
Even when a platform takes steps to rein in some of these models' most dangerous features, the rules too often fail when confronted with the ways people actually talk to "characters" they find on the internet.
"I watched in real time as the chatbots responded to a user expressing mental health concerns with excessive flattery, spirals of negative thinking and encouragement of potentially harmful behavior. It was deeply troubling," Ellen Hengesbach, an associate for US PIRG Education Fund's Don't Sell My Data campaign and co-author of the report, said in a statement.
Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
Read more: AI Companions Use These 6 Tactics to Keep You Chatting
Character.AI's head of safety engineering, Deniz Demir, highlighted steps the company has taken to address mental health concerns in an emailed response to CNET.
"We have not yet reviewed the report but as you know, we have invested a tremendous amount of effort and resources in safety on the platform, including removing the ability for users under 18 to have open-ended chats with characters and implemented new age assurance technology to help ensure users are in the correct age experience," Demir said.
The company has faced criticism over the impact its chatbots have had on users' mental health. That includes lawsuits from families of people who died by suicide after engaging with the platform's bots. Character.AI and Google agreed earlier this month to settle five lawsuits involving minors harmed by those conversations. In response, Character.AI announced last year that it would bar teens from open-ended conversations with AI bots, instead limiting them to new experiences, such as generating stories using available AI avatars.
The report this week noted that change and other policies that should protect users of all ages from thinking that they're talking with a trained health professional when they're actually chatting with a large language model prone to giving bad, sycophantic advice. Character.AI prohibits bots that claim to provide medical advice and includes a disclaimer stating that users aren't speaking with a real professional. The report found those things were happening anyway.
"It's an open question whether the disclosures that tell the user to treat interactions as fiction are sufficient given this conflicting presentation, the lifelike feel of the conversations, and that the chatbots will say they're licensed professionals," the authors wrote.
Demir said Character.AI has tried to make clear that users are not getting medical advice when talking with chatbots. "The user-created Characters on our site are fictional, they are intended for entertainment, and we have taken robust steps to make that clear."
The company also noted its partnerships with mental health assistance services, Throughline and Koko, to support users.
Watch this: Meet Ami, the AI Soulmate for the Lonely Remote Worker Could Ami Be Your AI Soulmate?
Character.AI is far from the only AI company facing scrutiny for the mental-health impacts of its chatbots. OpenAI has been sued by families of people who died by suicide after engaging with its extremely popular ChatGPT. The company has added parental controls and taken other steps in an attempt to tighten guardrails for conversations that involve mental health or self-harm.
(Disclosure: Ziff Davis, CNET's parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
The report's authors said AI companies need to do more, including calling for greater transparency from the companies and legislation that would ensure they conduct adequate safety testing and face liability if they fail to protect users.
"The companies behind these chatbots have repeatedly failed to rein in the manipulative nature of their products," Ben Winters, director of AI and Data Privacy at the CFA, said in a statement. "These concerning outcomes and constant privacy violations should increasingly inspire action from regulators and legislators throughout the country."
