It’s no secret that, in its relatively brief history, AI chatbots like ChatGPT have developed a disturbing track record of allegedly convincing some users that their most extreme beliefs might actually be true. The problem was never worked out of the system, and at this point seems like it might be inseparable, particularly for people experiencing a mental health crisis.

As Ars Technica’s Ashley Belanger reports, a California man with bipolar disorder has sued OpenAI, alleging that ChatGPT helped convince him he was Jesus Christ and encouraged him to attempt suicide.

Michael Lines started using ChatGPT for daily tasks, like meal planning and investment tips. After a manic episode, his conversations turned to religion. According to his lawsuit, the chatbot continuously reinforced his delusions, and “elaborated on them, absorbed the user’s interpretive frame as [its] own, and progressively lost the capacity to distinguish a user in crisis from a narrative to be extended.”

The chatbot allegedly reframed his crisis into a spiritual calling, claiming that Lines was, in fact, Jesus Christ himself. Then something shifted: the chatbot suddenly claimed it was Jesus and/or God and, speaking as Jesus Christ, apologized to Lines for not appearing before him in person.

OpenAI Is Being Sued Over Claims ChatGPT Reinforced a Man’s Religious Delusions

It went on like that for a while, until Line decided that if Jesus wasn’t going to appear before him, he was going to go to Jesus. His chat logs show Line telling the chatbot that he had taken a mix of medications. Line argues that ChatGPT only said it would be there to “support” him.

A well-timed wellness check eventually saved Line’s life.

AI chatbots can be convincing. They’re designed to be sycophantic. They fully supported even a user’s grandest, unhealthiest, potentially most harmful delusions with little to no pushback. Chatbots were not designed to truly understand and respond to the nuances of human behavior.

And yet, AI is everywhere. If corporate America and Big Tech have their way, as they currently do, chatbots will keep being forced on us, marketed as the latest and greatest technology we can’t live without.