Can a chatbot be judgmental? For Ryan Raimi, a researcher from the University of Texas at Dallas, that thought should have been impossible. And yet, after investigating patient perceptions of mental health chatbots, that sentiment became apparent.
That's a setback for AI optimists who believe the technology can help expand patient access to mental healthcare.
Indeed, mental healthcare access leaves much to be desired. According to the National Association on Mental Illness, only 52.1% of adults with mental illness received treatment in 2024. For those with serious mental illness, that figure was 70.8%. Meanwhile, about half of kids aged 6-17 went without needed mental healthcare in 2024.
AI chatbots have been cautiously viewed as one solution to the nation's healthcare access problems, and mental healthcare is no different. According to Raimi, chatbots are inexpensive to deploy. And because they are available around the clock and even internationally, they could be helpful for folks who never dreamed of accessing therapy.
Perhaps most importantly, AI chatbots could deliver stigma-free mental healthcare simply by virtue of being a machine instead of a human. That's a strong argument for chatbots, given 84% of U.S. adults still think the term "mental illness" carries stigma, per the American Psychological Association.
"It's all about the fear of being judged by another human being," Raimi noted. "But then what if on the other side of the table, it's not a human, it's a machine, which is inherently incapable of judging you."
Raimi and his team got to work assessing the feasibility -- plus safety and efficacy -- of deploying mental health chatbots, foremost by examining the role of trust. Trust is a key component of any healthcare interaction, but especially mental healthcare. If patients can trust that a mental health chatbot won't judge them, they might be willing to use it.
But what the research team found was unexpected.
Can mental health chatbots be judgmental?
Raimi and his colleagues showed nearly 2,000 study participants a few text-based messages exchanged between a therapist and a hypothetical client experiencing depression, they wrote in the journal MIS Quarterly. The participants were split into four groups: two were told the therapist was a human, and the others were told it was a chatbot.
Despite the messages being identical, the study participants who learned the messages were sent by a mental health chatbot overwhelmingly perceived them as judgmental.
But what, exactly, makes a chatbot incapable of feeling more judgmental? Upon further qualitative probing, Raimi found it's the very fact that it's a chatbot, not a human, that isolates users.
For example, because mental health chatbots are machines, they have no real-world experience to draw from when conversing with a user. When a client or a patient speaks with a human therapist, the therapist can contextualize what it means to feel "down" or "anxious" because they've lived in the real world.
Likewise, a human therapist has understanding and knows what it means to feel socially isolated.
All of that affects the mental health chatbot's ability to validate the user, which is a significant shortcoming when it comes to therapy and mental healthcare.
"One of the reasons people told us they reach out to human providers in the first place is just to be heard -- not necessarily to find a solution, but just to talk to someone and feel validated and heard," Raimi explained.
But when a chatbot says, "It must be difficult to feel socially isolated," it rings hollow.
Still, Raimi maintains that mental health chatbots hold great promise in a world where such care is often inaccessible. Many patients agree. Earlier this year, Cognitive FX and Pollfish found that about a third of adults are using AI for mental health advice, indicating growing popularity. Among adolescents, that figure is about a fifth, per June data from RAND Corporation.
However, as these tools proliferate, it will be essential for AI developers to intentionally design them to help, not isolate, users.
Rethinking mental health chatbot prompt engineering
According to Raimi, improving the efficacy of mental health chatbots is a matter of prompt engineering.
Foremost, there are the tweaks necessary to ensure AI is deployed safely and ethically, he acknowledged.
Mental health chatbots should be engineered to only interact with users with a low risk profile and to refer higher-risk patients to a mental health professional. Designers should also consider the psychotherapy approach the chatbot will employ and any necessary guardrails to prevent patients from breaking out of any "safety loops" built into the AI.
But to cultivate a trusting, judgment-free relationship with users, mental health chatbots also need to retool their responses. Specifically, Raimi suggested developers remove the proverbial "I know how you feel" responses from the bots -- because they don't know.
Instead, bots might say, "I acknowledge that I have not experienced what you just described, but based on others' experiences, here is a potential solution."
This might require developers to consider training AI models on different types of psychotherapy, such as Freudian or Kohutian approaches that de-center empathy, Raimi recommended.
Raimi also suggested that mental health chatbots be designed to do exactly what LLMs have proven very adept at: aggregating relevant information for end users.
In this case, a mental health chatbot could crawl the internet for other information about how someone dealt with a similar situation or feeling the current user expressed. For example, the chatbot could provide breakup advice from online forums like Reddit or the experiences of other users, while de-identifying any information.
The idea is in its nascency, Raimi stressed, and it could use some fine-tuning. However, it could be an effective way for chatbots to convey empathy and real-world experience without being inauthentic.
"It's really promising in terms of research, implementation, real-world policy and chatbot design implementation and operationalization, but it's in an infancy state," Raimi concluded. "We need more time as researchers and scholars. But I wholeheartedly believe that in two to three years, we will have made significant progress."
Sara Heath is an executive editor at Xtelligent Healthcare Media, where she covers patient engagement, healthcare policy and health IT.
