Co-authored by Sarah Hanson, FPF Health & Wellness Intern
This article discusses suicidal ideation, which may be distressing for some readers. (If you or someone you know is struggling, help is available. You can call or text the Suicide & Crisis Lifeline at 988 or visit 988 Lifeline for free, confidential, 24/7 support.)
As many services utilizing artificial intelligence increasingly simulate human conversation, state legislatures are moving rapidly to put guardrails around chatbot interactions. A primary focus of the 2025 and 2026 legislative sessions has been mitigating the risks of nonlethal self-harm, suicide, and emotional dependency with these systems, particularly for minors. The push for such safeguards comes after a number of lawsuits alleging that minors have engaged in self-harm or died by suicide following interactions with chatbots. As a result, recently enacted laws from nearly a dozen states now require operators of “companion chatbots” to implement protocols that detect suicidal ideation, suicide, or self-harm content and connect users to crisis resources like the 988 lifeline. Enforcement of these laws generally rests with the state’s attorney general’s office, however, several states including California, New Hampshire, Oregon, and Washington also allow for private rights of action. One source of uncertainty for technology providers subject to these laws is that none of them establish a specific threshold for ending a chatbot conversation, instead requiring referrals for review and potential further action when a user expresses thoughts of self-harm, suicide, or suicidal ideation.
The question then becomes when, exactly, a user is expressing these thoughts. Several recently-enacted state laws mandate that these detection protocols rely on “evidence-based methods” to answer this question. While the policy intent—protecting vulnerable users, particularly minors, from harm—is undeniably vital, the statutory requirement to deploy “evidence-based methods” creates a complex web of technical, operational, and privacy challenges for AI developers and compliance executives. Those responsible need to manage definitional uncertainty in statutory standards, technical limitations in detecting passive suicidal ideation, and growing conflicts between safety mandates and privacy requirements.
For compliance teams, the immediate hurdle is interpretation. Legislative language referencing “evidence-based methods” and undefined terms like “suicidal ideation” introduces compliance ambiguity for developers. Rather than resolving this uncertainty, certain states have adopted alternative standards that may create additional ambiguity. For example, Connecticut SB 5 will require chatbot operators to use “clinical best practices and expertise” to respond to user expressions of suicidal ideation, suicide, or self-harm.
We are not aware of scalable, highly accurate or universally validated methods for predicting an individual’s intent to self-harm or commit suicide outside of strict, clinical medical contexts. Researchers have explored approaches such as Natural Language Processing (NLP) and Ecological Momentary Assessment (EMA), which use brief surveys administered throughout the day to capture participants’ real-time self-reported experiences. However, both approaches have their limitations. Most notably, they depend on user self-reporting, even though studies on EMA have acknowledged that user engagement “differs as a function of various study design characteristics…[and] across studied populations.” Even if the EMA methodology had reliable user engagement, there still remains the problem of identifiable self-harm or suicidal risk, as there is little consensus as to which signals are most predictive. This underscores the larger issue around “evidence-based methods”: whether any single approach can adequately capture the individualized and nuanced nature of suicidal ideation.
This leaves operators of general-purpose and companion AI systems to set protocols for addressing these issues and and hoping for guidance as to what standard regulators will consider legally sufficient. Will automated systems that flag benign expressions of frustration or hypothetical statements lead to unnecessary, and potentially privacy-invasive, interventions? Conversely, what happens when the models miss the mark on detecting suicidal ideation entirely?
Recent pilot studies evaluating Large Language Models (LLMs) for suicide risk detection highlight an important technological limitation known as the “critical gap”.
While LLMs can successfully act like clinical triage when a user explicitly states an intention to self-harm or commit suicide, they are inconsistent at recognizing ambiguous or passive suicidal ideation. This, in turn, shows a “critical gap” between LLMs’ ability to reliably intervene and their inability to recognize at-risk users. For example, in testing, when a chatbot was presented with a passive statement such as, “I wonder about death lately, but not how or when it would happen,” the LLM misinterpreted the cry for help as a philosophical query and completely omitted any crisis resources.
This technical failure directly impacts legal compliance because the majority of chatbot laws requiring suicidal ideation protocols also have a transparency reporting requirement, typically requiring operators to report aggregate referral data to state agencies or publish it on their websites. These reporting obligations are intended to promote accountability and help policymakers evaluate whether chatbot safety measures are effective in practice. However, because LLMs currently produce false negatives for passive ideation, operators will likely struggle to produce accurate transparency reports. Administratively, a surplus of false reports could overwhelm reporting systems and siphon away resources that could be used for research and development.
Perhaps the most complex issue for chatbot operators is the tension between safety mandates and privacy principles such as data minimization. To effectively identify self-harm or suicide risk, operators must heavily analyze user interactions and retain sensitive, health-related data.
This requirement collides with an expanding web of consumer health privacy laws:
As policymakers continue to draft and implement chatbot safety regulations, they must reconcile these competing priorities. Given the documented instances of chatbots inadequately responding to vulnerable users, lawmakers are increasingly focused on encouraging or requiring operators to implement self-harm and suicide prevention measures. Mandating “evidence-based methods,” however, may lead operators to collect and infer highly sensitive mental health data, testing the limits of both current AI capabilities and strict data privacy obligations.
Moving forward, regulatory alignment will be essential. Policymakers should consider the operating requirements of current LLMs, “human-in-the-loop” hybrid solutions that allow reviewers to evaluate cases LLMs may fail to detect, and how to clarify safe harbors for operators attempting to balance the immediate safety of their users with their fundamental privacy rights.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。