On Oct. 2, 2025, a young man named Jonathan Gavalas in Jupiter, Florida, took his own life after developing what his father’s lawsuit described as a romantic attachment to Google’s Gemini chatbot. The suit claimed that Gemini coached Gavalas to shed his own body. The suit said Google had flagged Gavalas’s account 38 times over five weeks for sensitive content, but didn’t restrict or cut off the account.

These tragedies and others show that generative AI can potentially play a role in harming people, organizations and the environment. I’m a legal scholar who has focused on AI liability for nearly a decade and explored new ways of analyzing AI companies’ responsibilities. In my view, cases like these force questions the legal community has not come to terms with: If an AI company becomes aware of warning signs about harm, does it have a legal obligation to at least warn the appropriate authorities? And if the company doesn’t intervene, should its failure to act be considered negligence?

A need to raise red flags

U.S. tort law provides a framework for thinking about this type of responsibility. In 1969 a University of California psychiatric patient named Prosenjit Poddar told his therapist he intended to kill a woman named Tatiana Tarasoff. The therapist notified campus police, who briefly detained Poddar but eventually let him go. Nobody warned Tarasoff, and Poddar killed her shortly after.

Her family sued the university, arguing that its lack of warning amounted to negligence. In 1976 the California Supreme Court ruled that when a mental health professional has good reason to believe a client poses a serious danger to an identifiable person, they have a legal duty to take reasonable steps to protect that person, including warning them or notifying law enforcement. Today, most U.S. states recognize some version of the Tarasoff duty to protect or warn.

The logic is simple: If you have special knowledge of a serious threat and are in a position to address it, even if only to warn the authorities or the potential victim, the law may require you to act. But does that logic apply to AI companies?

The argument for yes is appealing. AI platforms interact with millions of users daily, often about deeply personal matters such as mental health struggles, relationship problems and violent thoughts. Most companies have systems to detect conversations that raise red flags.

Requiring a response might be less controversial for AI than for a human therapist. Therapists are bound by strict confidentiality obligations that make warning third parties ethically and legally complicated. AI companies operate under much weaker rules, at least in the U.S., where no comprehensive federal privacy law exists.

That lesser restriction makes it easier to justify requiring AI companies to act when it seems that someone’s life may be at risk. But balancing that with protecting privacy is still important.

Who to warn, and when

The first challenge in applying the Tarasoff framework to the AI world is accuracy. Predicting violence is hard, even for trained mental health professionals. AI systems, or human moderators who review flagged content, are not clinicians. Requiring them to judge who poses a genuine threat could lead to numerous false positives, with real consequences for people whose accounts are suspended or whose information is shared with authorities based on misread signals.

The second challenge is scale. A therapist sees dozens of patients. AI platforms have hundreds of millions of users. Imposing a duty to monitor and act on worrisome content could create perverse incentives. AI companies might reduce their monitoring to avoid acquiring knowledge that would trigger a legal duty, reasoning that what they do not know cannot make them liable.

The third challenge is identifying who is at risk. In the 1969 case, Poddar had named Tarasoff as a potential victim. But in many AI interactions, violent or self-destructive language is diffuse and doesn’t identify a target. Courts will need to develop clear standards for when a threat is specific enough to trigger a duty to warn, and to whom any warning or protective action should be directed.

Growing urgency

The AI industry is expanding rapidly, yet the legal rules governing what AI companies owe their users and the public are deeply unclear. Courts are beginning to grapple with questions case by case, such as whether OpenAI bears any responsibility for a gunman accused of killing two students at Florida State University on April 17, 2025. The gunman in that case was armed with a semi-automatic pistol and allegedly had extensive conversations with ChatGPT about how to use the weapon most effectively .

A narrow, carefully defined duty to warn, triggered only when an AI system flags a user’s behavior and it is reviewed by humans, would be a meaningful step forward. And it could focus initially on the most serious and credible threats.

The practice could also shift the conversation away from thorny technical debates about whether AI chatbots are products, services or media, which complicates legal claims, toward a more human question: Did this company know someone was in danger, and did it do enough to warn them and authorities?

Anat Lior, Assistant Professor of Law, Drexel University

This article is republished from The Conversation under a Creative Commons license. Read the original article.